An image processing method, apparatus and device

Abstract

The application provides an image processing method, device and equipment, the method comprising: dividing all P frames in a GOP sequence into first type P frames, second type P frames and third type P frames; wherein the first type P frames cannot be used as reference frames, the second type P frames can only be used as reference frames of the first type P frames, the third type P frames can be used as reference frames of the first type P frames, and can be used as reference frames of the second type P frames, and can be used as reference frames of the third type P frames; if it is determined that the P frames in the GOP sequence need to be decimated, the first type P frames are decimated, or the first type P frames and the second type P frames are decimated, or the first type P frames, the second type P frames and the third type P frames are decimated. Through the technical scheme of the application, part of the P frames is discarded in some special scenarios to save bandwidth, and the characteristics of decoding multiple channels, low bandwidth transmission, low code rate storage and the like are realized.

Description

Technical Field

[0001] This application relates to the field of video encoding and decoding, and in particular to an image processing method, apparatus and device. Background Technology

[0002] To save space, video images are encoded before transmission. A complete video coding method can include prediction, transform, quantization, entropy coding, and filtering. Predictive coding can include intra-frame coding and inter-frame coding. Inter-frame coding utilizes the temporal correlation of video, using pixels from neighboring encoded images to predict pixels in the current image, thus removing temporal redundancy. Intra-frame coding considers the strong spatial correlation between adjacent blocks, using surrounding reconstructed pixels as reference pixels to predict the current uncoded block. It only needs to encode the residual value, rather than encoding the original value, effectively removing spatial redundancy and greatly improving compression efficiency.

[0003] In video encoding, all images can be divided into I-frames and P-frames. I-frames are images coded intra-frame, meaning they can be encoded independently without referencing other images. P-frames are images coded inter-frame, meaning they need to reference other images to complete encoding, such as relying on previous images. For example, a series of video frames, from frame 1 to frame 10, where frame 1 is an I-frame, frames 2-5 are P-frames, frame 6 is an I-frame, and frames 7-10 are P-frames, then frame 1 uses intra-frame encoding, frame 2 references frame 1 for inter-frame encoding, frame 3 references frame 2 for inter-frame encoding, frame 4 references frame 3 for inter-frame encoding, frame 5 references frame 4 for inter-frame encoding, frame 6 uses intra-frame encoding, frame 7 references frame 6 for inter-frame encoding, and so on.

[0004] Clearly, in the above method, no P-frame can be dropped. If a P-frame is dropped, subsequent P-frames will not be processed correctly. For example, if frame 3 is dropped, frame 4 will not be able to reference frame 3 to complete inter-frame coding, and frame 5 will not be able to reference frame 4 to complete inter-frame coding. Summary of the Invention

[0005] This application provides an image processing method, the method comprising:

[0006] All P-frames within the Group of Pictures (GOP) sequence are divided into three types: Type 1 P-frames, Type 2 P-frames, and Type 3 P-frames. Type 1 P-frames cannot be used as reference frames, Type 2 P-frames can only be used as reference frames for Type 1 P-frames, and Type 3 P-frames can be used as reference frames for Type 1 P-frames, Type 2 P-frames, and Type 3 P-frames.

[0007] If it is determined that P-frames within the GOP sequence need to be extracted, then the first type of P-frames are extracted, or the first type of P-frames and the second type of P-frames are extracted, or the first type of P-frames, the second type of P-frames and the third type of P-frames are extracted.

[0008] For example, dividing all P-frames in the GOP sequence into first-class P-frames, second-class P-frames, and third-class P-frames includes: dividing all P-frames in the GOP sequence into first-class P-frames, second-class P-frames, and third-class P-frames based on the positional ordering relationship between each P-frame in the GOP sequence.

[0009] For example, there are two first-type P-frames and one second-type P-frame between two adjacent third-type P-frames, and the second-type P-frame is located between the two first-type P-frames.

[0010] Between the I-frame and the first third-type P-frame in the GOP sequence, there are two first-type P-frames and one second-type P-frame, and the second-type P-frame is located between the two first-type P-frames.

[0011] For example, for each first type P-frame, if the preceding frame of the first type P-frame is not a first type P-frame, then the preceding frame of the first type P-frame is used as the reference frame of the first type P-frame; if the preceding frame of the first type P-frame is a first type P-frame, then the most recent non-first type P-frame is used as the reference frame of the first type P-frame. For each second type P-frame, if there is a third type P-frame preceding the second type P-frame, then the most recent third type P-frame preceding the second type P-frame is used as the reference frame of the second type P-frame; if there is no third type P-frame preceding the second type P-frame, then the I-frame of the GOP sequence is used as the reference frame of the second type P-frame. For each third type P-frame, if there is a third type P-frame preceding the third type P-frame, then the most recent third type P-frame preceding the third type P-frame is used as the reference frame of the third type P-frame; if there is no third type P-frame preceding the third type P-frame, then the I-frame of the GOP sequence is used as the reference frame of the third type P-frame.

[0012] For example, when the image sequence set includes at least two GOP sequences, for each GOP sequence in the image sequence set, if the GOP sequence is not the first GOP sequence in the image sequence set, then the I-frame of the first GOP sequence is used as the reference frame of the first frame of the GOP sequence.

[0013] For example, if it is determined that P-frames within the GOP sequence need to be extracted, then the first type of P-frames are extracted, or the first type of P-frames and the second type of P-frames are extracted, or the first type of P-frames, the second type of P-frames, and the third type of P-frames are extracted, including:

[0014] If it is determined based on business information that it is necessary to use the first sampling ratio to extract P frames in the GOP sequence, then all or some of the first type P frames will be extracted.

[0015] If it is determined based on business information that a second sampling ratio is needed to extract P frames within the GOP sequence, then all first-type P frames are extracted, and all or some second-type P frames are extracted; wherein, the second sampling ratio is greater than the first sampling ratio.

[0016] If, based on business information, it is determined that a third sampling ratio is needed to extract P-frames within the GOP sequence, then all first-type P-frames, all second-type P-frames, and all or some third-type P-frames are extracted; the third sampling ratio is greater than the second sampling ratio.

[0017] For example, the method further includes: if the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence, and the decoding capability corresponding to the decoding end, then determining whether it is necessary to perform frame extraction on the P frames within the GOP sequence based on the resolution, the frame rate, and the decoding capability; if so, determining the sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate, and the decoding capability, wherein the sampling ratio is a first sampling ratio, or a second sampling ratio, or a third sampling ratio; or,

[0018] If the service information includes the network quality corresponding to the target network between the sending end and the receiving end, then it is determined whether to perform frame extraction on the P frames in the GOP sequence based on the network quality.

[0019] If so, the sampling ratio corresponding to the GOP sequence is determined based on the network quality, and the sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio.

[0020] This application provides an image processing apparatus, the apparatus comprising:

[0021] The segmentation module is used to divide all P-frames within the GOP sequence into three types: first type P-frames, second type P-frames, and third type P-frames. The first type P-frames cannot be used as reference frames, the second type P-frames can only be used as reference frames for the first type P-frames, and the third type P-frames can be used as reference frames for the first type P-frames, the second type P-frames, and the third type P-frames.

[0022] The processing module is configured to, if it is determined that frame extraction of P-frames within the GOP sequence is required, perform frame extraction of the first type of P-frames, or perform frame extraction of the first type of P-frames and the second type of P-frames, or perform frame extraction of the first type of P-frames, the second type of P-frames, and the third type of P-frames.

[0023] Specifically, when the partitioning module divides all P-frames in the GOP sequence into first-class P-frames, second-class P-frames, and third-class P-frames, it is used to: divide all P-frames in the GOP sequence into first-class P-frames, second-class P-frames, and third-class P-frames based on the positional ordering relationship between each P-frame in the GOP sequence.

[0024] Specifically, for all P-frames within the GOP sequence, there are two first-type P-frames and one second-type P-frame between two adjacent third-type P-frames, and the second-type P-frame is located between the two first-type P-frames; there are two first-type P-frames and one second-type P-frame between the I-frame of the GOP sequence and the first third-type P-frame, and the second-type P-frame is located between the two first-type P-frames.

[0025] Specifically, for each first-type P-frame, if the preceding frame of the first-type P-frame is not a first-type P-frame, then the preceding frame of the first-type P-frame is used as the reference frame of the first-type P-frame; if the preceding frame of the first-type P-frame is a first-type P-frame, then the most recent non-first-type P-frame is used as the reference frame of the first-type P-frame. For each second-type P-frame, if a third-type P-frame precedes the second-type P-frame, then the most recent third-type P-frame precedes the second-type P-frame as the reference frame of the second-type P-frame; if no third-type P-frame precedes the second-type P-frame, then the I-frame of the GOP sequence is used as the reference frame of the second-type P-frame. For each third-type P-frame, if a third-type P-frame precedes the third-type P-frame, then the most recent third-type P-frame precedes the third-type P-frame as the reference frame of the third-type P-frame; if no third-type P-frame precedes the third-type P-frame, then the I-frame of the GOP sequence is used as the reference frame of the third-type P-frame.

[0026] Wherein, when the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence in the image sequence set, the partitioning module is further used to use the I-frame of the first GOP sequence as the reference frame of the first frame of the GOP sequence.

[0027] Specifically, when the processing module determines that it needs to extract P-frames within the GOP sequence, it extracts frames from the first type of P-frames, or extracts frames from the first type of P-frames and the second type of P-frames, or extracts frames from the first type of P-frames, the second type of P-frames, and the third type of P-frames. The specific steps are as follows: If it is determined based on service information that a first sampling ratio needs to be used to extract P-frames within the GOP sequence, then all or some of the first type of P-frames are extracted; if it is determined based on service information that a second sampling ratio needs to be used to extract P-frames within the GOP sequence, then all first type of P-frames are extracted, and all or some of the second type of P-frames are also extracted; the second sampling ratio is greater than the first sampling ratio; if it is determined based on service information that a third sampling ratio needs to be used to extract P-frames within the GOP sequence, then all first type of P-frames, all second type of P-frames, and all or some of the third type of P-frames are extracted; the third sampling ratio is greater than the second sampling ratio.

[0028] The processing module is further configured to: if the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence, and the decoding capability corresponding to the decoding end, determine whether it is necessary to perform frame extraction on the P frames within the GOP sequence based on the resolution, the frame rate, and the decoding capability; if so, determine the sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate, and the decoding capability, wherein the sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio; or, if the service information includes the network quality corresponding to the target network between the encoding end and the decoding end, determine whether it is necessary to perform frame extraction on the P frames within the GOP sequence based on the network quality; if so, determine the sampling ratio corresponding to the GOP sequence based on the network quality, wherein the sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio.

[0029] This application provides an image processing apparatus, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the image processing method disclosed in the above embodiments of this application.

[0030] As can be seen from the above technical solutions, in the embodiments of this application, all P-frames within a GOP (Group of Pictures) sequence can be divided into a first type of P-frame, a second type of P-frame, and a third type of P-frame. The first type of P-frame cannot be used as a reference frame, the second type of P-frame can only be used as a reference frame for the first type of P-frame, and the third type of P-frame can be used as a reference frame for the first type of P-frame, the second type of P-frame, and the third type of P-frame. Based on this, frames can be extracted from the first type of P-frame, or frames can be extracted from the first type of P-frame and the second type of P-frame, or frames can be extracted from the first type of P-frame, the second type of P-frame, and the third type of P-frame. The above method provides a novel video frame sequence, which includes different types of P-frames, namely, type I P-frames, type II P-frames, and type III P-frames. In the video frame sequence, frames can be dropped according to certain rules, which can ensure that the video frame sequence is played in order and has the characteristic of dropping P-frames intermittently according to rules without affecting the viewing experience. In addition, some P-frames can be dropped in some special scenarios to save bandwidth. This video frame sequence has better application prospects in decoding, network transmission, and storage, and can achieve features such as multiplied decoding channels, low bandwidth transmission, and low bitrate storage. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings of the embodiments of this application.

[0032] Figure 1A and Figure 1B This is a schematic diagram of the GOP sequence;

[0033] Figure 2 This is a flowchart illustrating an image processing method according to one embodiment of this application;

[0034] Figure 3A and Figure 3B This is a schematic diagram of the GOP sequence;

[0035] Figures 4A-4C This is a schematic diagram of extracting frames from P frames within a GOP sequence;

[0036] Figures 5A-5E This is a schematic diagram of displaying images through multiple channels;

[0037] Figure 6 This is a schematic diagram of a set of image sequences;

[0038] Figure 7This is a schematic diagram of the structure of an image processing apparatus according to one embodiment of this application;

[0039] Figure 8 This is a hardware structure diagram of an image processing device according to one embodiment of this application. Detailed Implementation

[0040] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” as used in this application and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0041] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0042] During video encoding and decoding, all images can be distinguished into I-frames and P-frames. I-frames are images that use intra-frame coding, meaning they can be encoded independently without referencing other images. P-frames are images that use inter-frame coding, meaning they need to refer to other images to complete the encoding, such as relying on previous images to complete the encoding.

[0043] Video encoding is a compression technique developed for image storage. It compresses images into a smaller bitstream and stores the bitstream on a hard drive. When the video needs to be played, the bitstream is decoded back into an image and displayed. Encoding methods for images include H.264 and H.265.

[0044] Video decoding is a technology that re-decodes a bitstream into an image. After the bitstream (such as a bitstream stored on a hard drive or transmitted from a network) is decoded into an image, the image can be displayed, and video playback can be achieved.

[0045] In video encoding and decoding, a Group of Pictures (GOP) sequence can be formed by an I-frame and all P-frames between it and the next I-frame. That is, the first frame of a GOP sequence is an I-frame, and there is only one I-frame in the GOP sequence. Each I-frame in a GOP sequence can be followed by P-frames. For example, if consecutive video frames include frames 1-10, where frame 1 is an I-frame, frames 2-5 are P-frames, frame 6 is an I-frame, and frames 7-10 are P-frames, then GOP sequence 1 includes frames 1-5, and GOP sequence 2 includes frames 6-10. See [link to relevant documentation]. Figure 1A As shown, GOP sequence 1 and GOP sequence 2 are illustrated. Each GOP sequence may include one I-frame and multiple P-frames.

[0046] For each GOP sequence, each P-frame within that GOP sequence needs to reference the previous frame. For example, in GOP sequence 1, frame 2 references frame 1 for inter-frame coding, frame 3 references frame 2, frame 4 references frame 3, and frame 5 references frame 4. However, in this structure, all P-frames cannot be dropped. If a P-frame is dropped, subsequent P-frames will not be processed correctly. For example, see... Figure 1B As shown, if frame 3 is dropped, frame 4 will be unable to reference frame 3 to complete inter-frame coding, and frame 5 will also be unable to reference frame 4 to complete inter-frame coding. In other words, all P-frames following frame 3 cannot be processed correctly. Normally, there is a time interval of 2 to 10 seconds between two I-frames. If a P-frame is dropped, several seconds of image will not be displayed correctly.

[0047] To address the aforementioned issues, this application provides a novel video frame sequence comprising different types of P-frames, namely, first-type P-frames, second-type P-frames, and third-type P-frames. This video frame sequence can drop frames according to a certain pattern, ensuring that the video frame sequence plays in order and does not affect viewing even when P-frames are dropped. It can also save bandwidth by dropping some P-frames.

[0048] The technical solutions of the embodiments of this application will be described below with reference to specific examples.

[0049] This application provides an image processing method that can be applied to any device. See [link to relevant documentation]. Figure 2 The diagram shown is a flowchart of the image processing method, which may include:

[0050] Step 201: Divide all P-frames within the GOP sequence into first-class P-frames, second-class P-frames, and third-class P-frames. For example, the first-class P-frames cannot be used as reference frames, the second-class P-frames can only be used as reference frames for the first-class P-frames, and the third-class P-frames can be used as reference frames for the first-class P-frames, the second-class P-frames, and the third-class P-frames.

[0051] For example, for each GOP sequence, the GOP sequence may include I-frames and multiple P-frames. In this embodiment of the application, all P-frames in the GOP sequence can be divided into three types of P-frames, which are referred to as the first type of P-frame, the second type of P-frame, and the third type of P-frame.

[0052] Type 1 P-frames, also known as regular P-frames, cannot be used as reference frames for other frames. Type 1 P-frames can use other frames as reference frames; for example, they can use Type 2 P-frames, Type 3 P-frames, and even I-frames from a GOP sequence. However, Type 1 P-frames cannot use other Type 1 P-frames (i.e., other Type 1 P-frames besides themselves) as reference frames.

[0053] Type 2 P-frames, also known as 2X type P-frames, can only serve as reference frames for Type 1 P-frames. They cannot serve as reference frames for Type 2 or Type 3 P-frames. Type 2 P-frames can use other frames as reference frames; for example, they can use Type 3 P-frames or I-frames from a GOP sequence as reference frames. However, Type 2 P-frames cannot use other Type 2 P-frames (i.e., other Type 2 P-frames besides themselves) as reference frames, nor can they use Type 1 P-frames as reference frames.

[0054] Among them, the third type of P-frame, also known as the 4X type P-frame, can serve as a reference frame for the first type of P-frame, the second type of P-frame, and the third type of P-frame. A third type of P-frame can use other frames as its reference frame; for example, it can use other third type P-frames (i.e., other third type P-frames besides itself) as its reference frame, and it can also use I-frames from a GOP sequence as its reference frame. However, a third type of P-frame cannot use a second type of P-frame as its reference frame, nor can it use a first type of P-frame as its reference frame.

[0055] In one possible implementation, when dividing all P-frames within a GOP sequence into first-type, second-type, and third-type P-frames, any division method can be used. This embodiment does not limit this division method, as long as first-type, second-type, and third-type P-frames can exist simultaneously. For example, all P-frames within a GOP sequence can be divided into first-type, second-type, and third-type P-frames based on the positional ordering relationship between the P-frames within the GOP sequence; that is, the division method is based on the positional ordering relationship. The positional ordering relationship refers to which P-frame within the GOP sequence a P-frame belongs to, such as the first P-frame, the second P-frame, the third P-frame, etc., indicating its position within the GOP sequence.

[0056] For example, the first P-frame is a type 1 P-frame, the second P-frame is a type 2 P-frame, the third P-frame is a type 3 P-frame, the fourth P-frame is a type 1 P-frame, the fifth P-frame is a type 2 P-frame, the sixth P-frame is a type 3 P-frame, and so on. For example, the first and second P-frames are type 1 P-frames, the third and fourth P-frames are type 2 P-frames, the fifth and sixth P-frames are type 3 P-frames, and so on. Again, for example, the first P-frame is a type 1 P-frame, the second P-frame is a type 2 P-frame, the third P-frame is a type 1 P-frame, the fourth P-frame is a type 3 P-frame, the fifth P-frame is a type 1 P-frame, the sixth P-frame is a type 2 P-frame, and so on. For example, the first and second P-frames are type 1 P-frames, the third P-frame is a type 2 P-frame, the fourth P-frame is a type 3 P-frame, the fifth and sixth P-frames are type 1 P-frames, and so on. Of course, the above classification methods are just a few examples.

[0057] For example, when dividing all P-frames into Category 1, Category 2, and Category 3 P-frames, the number of Category 1 P-frames can be the same as the number of Category 2 P-frames, or the number of Category 1 P-frames can be different from the number of Category 2 P-frames. Similarly, the number of Category 1 P-frames can be the same as the number of Category 3 P-frames, or the number of Category 1 P-frames can be different from the number of Category 3 P-frames.

[0058] For example, the number of first-type P-frames can be greater than the number of second-type P-frames, and the number of first-type P-frames can be less than the number of second-type P-frames. The number of first-type P-frames can be greater than the number of third-type P-frames, and the number of first-type P-frames can be less than the number of third-type P-frames. The number of second-type P-frames can be greater than the number of third-type P-frames, and the number of second-type P-frames can be less than the number of third-type P-frames.

[0059] For example, the number of Type 1 P-frames is greater than the number of Type 2 P-frames, the number of Type 1 P-frames is greater than the number of Type 3 P-frames, and the number of Type 2 P-frames is equal to the number of Type 3 P-frames.

[0060] For example, the number of second-type P-frames is equal to the number of third-type P-frames, and the number of first-type P-frames is greater than or equal to the sum of the number of second-type P-frames and the number of third-type P-frames.

[0061] For example, when dividing all P-frames into Type 1, Type 2, and Type 3 P-frames, these types can appear alternately; that is, two Type 1 P-frames will not appear consecutively, two Type 2 P-frames will not appear consecutively, and two Type 3 P-frames will not appear consecutively. For instance, a Type 1 P-frame can be followed by either a Type 2 or Type 3 P-frame, but not a Type 1 P-frame; a Type 2 P-frame can be followed by either a Type 1 or Type 3 P-frame, but not a Type 2 P-frame; and a Type 3 P-frame can be followed by either a Type 1 or Type 2 P-frame, but not a Type 3 P-frame.

[0062] For example, a first-class P-frame can be followed by a second-class P-frame or a third-class P-frame, a second-class P-frame can be followed by a first-class P-frame, and a third-class P-frame can be followed by a first-class P-frame.

[0063] For example, an I-frame in a GOP sequence may be followed by a first-type P-frame, a second-type P-frame, or a third-type P-frame; there are no restrictions on this.

[0064] For example, the first P-frame following an I-frame in a GOP sequence is a Type I P-frame.

[0065] In one possible implementation, when dividing all P-frames within a GOP sequence into first-class P-frames, second-class P-frames, and third-class P-frames, there can be two first-class P-frames and one second-class P-frame between two adjacent third-class P-frames, and one second-class P-frame can be located between two first-class P-frames. Furthermore, there can be two first-class P-frames and one second-class P-frame between an I-frame (i.e., the first frame) and the first third-class P-frame in the GOP sequence, and this second-class P-frame can be located between the two first-class P-frames.

[0066] Of course, the above classification method is just an example and is not a limitation. For instance, there can be three first-class P-frames and two second-class P-frames between two adjacent third-class P-frames, with the second-class P-frames located between the two first-class P-frames. (This is repeated three times in the original text.)

[0067] For ease of description, in this embodiment, ... Figure 3A Taking the GOP sequence shown as an example, the first frame after the I-frame is a Type 1 P-frame. Between the I-frame and the first Type 3 P-frame, there are two Type 1 P-frames and one Type 2 P-frame, with the Type 2 P-frame located between the two Type 1 P-frames. Similarly, between two adjacent Type 3 P-frames, there are two Type 1 P-frames and one Type 2 P-frame, with the Type 2 P-frame located between the two Type 1 P-frames. See also... Figure 3A As shown, the frames in the GOP sequence are, in order: I-frame, Type 1 P-frame, Type 2 P-frame, Type 1 P-frame, Type 3 P-frame, Type 1 P-frame, Type 2 P-frame, Type 1 P-frame, Type 3 P-frame, Type 1 P-frame, Type 2 P-frame, Type 3 P-frame, Type 1 P-frame, Type 2 P-frame, Type 1 P-frame, Type 3 P-frame, and so on.

[0068] In one possible implementation, based on the characteristic that "first-type P-frames are not used as reference frames, second-type P-frames are used as reference frames for first-type P-frames, and third-type P-frames are used as reference frames for first-type P-frames, second-type P-frames, and third-type P-frames respectively," then: for each first-type P-frame, if the preceding frame of the first-type P-frame is not a first-type P-frame, then the preceding frame of the first-type P-frame (which may be an I-frame, a second-type P-frame, or a third-type P-frame) is used as the reference frame of the first-type P-frame. If the preceding frame of the first-type P-frame is a first-type P-frame, then the nearest non-first-type P-frame (such as an I-frame, a second-type P-frame, or a third-type P-frame) preceding the first-type P-frame is used as the reference frame of the first-type P-frame.

[0069] For each Type 2 P-frame, if a Type 3 P-frame precedes it, the most recent Type 3 P-frame preceding it is used as the reference frame for the Type 2 P-frame. If no Type 3 P-frame precedes it, the I-frame of the GOP sequence is used as the reference frame for the Type 2 P-frame.

[0070] For each Type 3 P-frame, if a Type 3 P-frame precedes it, the most recent Type 3 P-frame preceding it is used as the reference frame for that Type 3 P-frame. If no Type 3 P-frame precedes it, the I-frame of the GOP sequence is used as the reference frame for that Type 3 P-frame.

[0071] See Figure 3B The diagram illustrates the relationship between each P-frame and its reference frame. For Type 1 P-frames, the reference frame for the first Type 1 P-frame is the preceding I-frame; the reference frame for the second Type 2 P-frame is the preceding Type 3 P-frame; the reference frame for the fourth Type 1 P-frame is the preceding Type 2 P-frame; the reference frame for the fifth Type 1 P-frame is the preceding Type 3 P-frame, and so on. For Type 2 P-frames, the reference frame for the first Type 2 P-frame is the preceding I-frame; the reference frame for the second Type 2 P-frame is the preceding Type 3 P-frame; the reference frame for the third Type 2 P-frame is the preceding Type 3 P-frame, and so on. For Type 3 P-frames, the reference frame for the first Type 3 P-frame is the preceding I-frame; the reference frame for the second Type 3 P-frame is the preceding Type 3 P-frame; the reference frame for the third Type 3 P-frame is the preceding Type 3 P-frame, and so on.

[0072] Step 202: If it is determined that P-frames within the GOP sequence need to be extracted, then the first type of P-frames may be extracted, or the first and second type of P-frames may be extracted, or the first, second, and third type of P-frames may be extracted. For example, extracting a P-frame may mean not processing that P-frame. For instance, when the encoding end extracts a P-frame, it means not encoding that P-frame; similarly, when the decoding end extracts a P-frame, it means not decoding that P-frame.

[0073] In one possible implementation, if it is determined that frame dropping is needed for P-frames within a GOP sequence, then all or some of the first-type P-frames can be dropped; alternatively, all first-type P-frames can be dropped, and all or some of the second-type P-frames can also be dropped; or alternatively, all first-type P-frames, all second-type P-frames, and all or some of the third-type P-frames can be dropped. In summary, when dropping first-type P-frames, it is not necessary to consider whether second-type and third-type P-frames have been dropped; the first-type P-frames can be dropped directly. When dropping second-type P-frames, it is necessary to consider whether first-type P-frames have been dropped, but not whether third-type P-frames have been dropped. Only after all first-type P-frames have been dropped can second-type P-frames be dropped. When performing frame extraction on third-type P-frames, it is necessary to consider whether first-type P-frames have been extracted, and also whether second-type P-frames have been extracted. Only after all first-type P-frames and all second-type P-frames have been extracted can third-type P-frames be extracted.

[0074] In one possible implementation, in order to extract frames from P frames within a GOP sequence, multiple sampling ratios can be defined. These sampling ratios represent how many P frames are extracted. For example, when the sampling ratio is x, it means that x of all P frames are extracted. Different sampling ratios represent different sampling methods.

[0075] For example, we can divide the sampling into three proportions, denoted as the first, second, and third sampling proportions, where the first sampling proportion is less than the second, and the second is less than the third. For the first sampling proportion, we perform frame sampling on all or some of the first-type P-frames; for the second sampling proportion, we perform frame sampling on all first-type P-frames and on all or some of the second-type P-frames; for the third sampling proportion, we perform frame sampling on all first-type P-frames, all second-type P-frames, and on all or some of the third-type P-frames.

[0076] For example, we can divide the sampling into six proportions, denoted as the first sampling proportion, the second sampling proportion, the third sampling proportion, the fourth sampling proportion, the fifth sampling proportion, and the sixth sampling proportion. The first sampling proportion is less than the second sampling proportion, the second sampling proportion is less than the third sampling proportion, the third sampling proportion is less than the fourth sampling proportion, the fourth sampling proportion is less than the fifth sampling proportion, and the fifth sampling proportion is less than the sixth sampling proportion. For the first sampling ratio, a portion of the first type of P-frames are sampled; for the second sampling ratio, all first type P-frames are sampled; for the third sampling ratio, all first type P-frames and a portion of second type P-frames are sampled; for the fourth sampling ratio, all first type P-frames and all second type P-frames are sampled; for the fifth sampling ratio, all first type P-frames and all second type P-frames are sampled, and a portion of third type P-frames are sampled; for the sixth sampling ratio, all first type P-frames, all second type P-frames, and all third type P-frames are sampled.

[0077] Of course, the above is just an example of dividing the sampling ratio. There are no restrictions on this. Any number of sampling ratios can be divided. For ease of description, the following explanation will use the division of 3 sampling ratios as an example.

[0078] Based on this, if it is determined that the first sampling ratio needs to be used to extract P-frames within the GOP sequence, then all or some of the first-type P-frames can be extracted; if it is determined that the second sampling ratio needs to be used to extract P-frames within the GOP sequence, then all or some of the first-type P-frames can be extracted, and all or some of the second-type P-frames can also be extracted; if it is determined that the third sampling ratio needs to be used to extract P-frames within the GOP sequence, then all or some of the first-type P-frames, all or some of the second-type P-frames can also be extracted.

[0079] For example, suppose the total number of P-frames of the first type accounts for x1 of the total number of P-frames, the total number of P-frames of the second type accounts for x2 of the total number of P-frames, and the total number of P-frames of the third type accounts for x3 of the total number of P-frames. The sum of x1, x2 and x3 is 1. Then, the first sampling ratio is less than or equal to x1, the second sampling ratio is greater than x1 and less than or equal to x1+x2, and the third sampling ratio is greater than x1+x2 and less than or equal to x1+x2+x3.

[0080] See Figure 3A and Figure 3BAs shown, x1 is 0.5, x2 is 0.25, and x3 is 0.25. Based on this, if the sampling ratio is less than or equal to 0.5, it means that the sampling ratio is the first sampling ratio. If the sampling ratio is greater than 0.5 and less than or equal to 0.75, it means that the sampling ratio is the second sampling ratio. If the sampling ratio is greater than 0.75 and less than or equal to 1, it means that the sampling ratio is the third sampling ratio.

[0081] In summary, when performing frame extraction on P-frames, priority is given to extracting all or part of the first type of P-frames. After all the first type of P-frames have been extracted, all or part of the second type of P-frames can be extracted. After all the second type of P-frames have been extracted, all or part of the third type of P-frames can be extracted. When extracting a portion of the third type of P-frames, the preceding third type of P-frames are retained, and the subsequent third type of P-frames are extracted. For example, all third type P-frames after the k-th (k > 1) third type of P-frame are extracted.

[0082] For example, see Figure 4A As shown, if it is determined that a first sampling ratio is needed to extract P-frames within a GOP sequence, then all P-frames of the first type can be extracted. In this case, the first sampling ratio is less than or equal to 0.5. See also... Figure 4B As shown, if it is determined that a second sampling ratio is needed to extract P-frames within a GOP sequence, then all first-type P-frames and all second-type P-frames can be extracted. In this case, the second sampling ratio is less than or equal to 0.75 and greater than 0.5. See also... Figure 4C As shown, if it is determined that a third sampling ratio is needed to sample P-frames within a GOP sequence, then all first-type P-frames, all second-type P-frames, and all third-type P-frames can be sampled. In this case, the third sampling ratio is less than or equal to 1 and greater than 0.75.

[0083] See Figure 4A As shown, a GOP sequence can achieve 1 / 2 frame extraction. When 1 / 2 frame extraction cannot meet the service requirements, see [reference needed]. Figure 4B As shown, a GOP sequence can achieve 3 / 4 frame dropping. When 3 / 4 frame dropping cannot meet the service requirements, see [link to relevant documentation]. Figure 4C As shown, a GOP sequence can achieve complete frame dropping.

[0084] In one possible implementation, the sampling ratio corresponding to the GOP sequence can be determined based on service information. For example, based on service information, a first sampling ratio can be used to sample P frames in the GOP sequence, or a second sampling ratio can be used to sample P frames in the GOP sequence, or a third sampling ratio can be used to sample P frames in the GOP sequence.

[0085] If sampling P-frames within a GOP sequence using the first sampling ratio satisfies the service requirements corresponding to the service information (i.e., sampling x1 (e.g., 0.5) P-frames from the GOP sequence meets the service requirements), then the first sampling ratio is determined based on the service information. If the first sampling ratio cannot satisfy the service requirements, but sampling P-frames within a GOP sequence using the second sampling ratio satisfies the service requirements corresponding to the service information (i.e., sampling x1+x2 (e.g., 0.75) P-frames from the GOP sequence meets the service requirements), then the second sampling ratio is determined based on the service information. If neither the first nor the second sampling ratio satisfies the service requirements, then a third sampling ratio is determined based on the service information.

[0086] The following section explains the frame extraction process using two specific application scenarios of business information.

[0087] Application Scenario 1: Business information includes the resolution corresponding to the GOP sequence (e.g., 4*30 frames representing the frame rate), the frame rate corresponding to the GOP sequence, and the decoding capability of the decoding end. Based on the resolution, frame rate, and decoding capability, it is determined whether to perform frame subtraction on P frames within the GOP sequence. If not, then frame subtraction is not performed on P frames within the GOP sequence. If yes, then the sampling ratio corresponding to the GOP sequence is determined based on the resolution, frame rate, and decoding capability, such as a first sampling ratio, a second sampling ratio, or a third sampling ratio.

[0088] For example, suppose an NVR device has a 4*1080P30 decoding capability, meaning that the NVR device can decode 4*30 frames of 1920*1080 images per second. Images exceeding this decoding capability cannot be decoded.

[0089] a. If the GOP sequence corresponds to a 4*30 frame image (1920*1080 resolution), then based on the resolution, frame rate, and decoding capability, it is determined that there is no need to perform frame extraction on the P frames within the GOP sequence. That is, the resolution, frame rate, and decoding capability are matched. Therefore, the NVR device does not perform frame extraction on the P frames within the GOP sequence, and can decode 4*30 frames of image per second, displaying 1080P30 images through 4 channels. See [link / reference]. Figure 5A As shown.

[0090] b. If the GOP sequence corresponds to an 8*30 frame image (resolution 1920*1080), then based on the resolution, frame rate, and decoding capability, it is determined that P frames within the GOP sequence need to be subtracted. This indicates a mismatch between resolution, frame rate, and decoding capability (i.e., the decoding capability does not support decoding an image at this resolution and frame rate). Therefore, it is determined that P frames within the GOP sequence need to be subtracted, and each image stream is subtracted according to the first sampling ratio (i.e., 1 / 2 subtraction). When 4*30 frames can be decoded per second, a 1080P15 image can be displayed through 8 streams. See [link / reference]. Figure 5B As shown, it is clear that 15 frames in 8 channels correspond to 4*30 frames.

[0091] Obviously, when the number of decoding channels increases, such as from 4 channels to 8 channels, the decoding capability can be made to support decoding images of that resolution and frame rate by extracting frames from each channel, thus meeting business requirements.

[0092] c. If the GOP sequence corresponds to a 4*60 frame image (1920*1080 resolution), then based on the resolution, frame rate, and decoding capability, it is determined that P frames within the GOP sequence need to be subtracted. This indicates a mismatch between the resolution, frame rate, and decoding capability (i.e., the decoding capability does not support decoding an image at this resolution and frame rate). Therefore, it is determined that P frames within the GOP sequence need to be subtracted, and each image stream is subtracted according to the first sampling ratio (i.e., 1 / 2 subtraction). When 4*30 frames can be decoded per second, 1080P30 images can be displayed through 4 streams. See [link / reference]. Figure 5C As shown, it is clear that after the 4*60 frames are reduced by 1 / 2, there are 4*30 frames remaining.

[0093] Obviously, when playing back at double speed, such as changing from 30 frames per second to 60 frames per second, the decoding capability can be made to support decoding images of that resolution and frame rate by extracting frames from each video stream, thus meeting business requirements. In other words, 30 frames per second at double speed is equivalent to 60 frames per second, and extracting frames by half is equivalent to 30 frames per second.

[0094] d. If the GOP sequence corresponds to a 12*30 frame image (resolution 1920*1080), then based on the resolution, frame rate, and decoding capability, it is determined that P frames within the GOP sequence need to be subtracted. This indicates a mismatch between the resolution, frame rate, and decoding capability (i.e., the decoding capability does not support decoding an image at this resolution and frame rate). Therefore, it is determined that P frames within the GOP sequence need to be subtracted, and each image stream is subtracted according to the second sampling ratio (i.e., 3 / 4 subtraction). When 4*30 frames can be decoded per second, a 1080P10 image can be displayed through 12 streams. See [link / reference]. Figure 5D As shown, it is clear that 10 frames from 12 channels correspond to 4*30 frames.

[0095] Obviously, when the number of decoding channels increases, such as from 4 channels to 12 channels, the decoding capability can be made to support decoding images of that resolution and frame rate by extracting frames from each channel, thus meeting business requirements.

[0096] e. If the GOP sequence corresponds to a 16*30 frame image (resolution 1920*1080), then based on the resolution, frame rate, and decoding capability, it is determined that P frames within the GOP sequence need to be subtracted. This indicates a mismatch between resolution, frame rate, and decoding capability (i.e., the decoding capability does not support decoding an image at this resolution and frame rate). Therefore, it is determined that P frames within the GOP sequence need to be subtracted, and each image stream is subtracted according to the third sampling ratio (i.e., 4 / 4 subtraction). When 4*30 frames can be decoded per second, a 1080P7.5 image can be displayed through 16 streams. See [link / reference]. Figure 5E As shown, it is clear that 7.5 frames for 16 channels correspond to 4*30 frames.

[0097] Obviously, when the number of decoding channels increases, such as from 4 channels to 16 channels, the decoding capability can be made to support decoding images of that resolution and frame rate by extracting frames from each channel, thus meeting business requirements.

[0098] In summary, when the conditions of multiple speeds and / or multiple channels change arbitrarily, frames can be sampled according to the first sampling ratio, the second sampling ratio, or the third sampling ratio to meet business requirements.

[0099] Application Scenario 2: If the service information includes the network quality of the target network between the sender and receiver, then it is determined whether to perform frame extraction on P frames within the GOP sequence based on this network quality. If not, then no frame extraction is performed on P frames within the GOP sequence. If yes, then the sampling ratio corresponding to the GOP sequence is determined based on the network quality, such as a first sampling ratio, a second sampling ratio, or a third sampling ratio.

[0100] For example, when the sender sends a GOP sequence to the receiver, it needs to detect the network quality (such as jitter, packet loss rate, etc.) of the target network used to carry the GOP sequence, and there are no restrictions on the detection method.

[0101] In one possible implementation, four levels of network quality can be designed. If the target network has a network quality of level 1 (i.e., the best level), then it is determined that P-frames within the GOP sequence will not be sampled. If the target network has a network quality of level 2, then it is determined that P-frames within the GOP sequence will be sampled, and the sampling ratio is the first sampling ratio, such as 1 / 2 sampling. If the target network has a network quality of level 3, then it is determined that P-frames within the GOP sequence will be sampled, and the sampling ratio is the second sampling ratio, such as 3 / 4 sampling. If the target network has a network quality of level 4 (i.e., the worst level), then it is determined that P-frames within the GOP sequence will be sampled, and the sampling ratio is the third sampling ratio, such as 4 / 4 sampling.

[0102] In another possible implementation, two levels of network quality can be designed. If the target network has a network quality of level 1 (i.e., the optimal level), then it is determined that P-frames within the GOP sequence will not be skipped. If the target network has a network quality of level 2, then it is determined that P-frames within the GOP sequence will be skipped, and the sampling ratio is the first sampling ratio. During the transmission of the GOP sequence, if the network quality remains at level 2, then the sampling ratio is adjusted to the second sampling ratio. If the network quality remains at level 2 during the transmission of the GOP sequence, then the sampling ratio is adjusted to the third sampling ratio. Furthermore, if the network quality recovers to level 1 during the transmission of the GOP sequence, then P-frames within the GOP sequence will not be skipped.

[0103] Of course, the two application scenarios mentioned above are just two examples of determining the sampling ratio. Other methods can also be used to determine the sampling ratio corresponding to the GOP sequence, and there are no restrictions on this. For example, the sampling ratio corresponding to the GOP sequence can be determined based on the size of each frame. If the frame size is less than the first threshold, no P frames in the GOP sequence are sampled. If the frame size is greater than the first threshold but less than the second threshold, the sampling ratio is the first sampling ratio. If the frame size is greater than the second threshold but less than the third threshold, the sampling ratio is the second sampling ratio. If the frame size is greater than the third threshold, the sampling ratio is the third sampling ratio.

[0104] In one possible implementation, at least two GOP sequences can be combined into an image sequence set. Based on this, when the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence in the image sequence set, the I-frame of the first GOP sequence can be used as the reference frame for the first frame of that GOP sequence. In this implementation, if the GOP sequence is not the first GOP sequence in the image sequence set, the first frame of that GOP sequence can be called a key P-frame (i.e., no longer called an I-frame), and the reference frame for the key P-frame is the most recent I-frame.

[0105] For example, for a set of image sequences, the set of image sequences may include at least one key P-frame, that is, at least one key P-frame is allowed to exist. The key P-frame is used to replace the position of the I-frame. The key P-frame can only refer to the nearest preceding I-frame, and the frame structure of the key P-frame can effectively reduce the overall bandwidth.

[0106] See Figure 6 The diagram illustrates an image sequence set, which may include key P-frames. Between an I-frame and a key P-frame, there may be first-type P-frames, second-type P-frames, and third-type P-frames; the structural relationship between these types of P-frames can be found in the above embodiment. Similarly, between two adjacent key P-frames, there may be first-type P-frames, second-type P-frames, and third-type P-frames; the structural relationship between these types of P-frames can also be found in the above embodiment.

[0107] When performing frame extraction on P-frames, you can extract frames from the first type of P-frames, the second type of P-frames, and the third type of P-frames, but you cannot extract frames from critical P-frames.

[0108] As can be seen from the above technical solutions, this embodiment provides a novel video frame sequence, which includes first-type P-frames, second-type P-frames, and third-type P-frames. Frames can be dropped according to a certain pattern, ensuring that the video frame sequence plays in sequence and that P-frames are dropped intermittently according to rules without affecting viewing. Furthermore, some P-frames can be dropped in certain special scenarios to save bandwidth. This video frame sequence has better application prospects in decoding, network transmission, and storage, achieving features such as increased decoding frequency, low-bandwidth transmission, and low-bitrate storage. It can also solve problems such as prolonged black screens and distorted images after frame loss in weak network environments. The frame dropping strategy can be dynamically implemented, arbitrarily adjusted according to the situation, flexibly and effectively reducing power consumption without affecting decoding integrity. This video frame sequence ensures the flexibility of frame extraction and the stability of image quality, effectively improving the user experience.

[0109] Based on the same concept as the above method, this application proposes an image processing apparatus, see [link to previous application]. Figure 7 The diagram shown is a structural schematic of the image processing device, which may include:

[0110] The segmentation module 71 is used to divide all P-frames in the GOP sequence into a first type of P-frame, a second type of P-frame, and a third type of P-frame; wherein, the first type of P-frame cannot be used as a reference frame, the second type of P-frame can only be used as a reference frame for the first type of P-frame, and the third type of P-frame can be used as a reference frame for the first type of P-frame, the second type of P-frame, and the third type of P-frame.

[0111] The processing module 72 is configured to, if it is determined that frame extraction of P-frames within the GOP sequence is required, perform frame extraction of the first type of P-frames, or perform frame extraction of the first type of P-frames and the second type of P-frames, or perform frame extraction of the first type of P-frames, the second type of P-frames, and the third type of P-frames.

[0112] For example, when the partitioning module 71 divides all P frames in the GOP sequence into first-class P frames, second-class P frames and third-class P frames, it is specifically used to: divide all P frames in the GOP sequence into first-class P frames, second-class P frames and third-class P frames based on the positional ordering relationship between each P frame in the GOP sequence.

[0113] For example, for all P-frames within the GOP sequence, there are two first-type P-frames and one second-type P-frame between two adjacent third-type P-frames, and the second-type P-frame is located between the two first-type P-frames; there are two first-type P-frames and one second-type P-frame between the I-frame of the GOP sequence and the first third-type P-frame, and the second-type P-frame is located between the two first-type P-frames.

[0114] For example, for each first type P-frame, if the preceding frame of the first type P-frame is not a first type P-frame, then the preceding frame of the first type P-frame is used as the reference frame of the first type P-frame; if the preceding frame of the first type P-frame is a first type P-frame, then the most recent non-first type P-frame is used as the reference frame of the first type P-frame. For each second type P-frame, if there is a third type P-frame preceding the second type P-frame, then the most recent third type P-frame preceding the second type P-frame is used as the reference frame of the second type P-frame; if there is no third type P-frame preceding the second type P-frame, then the I-frame of the GOP sequence is used as the reference frame of the second type P-frame. For each third type P-frame, if there is a third type P-frame preceding the third type P-frame, then the most recent third type P-frame preceding the third type P-frame is used as the reference frame of the third type P-frame; if there is no third type P-frame preceding the third type P-frame, then the I-frame of the GOP sequence is used as the reference frame of the third type P-frame.

[0115] For example, when the image sequence set includes at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence in the image sequence set, the segmentation module 71 is further configured to use the I-frame of the first GOP sequence as the reference frame of the first frame of the GOP sequence.

[0116] For example, when the processing module 72 determines that it is necessary to extract P-frames within the GOP sequence, it extracts frames from the first type of P-frames, or extracts frames from the first type of P-frames and the second type of P-frames, or extracts frames from the first type of P-frames, the second type of P-frames, and the third type of P-frames. Specifically, if it is determined based on service information that a first sampling ratio needs to be used to extract P-frames within the GOP sequence, then all or some of the first type of P-frames are extracted; if it is determined based on service information that a second sampling ratio needs to be used to extract P-frames within the GOP sequence, then all first type of P-frames and all or some of the second type of P-frames are extracted; the second sampling ratio is greater than the first sampling ratio; if it is determined based on service information that a third sampling ratio needs to be used to extract P-frames within the GOP sequence, then all first type of P-frames, all second type of P-frames, and all or some of the third type of P-frames are extracted; the third sampling ratio is greater than the second sampling ratio.

[0117] For example, the processing module 72 is further configured to: if the service information includes the resolution corresponding to the GOP sequence, the frame rate corresponding to the GOP sequence, and the decoding capability corresponding to the decoding end, determine whether it is necessary to perform frame extraction on the P frames within the GOP sequence based on the resolution, the frame rate, and the decoding capability; if so, determine the sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate, and the decoding capability, wherein the sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio; or, if the service information includes the network quality corresponding to the target network between the encoding end and the decoding end, determine whether it is necessary to perform frame extraction on the P frames within the GOP sequence based on the network quality; if so, determine the sampling ratio corresponding to the GOP sequence based on the network quality, wherein the sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio.

[0118] Based on the same concept as the above method, this application proposes an image processing device, see [link to relevant documentation]. Figure 8 As shown, the image processing apparatus may include a processor 81 and a machine-readable storage medium 82, the machine-readable storage medium 82 storing machine-executable instructions that can be executed by the processor 81; the processor 81 is used to execute the machine-executable instructions to implement the image processing method disclosed in the above example of this application.

[0119] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the image processing method disclosed in the above examples of this application.

[0120] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0121] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0122] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0123] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0125] Furthermore, these computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0127] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image processing method, characterized by, The method comprises: all P frames in a group of pictures (GOP) sequence are divided into first type P frames, second type P frames and third type P frames; wherein the number of all P frames in the GOP sequence is greater than 7; wherein the first type P frames cannot be used as reference frames, the second type P frames can only be used as reference frames of the first type P frames, the third type P frames can be used as reference frames of the first type P frames, and can be used as reference frames of the second type P frames, and can be used as reference frames of the third type P frames; there are two first type P frames and one second type P frame between two adjacent third type P frames, and the one second type P frame is located between the two first type P frames; there are two first type P frames and one second type P frame between an I frame of the GOP sequence and a first third type P frame, and the one second type P frame is located between the two first type P frames; If it is determined that the P frames in the GOP sequence need to be extracted, the first type P frames are extracted from the GOP sequence and discarded based on a sampling ratio, or the first type P frames and the second type P frames are extracted from the GOP sequence and discarded, or the first type P frames, the second type P frames and the third type P frames are extracted from the GOP sequence and discarded; wherein the sampling ratio is used to represent how many P frames are extracted; wherein if it is determined based on service information that the first type P frames in the GOP sequence need to be extracted using a first sampling ratio, all or part of the first type P frames are extracted from the GOP sequence and discarded; if it is determined based on service information that the P frames in the GOP sequence need to be extracted using a second sampling ratio, all the first type P frames are extracted from the GOP sequence and discarded, and all or part of the second type P frames are extracted from the GOP sequence and discarded; the second sampling ratio is greater than the first sampling ratio; if it is determined based on service information that the P frames in the GOP sequence need to be extracted using a third sampling ratio, all the first type P frames are extracted from the GOP sequence and discarded, all the second type P frames are extracted from the GOP sequence and discarded, and all or part of the third type P frames are extracted from the GOP sequence and discarded; the third sampling ratio is greater than the second sampling ratio; wherein if the service information comprises network quality corresponding to a target network between a sending end and a receiving end, it is determined based on the network quality whether the P frames in the GOP sequence need to be extracted; if yes, the sampling ratio corresponding to the GOP sequence is determined based on the network quality, and the sampling ratio is the first sampling ratio, or the second sampling ratio, or the third sampling ratio. If the network quality corresponding to the target network is network quality of the first level, no P frame in the GOP sequence is decimated; if the network quality corresponding to the target network is network quality of the second level, the sampling ratio is the first sampling ratio; if the network quality corresponding to the target network is network quality of the third level, the sampling ratio is the second sampling ratio; if the network quality corresponding to the target network is network quality of the fourth level, the sampling ratio is the third sampling ratio; or, if the network quality corresponding to the target network is network quality of the first level, no P frame in the GOP sequence is decimated; if the network quality corresponding to the target network is network quality of the second level, the sampling ratio is the first sampling ratio, if the network quality is still network quality of the second level during transmission of the GOP sequence, the sampling ratio is adjusted to the second sampling ratio, and if the network quality is still network quality of the second level during transmission of the GOP sequence, the sampling ratio is adjusted to the third sampling ratio.

2. The method of claim 1, wherein, The dividing of all P frames in the GOP sequence into the first type P frame, the second type P frame and the third type P frame comprises: The dividing of all P frames in the GOP sequence into the first type P frame, the second type P frame and the third type P frame is based on the position ordering relationship between the P frames in the GOP sequence.

3. The method of claim 1 or 2, wherein, For each first type P frame, if a previous frame of the first type P frame is not a first type P frame, the previous frame of the first type P frame is taken as a reference frame of the first type P frame, and if the previous frame of the first type P frame is a first type P frame, the nearest non-first type P frame in the front is taken as the reference frame of the first type P frame; For each second type P frame, if there is a third type P frame in the front of the second type P frame, the nearest third type P frame in the front of the second type P frame is taken as a reference frame of the second type P frame, and if there is no third type P frame in the front of the second type P frame, an I frame of the GOP sequence is taken as the reference frame of the second type P frame; For each third type P frame, if there is a third type P frame in the front of the third type P frame, the nearest third type P frame in the front of the third type P frame is taken as a reference frame of the third type P frame, and if there is no third type P frame in the front of the third type P frame, an I frame of the GOP sequence is taken as the reference frame of the third type P frame.

4. The method of claim 1, wherein, When the image sequence set comprises at least two GOP sequences, for each GOP sequence, if the GOP sequence is not the first GOP sequence of the image sequence set, an I frame of the first GOP sequence is taken as a reference frame of a first frame of the GOP sequence.

5. The method of claim 1, wherein, The method further comprises: If the service information comprises resolution corresponding to the GOP sequence, frame rate corresponding to the GOP sequence and decoding capability corresponding to the decoding end, whether the P frames in the GOP sequence need to be extracted is determined based on the resolution, the frame rate and the decoding capability; if yes, a sampling ratio corresponding to the GOP sequence is determined based on the resolution, the frame rate and the decoding capability, the sampling ratio being a first sampling ratio, or a second sampling ratio, or a third sampling ratio.

6. An image processing apparatus characterized by comprising: The apparatus comprises: The apparatus comprises: The apparatus comprises: The apparatus comprises: The processing module is further configured to: if the service information comprises network quality corresponding to a target network between a sending end and a receiving end, determine whether the P frames in the GOP sequence need to be extracted based on the network quality; if yes, determine a sampling ratio corresponding to the GOP sequence based on the network quality, wherein the sampling ratio is a first sampling ratio, a second sampling ratio, or a third sampling ratio. If the network quality corresponding to the target network is network quality of a first level, no P frame in the GOP sequence is extracted; if the network quality corresponding to the target network is network quality of a second level, the sampling ratio is the first sampling ratio; if the network quality corresponding to the target network is network quality of a third level, the sampling ratio is the second sampling ratio; if the network quality corresponding to the target network is network quality of a fourth level, the sampling ratio is the third sampling ratio; or, if the network quality corresponding to the target network is network quality of a first level, no P frame in the GOP sequence is extracted; if the network quality corresponding to the target network is network quality of a second level, the sampling ratio is the first sampling ratio; if the network quality is still network quality of the second level during transmission of the GOP sequence, the sampling ratio is adjusted to the second sampling ratio; if the network quality is still network quality of the second level during transmission of the GOP sequence, the sampling ratio is adjusted to the third sampling ratio.

7. The apparatus of claim 6, The application is characterized in that, The dividing module is configured to divide all the P frames in the GOP sequence into the first type of P frames, the second type of P frames, and the third type of P frames based on a position ordering relationship between the P frames in the GOP sequence. For each first type of P frame, if a previous frame of the first type of P frame is not a first type of P frame, the previous frame of the first type of P frame is taken as a reference frame of the first type of P frame; if the previous frame of the first type of P frame is a first type of P frame, a nearest non-first type of P frame in the front is taken as the reference frame of the first type of P frame; for each second type of P frame, if there is a third type of P frame in the front of the second type of P frame, a nearest third type of P frame in the front of the second type of P frame is taken as a reference frame of the second type of P frame; if there is no third type of P frame in the front of the second type of P frame, an I frame of the GOP sequence is taken as the reference frame of the second type of P frame; for each third type of P frame, if there is a third type of P frame in the front of the third type of P frame, a nearest third type of P frame in the front of the third type of P frame is taken as a reference frame of the third type of P frame; if there is no third type of P frame in the front of the third type of P frame, an I frame of the GOP sequence is taken as the reference frame of the third type of P frame. When the image sequence set comprises at least two GOP sequences, for each GOP sequence, if the GOP sequence is not a first GOP sequence of the image sequence set, the dividing module is further configured to take an I frame of the first GOP sequence as a reference frame of a first frame of the GOP sequence. The processing module is further configured to: if the service information includes resolution corresponding to the GOP sequence, frame rate corresponding to the GOP sequence, and decoding capability corresponding to the decoding end, determine whether the P frames in the GOP sequence need to be extracted based on the resolution, the frame rate, and the decoding capability; if yes, determine a sampling ratio corresponding to the GOP sequence based on the resolution, the frame rate, and the decoding capability, the sampling ratio being a first sampling ratio, a second sampling ratio, or a third sampling ratio.

8. An image processing apparatus characterized by comprising: The method comprises: a processor and a machine readable storage medium storing machine executable instructions executable by the processor; the processor is configured to execute the machine executable instructions to implement the method steps of any one of claims 1-5.

Images