Image encoding and decoding method and device

Abstract

The invention provides an image encoding and decoding method and device. The image coding method comprises the following steps: acquiring a preset background frame; acquiring a current frame; a foreground frame corresponding to the current frame is obtained according to the background frame and the current frame, the pixel value of a first area in the foreground frame is 0, the pixel value of a second area in the foreground frame is not 0, the first area is the area with the same pixel value as the background frame of the current frame, and the second area is the area with the same pixel value as the background frame of the current frame; the second region is a region with different pixel values of the current frame and the background frame; and encoding the foreground frame to obtain a first code stream. According to the invention, redundant coding of background information in a video can be effectively reduced, and the size of a code stream is further reduced.

Description

Technical Field

[0001] The present application relates to video coding and decoding technology, and in particular to an image coding and decoding method and device. Background Art

[0002] In security cameras, in order to reduce the cost of transmission and storage, the video will be encoded and compressed. Usually, during encoding, a video can be divided into multiple video segments (group of picture, GOP) of the same length. In each GOP, the I frame is independently encoded using intra-frame prediction, and the P frame is encoded using inter-frame prediction and depends on the encoded frame.

[0003] However, the above encoding method has bit rate redundancy when the video background does not change much. Summary of the invention

[0004] The present application provides an image encoding and decoding method and device, which can effectively reduce the redundant encoding of background information in a video and further reduce the size of the bit stream.

[0005] In a first aspect, the present application provides an image encoding method, comprising: obtaining a preset background frame; obtaining a current frame; obtaining a foreground frame corresponding to the current frame based on the background frame and the current frame, wherein the pixel value of a first area in the foreground frame is 0, and the pixel value of a second area in the foreground frame is not 0, the first area is an area where the current frame and the background frame have the same pixel value, and the second area is an area where the current frame and the background frame have different pixel values; encoding the foreground frame to obtain a first code stream.

[0006] In the embodiment of the present application, by calculating the difference between the current frame and the background frame, the values ​​of the positions where the pixel values ​​of the two are the same can be set to 0, and then the values ​​of the positions where the pixel values ​​are different can be encoded, which can greatly reduce the amount of data. Only encoding the pixel values ​​that are different from the background frame can effectively reduce the redundant encoding of the background information in the video and further reduce the size of the bitstream.

[0007] The background frame is required to include as few foreground objects as possible, such as machines rather than people. For example, in a video shot by a security camera with a fixed viewing angle, image frames without dynamic foreground objects such as machines rather than people can be used as background frames.

[0008] The image content of the background frame is the background targets that remain unchanged for a long time within the camera's field of view, such as walls, curtains, bookshelves, beds, tables, etc. These objects are in fixed positions for a long time. When there are no cameras or non-human objects in the frame, multiple consecutive image frames (for example, all image frames or most image frames of the video) in the video shot by the camera will contain these background targets, and their positions, orientations, etc. are unchanged, and their corresponding pixel values ​​are also unchanged; even when there are cameras or non-human objects in the frame, at least one image frame in the video shot by the camera still contains these background targets at a position where they are not blocked by cameras or non-human objects.

[0009] The reason why non-human targets such as machines are called dynamic foreground targets is that in a GOP or a long video, these targets do not appear in all or most of the image frames, and may only appear in some image frames. Compared with the above-mentioned background targets, non-human targets such as machines only appear briefly in the video. Therefore, non-human targets such as machines can be considered dynamic foreground targets.

[0010] In a possible implementation, the background frame may be specified by the user, who, by watching a video, specifies an image frame that meets the above requirements as the background frame.

[0011] In a possible implementation, the background frame can be determined by a target recognition or target segmentation method. The encoder performs target recognition or target segmentation on the image frames in the video to obtain whether the target contained in the image frames has changed. If the targets contained in multiple consecutive image frames are the same, one of the multiple image frames can be determined as the background frame (for example, the first image frame among the multiple image frames is determined as the background frame).

[0012] It should be noted that the embodiments of the present application may also use other methods to determine the background frame, such as a neural network, etc., so there is no specific limitation on the method for obtaining the background frame.

[0013] In an embodiment of the present application, the background frame can be the first frame of the video currently being processed, or it can be a frame in a video shot within an initial period of time after the camera is turned on, or it can be an image frame in any video shot by the camera. Therefore, the embodiment of the present application does not specifically limit which image frame is used as the background frame.

[0014] Optionally, the encoder may periodically acquire a background frame. Whenever a background frame is updated, subsequent image frames are processed based on the latest background frame.

[0015] The current frame is an image frame to be encoded in the current GOP, and the image frame may be the first frame of the current GOP or a non-first frame of the current GOP. The previous frame and the background frame may or may not be adjacent to each other, and this is not specifically limited.

[0016] The foreground frame can reflect the difference between the current frame and the background frame. For example, if the image content of the current frame includes a foreground object such as a machine but not a person, then the pixel value at the position corresponding to the foreground object such as a machine but not a person in the current frame is different from the pixel value at the same position in the background frame, and the foreground frame can represent the difference in the aforementioned pixel values.

[0017] In a possible implementation, the method for acquiring the foreground frame may include: acquiring a foreground template according to a background frame and a current frame; and then calculating the difference between the current frame and the foreground template to obtain the foreground frame.

[0018] The foreground template can be obtained by the following formula:

[0019]

[0020] Among them, t (x,y) Represents the pixel value at the (x,y) position in the foreground template, e (x,y) Represents the pixel value at the (x,y) position in the background frame, y (x,y) Represents the pixel value at position (x,y) in the current frame.

[0021] That is, taking the position (x, y) in the image frame as an example, the pixel value in the background frame is e (x,y) , whose pixel value in the current frame is y (x,y) , in the foreground template, its pixel value is t (x,y) When the pixel value in the current frame is the same as the pixel value in the background frame, that is, e (x,y) =y (x,y) , the pixel value in the current frame can be assigned to the foreground template; when the pixel value in the current frame is different from the pixel value in the background frame, that is, e (x,y) ≠y (x,y) , the foreground template can be assigned a value of 0. In the obtained foreground template, the pixel value in the first area is not 0, which is taken from the first area of ​​the current frame, and the pixel value in the second area is 0, where the first area is the same as the pixel value of the background frame (i.e., e (x,y) =y (x,y) ) area, and the second area is the area where the pixel values ​​of the current frame and the background frame are different (i.e., e (x,y) ≠y (x,y) ) area.

[0022] The foreground frame can be obtained by the following formula:

[0023] f (x,y) =y (x,y) -t (x,y)

[0024] Among them, f (x,y) Represents the pixel value at the (x,y) position in the foreground frame, y(x,y) Represents the pixel value at the (x,y) position in the current frame, t (x,y) Represents the pixel value at the (x,y) position in the foreground template.

[0025] The foreground frame obtained through the above calculation satisfies the following formula:

[0026]

[0027] Among them, f (x,y) Represents the pixel value at the (x,y) position in the foreground frame, e (x,y) Represents the pixel value at the (x,y) position in the background frame, y (x,y) Represents the pixel value at position (x,y) in the current frame.

[0028] That is, taking the position (x, y) in the image frame as an example, the pixel value in the background frame is e (x,y) , whose pixel value in the current frame is y (x,y) , whose pixel value in the foreground frame is f (x,y) When the pixel value in the current frame is the same as the pixel value in the background frame, that is, e (x,y) =y (x,y) , the foreground frame can be assigned a value of 0; when the pixel value in the current frame is different from the pixel value in the background frame, that is, e (x,y) ≠y (x,y) , the pixel values ​​in the current frame can be assigned to the foreground template. In the obtained foreground frame, the pixel values ​​in the first area are 0, and the pixel values ​​in the second area are not 0, which are taken from the second area of ​​the current frame.

[0029] The above difference calculation is similar to the principle of residual calculation. The pixel values ​​that are the same as the background frame (located in the first area) can be changed to 0, and only the pixel values ​​in the current frame that are different from the background frame (located in the second area) are retained. Such a background frame may contain a large number of zero-valued pixel values, so the amount of data can be greatly reduced in the subsequent encoding of the background frame. Only encoding the foreground frame that is different from the background frame (also called foreground encoding) can effectively reduce the redundant encoding of background information in the video and further reduce the size of the bitstream.

[0030] In a possible implementation, when the current frame is the first frame of the current video segment, the foreground frame is encoded as an I frame to obtain a first bitstream. Alternatively, when the current frame is a non-first frame of the current video segment, the foreground frame is encoded as a P frame to obtain a first bitstream.

[0031] As mentioned above, the current frame can be the first frame of the current video segment (GOP) or a non-first frame of the current GOP. Then according to the IPPP inter-frame reference mode, the first frame of a GOP is encoded as an I frame according to the intra-frame prediction method, and each image frame (non-first frame) after the first frame of the GOP is encoded as a P frame according to the inter-frame prediction method based on the previous image frame. Therefore, the encoding method for the foreground frame can refer to the corresponding current frame. If the current frame is the first frame, the foreground frame is encoded as an I frame. If the current frame is not the first frame, the foreground frame is encoded as a P frame. In this scenario, the image frames in multiple GOPs are respectively based on the background frames to obtain the foreground frames, which has reduced the amount of encoded data (i.e., background information). Therefore, even when the first frame of each GOP is encoded as an I frame, compared with the related technology that encodes the complete image frame as an I frame, the purpose of reducing redundant encoding and improving encoding efficiency has been achieved.

[0032] It should be noted that other encoding methods may be used for the foreground frame, and reference may be made to existing video encoding and decoding standards, which are not specifically limited in the embodiments of the present application.

[0033] In a possible implementation manner, the background frame is encoded as an I frame to obtain a second code stream.

[0034] According to the above description, the background frame is the basis for converting subsequent image frames into foreground frames. Therefore, in order for the decoder to have a basis, the complete image information of the background frame needs to be retained. Therefore, the encoder can encode the background frame as an I frame, and the processing of the background frame needs to be before all image frames that refer to it.

[0035] In a second aspect, the present application provides an image decoding method, comprising: obtaining a code stream; parsing the code stream to obtain a background frame and a current foreground frame; and obtaining a reconstructed frame corresponding to the current foreground frame based on the background frame and the current foreground frame.

[0036] In the embodiment of the present application, the decoded bit stream can obtain a foreground frame and a background frame. The foreground frame only contains the values ​​of the pixel values ​​​​of the image frame and the background frame that are different, and the other positions are all 0, which effectively reduces the redundant encoding of background information and further reduces the bit rate. The separate processing of the background frame and the foreground frame can effectively reduce the computational loss caused by quantization, and the decoder can achieve lossless restoration of the reconstructed frame after encoding.

[0037] Corresponding to the encoder, the decoder can perform I frame decoding on the bitstream to obtain a background frame. The bitstream is decoded by I frame decoding or P frame decoding to obtain a current foreground frame. The method of decoding the bitstream by the decoder can be determined based on the syntax elements carried in the bitstream.

[0038] Corresponding to the encoder, the current foreground frame is obtained after decoding the code stream. The foreground frame can refer to the above description, which reflects the difference between the current frame and the background frame.

[0039] In the foreground frame, the pixel value in the first area is 0, indicating that the pixel values ​​of the image frame and the background frame in the area are the same; the pixel value in the second area is not 0, indicating that the pixel values ​​of the image frame and the background frame in the area are different, and the pixel value of the second area that is not 0 is taken from the image frame at the same position, not the background frame.

[0040] Therefore, when obtaining the reconstructed frame, based on its corresponding foreground frame, in the first area, the pixel value of any position can be taken as the pixel value of the same position in the background frame, and in the second area, the pixel value of any position can be taken as the pixel value of the same position in the foreground frame.

[0041] Based on this, the reconstructed frame satisfies the following formula:

[0042]

[0043] in, represents the pixel value at the (x,y) position in the reconstructed frame, Represents the pixel value at the (x,y) position in the current foreground frame, Represents the pixel value at position (x,y) in the background frame.

[0044] In a third aspect, the present application provides an image encoding device, comprising: an acquisition module, used to acquire a preset background frame; acquire a current frame; a processing module, used to acquire a foreground frame corresponding to the current frame based on the background frame and the current frame, the pixel value of a first area in the foreground frame is 0, and the pixel value of a second area in the foreground frame is not 0, the first area is an area where the current frame and the background frame have the same pixel value, and the second area is an area where the current frame and the background frame have different pixel values; an encoding module, used to encode the foreground frame to obtain a first code stream.

[0045] In a possible implementation, the foreground frame satisfies the following formula:

[0046]

[0047] Among them, f (x,y) represents the pixel value at the (x, y) position in the foreground frame, e (x,y) Represents the pixel value at the (x, y) position in the background frame, y (x,y) Represents the pixel value at the (x, y) position in the current frame.

[0048] In a possible implementation, the processing module is specifically configured to obtain a foreground template according to the background frame and the current frame; and obtain the foreground frame by subtracting the current frame from the foreground template.

[0049] In a possible implementation, the foreground template is obtained by the following formula:

[0050]

[0051] Among them, t (x,y) represents the pixel value at the (x, y) position in the foreground template, e (x,y) Represents the pixel value at the (x, y) position in the background frame, y (x,y) Represents the pixel value at the (x, y) position in the current frame.

[0052] In a possible implementation, the foreground frame is obtained by the following formula:

[0053] f (x,y) =y (x,y) -t (x,y)

[0054] Among them, f (x,y) Represents the pixel value at the (x, y) position in the foreground frame, y (x,y) represents the pixel value at the (x, y) position in the current frame, t (x,y) Represents the pixel value at the (x, y) position in the foreground template.

[0055] In a possible implementation, the encoding module is specifically used to, when the current frame is the first frame of the current video segment, perform I frame encoding on the foreground frame to obtain the first code stream; or, when the current frame is not the first frame of the current video segment, perform P frame encoding on the foreground frame to obtain the first code stream.

[0056] In a possible implementation manner, the encoding module is further configured to perform I-frame encoding on the background frame to obtain a second code stream.

[0057] In a possible implementation manner, the background frame is specified by a user; or, the background frame is determined by a target recognition or target segmentation device.

[0058] In a fourth aspect, the present application provides an image decoding device, comprising: an acquisition module for acquiring a code stream; a decoding module for parsing the code stream to obtain a background frame and a current foreground frame; and a processing module for acquiring a reconstructed frame corresponding to the current foreground frame based on the background frame and the current foreground frame.

[0059] In a possible implementation, the reconstructed frame satisfies the following formula:

[0060]

[0061] in, represents the pixel value at the (x, y) position in the reconstructed frame, represents the pixel value at the (x, y) position in the current foreground frame, Represents the pixel value at the (x, y) position in the background frame.

[0062] In a possible implementation, the decoding module is specifically configured to perform I frame decoding on the code stream to obtain the background frame; and perform I frame decoding or P frame decoding on the code stream to obtain the current foreground frame.

[0063] In a fifth aspect, the present application provides an encoder, comprising: one or more processors; a non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein when the program is executed by the processor, the encoder executes the method described in any one of the first aspects above.

[0064] In a sixth aspect, the present application provides a decoder comprising: one or more processors; a non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein when the program is executed by the processor, the decoder executes the method described in any one of the second aspects above.

[0065] In a seventh aspect, the present application provides a computer-readable storage medium, comprising a program code, which, when executed by a computer device, is used to execute the method according to any one of the first to second aspects above.

[0066] In an eighth aspect, the present application provides a computer-readable storage medium, comprising a bit stream encoded according to the method described in any one of the first aspects above. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1A is an exemplary block diagram of a decoding system 10 according to an embodiment of the present application;

[0068] Figure 1B is an exemplary block diagram of a video decoding system 40 according to an embodiment of the present application;

[0069] Figure 2 is an exemplary block diagram of a video encoder 20 according to an embodiment of the present application;

[0070] Figure 3 is an exemplary block diagram of a video decoder 30 according to an embodiment of the present application;

[0071] Figure 4 is an exemplary block diagram of a video decoding device 400 according to an embodiment of the present application;

[0072] Figure 5a-5c This is a schematic diagram of the commonly used sampling formats for YUV format images;

[0073] Figure 6 A flowchart of a process 600 of the image encoding method of the present application;

[0074] Figure 7 A flowchart of a process 700 of the image decoding method of the present application;

[0075] Figure 8 A flowchart of an image encoding method according to an embodiment of the present application;

[0076] Fig. 9 A schematic diagram of an encoding method according to an embodiment of the present application;

[0077] Fig.10 A schematic diagram of a decoding method according to an embodiment of the present application;

[0078] Fig.11 Schematic diagram of code stream structure;

[0079] Fig.12 Schematic diagram of decoding image;

[0080] Fig.13 13 is a schematic diagram of the structure of an image encoding device 1300 according to an embodiment of the present application;

[0081] Fig.14 14 is a schematic diagram of the structure of an image decoding device 1400 according to an embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0083] The terms "first", "second", etc. in the specification embodiments, claims, and drawings of the present application are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product, or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0084] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0085] Video coding generally refers to processing a sequence of images that form a video or video sequence. In the field of video coding, the terms "picture", "frame" or "image" can be used as synonyms. Video coding (or commonly referred to as coding) includes two parts: video encoding and video decoding. Video coding is performed on the source side and generally includes processing (e.g., compressing) the original video image to reduce the amount of data required to represent the video image (thereby more efficiently storing and / or transmitting). Video decoding is performed on the destination side and generally includes inverse processing relative to the encoder to reconstruct the video image. The "encoding" of the video image (or commonly referred to as the image) involved in the embodiment should be understood as the "encoding" or "decoding" of the video image or video sequence. The encoding part and the decoding part are also collectively referred to as codec (encoding and decoding, CODEC).

[0086] In the case of lossless video coding, the original video image can be reconstructed, that is, the reconstructed video image has the same quality as the original video image (assuming no transmission loss or other data loss during storage or transmission). In the case of lossy video coding, further compression is performed by quantization, etc. to reduce the amount of data required to represent the video image, but the decoder side cannot fully reconstruct the video image, that is, the quality of the reconstructed video image is lower or worse than the quality of the original video image.

[0087] Several video coding standards belong to the category of "lossy hybrid video codecs" (i.e., combining spatial and temporal prediction in the pixel domain with 2D transform coding in the transform domain for applying quantization). Each image in a video sequence is usually divided into a set of non-overlapping blocks, which are usually encoded at the block level. In other words, the encoder usually processes, i.e., encodes the video at the block (video block) level, for example, by generating a prediction block through spatial (intra-frame) prediction and temporal (inter-frame) prediction; subtracting the prediction block from the current block (currently processed / to be processed block) to obtain a residual block; transforming the residual block in the transform domain and quantizing the residual block to reduce the amount of data to be transmitted (compressed), while the decoder side applies the inverse processing part relative to the encoder to the encoded or compressed block to reconstruct the current block for representation. In addition, the encoder needs to repeat the processing steps of the decoder so that the encoder and the decoder generate the same predictions (e.g., intra-frame predictions and inter-frame predictions) and / or reconstructed pixels for processing, i.e., encoding subsequent blocks.

[0088] In the following embodiment of the decoding system 10, the encoder 20 and the decoder 30 are based on Figures 1A to 3 Give a description.

[0089] Figure 1A 1 is an exemplary block diagram of a decoding system 10 of an embodiment of the present application, for example, a video decoding system 10 (or simply a decoding system 10) that can utilize the techniques of the present application. The video encoder 20 (or simply an encoder 20) and the video decoder 30 (or simply a decoder 30) in the video decoding system 10 represent devices that can be used to perform various techniques according to various examples described in the present application.

[0090] like Figure 1A As shown, the decoding system 10 includes a source device 12, which is used to provide encoded image data 21 such as encoded images to a destination device 14 for decoding the encoded image data 21.

[0091] The source device 12 includes an encoder 20 , and may additionally or optionally include an image source 16 , a preprocessor (or a preprocessing unit) 18 such as an image preprocessor, and a communication interface (or a communication unit) 22 .

[0092] The image source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage that stores any of the above images.

[0093] In order to distinguish the processing performed by the pre-processor (or pre-processing unit) 18 , the image (or image data) 17 may also be referred to as a raw image (or raw image data) 17 .

[0094] The preprocessor 18 is used to receive the original image data 17 and preprocess the original image data 17 to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color adjustment, or denoising. It is understood that the preprocessing unit 18 may be an optional component.

[0095] The video encoder (or encoder) 20 is used to receive the pre-processed image data 19 and provide the encoded image data 21 (hereinafter referred to as Figure 2 etc. for further description).

[0096] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device through the communication channel 13 for storage or direct reconstruction.

[0097] The destination device 14 includes a decoder 30 and, in addition or optionally, may include a communication interface (or communication unit) 28 , a post-processor (or post-processing unit) 32 , and a display device 34 .

[0098] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is a encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0099] The communication interface 22 and the communication interface 28 can be used to send or receive encoded image data (or encoded data) 21 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any type of combination thereof.

[0100] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0101] The communication interface 28 corresponds to the communication interface 22 , for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21 .

[0102] Both the communication interface 22 and the communication interface 28 can be configured as follows Figure 1A A unidirectional communication interface, or a bidirectional communication interface, indicated by an arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0103] The video decoder (or decoder) 30 is used to receive the encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as Figure 3 etc. for further description).

[0104] The post-processor 32 is used to post-process the decoded image data 31 (also called reconstructed image data) such as the decoded image to obtain the post-processed image data 33 such as the post-processed image. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, cropping or resampling, or any other processing for generating the decoded image data 31 for display by the display device 34 or the like.

[0105] The display device 34 is used to receive the post-processed image data 33 to display the image to a user or viewer, etc. The display device 34 can be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or display. For example, the display screen can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0106] although Figure 1AThe source device 12 and the destination device 14 are shown as independent devices, but the device embodiment may also include the source device 12 and the destination device 14 or the functions of the source device 12 and the destination device 14 at the same time, that is, the source device 12 or the corresponding function and the destination device 14 or the corresponding function at the same time. In these embodiments, the source device 12 or the corresponding function and the destination device 14 or the corresponding function can be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0107] According to the description, Figure 1A The presence and (exact) division of different units or functions in the source device 12 and / or the destination device 14 shown may vary depending on the actual device and application, which will be obvious to the skilled person.

[0108] The encoder 20 (eg, video encoder 20) or the decoder 30 (eg, video decoder 30) or both may be configured as follows: Figure 1B The processing circuitry shown may be implemented, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding dedicated processors, or any combination thereof. The encoder 20 may be implemented by processing circuitry 46 to include reference Figure 2 The various modules discussed in encoder 20 and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented by processing circuit 46 to include reference Figure 3 The processing circuit 46 may be used to perform the various operations discussed below. Figure 4 As shown, if part of the technology is implemented in software, the device can store the instructions of the software in a suitable computer-readable storage medium and use one or more processors to execute the instructions in hardware to perform the technology of the present invention. One of the video encoder 20 and the video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as Figure 1B shown.

[0109] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or a content distribution server), a broadcast receiving device, a broadcast transmitting device, etc., and may not use or use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0110] In some cases, Figure 1A The video decoding system 10 shown is merely exemplary, and the techniques provided herein may be applicable to video encoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data is retrieved from a local memory, sent over a network, and so on. A video encoding device may encode data and store the data in a memory, and / or a video decoding device may retrieve data from a memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to a memory and / or retrieve and decode data from a memory.

[0111] Figure 1B FIG. 4 is an exemplary block diagram of a video decoding system 40 according to an embodiment of the present application. Figure 1B As shown, the video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by a processing circuit 46), an antenna 42, one or more processors 43, one or more memory storages 44 and / or a display device 45.

[0112] like Figure 1B As shown, imaging device 41, antenna 42, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory storage 44 and / or display device 45 can communicate with each other. In different examples, video decoding system 40 can include only video encoder 20 or only video decoder 30.

[0113] In some instances, antenna 42 can be used to transmit or receive a coded bit stream of video data. In addition, in some instances, display device 45 can be used to present video data. Processing circuit 46 can include application-specific integrated circuit (ASIC) logic, graphics processor, general processor, etc. Video decoding system 40 can also include optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, graphics processor, general processor, etc. In addition, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory storage 44 can be implemented by cache memory. In other instances, processing circuit 46 can include memory (e.g., cache, etc.) for implementing image buffer, etc.

[0114] In some examples, video encoder 20 implemented by logic circuits may include an image buffer (e.g., implemented by processing circuit 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented by processing circuit 46 to implement reference Figure 2 The various modules discussed in connection with the encoder and / or any other encoder system or subsystem described herein. Logic circuits may be used to perform the various operations discussed herein.

[0115] In some examples, video decoder 30 may be implemented in a similar manner by processing circuitry 46 to implement reference Figure 3 The various modules discussed in connection with the video decoder 30 of the present invention and / or any other decoder system or subsystem described herein. In some examples, the video decoder 30 implemented by logic circuitry may include an image buffer (implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (implemented, for example, by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented by processing circuitry 46 to implement the video decoder 30 of the present invention. Figure 3 and / or the various modules discussed above for any other decoder system or subsystem described herein.

[0116] In some examples, antenna 42 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data related to the encoded video frames discussed herein, indicators, index values, mode selection data, etc., such as data related to the encoded partitions (e.g., transform coefficients or quantized transform coefficients, (as discussed) optional indicators, and / or data defining the encoded partitions). Video decoding system 40 may also include video decoder 30 coupled to antenna 42 and used to decode the encoded bitstream. Display device 45 is used to present the video frames.

[0117] It should be understood that for the examples described with reference to the video encoder 20 in the embodiments of the present application, the video decoder 30 can be used to perform the reverse process. With respect to the signaling syntax elements, the video decoder 30 can be used to receive and parse such syntax elements and decode the related video data accordingly. In some examples, the video encoder 20 can entropy encode the syntax elements into an encoded video bitstream. In such examples, the video decoder 30 can parse such syntax elements and decode the related video data accordingly.

[0118] For ease of description, the embodiments of the present invention are described with reference to the universal video coding (VVC) reference software or the high-efficiency video coding (HEVC) developed by the video coding joint working group (joint collaboration team on video coding, JCT-VC) of the ITU-T video coding experts group (VCEG) and the ISO / IEC motion picture experts group (MPEG). Those skilled in the art understand that the embodiments of the present invention are not limited to HEVC or VVC.

[0119] Encoders and encoding methods

[0120] Figure 2 FIG. 2 is an exemplary block diagram of a video encoder 20 according to an embodiment of the present application. Figure 2As shown, the video encoder 20 includes an input terminal (or input interface) 201, a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270 and an output terminal (or output interface) 272. The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254 and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 2 The illustrated video encoder 20 may also be referred to as a hybrid video encoder or a hybrid video codec-based video encoder.

[0121] The residual calculation unit 204, the transform processing unit 206, the quantization unit 208 and the mode selection unit 260 constitute the forward signal path of the encoder 20, while the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 constitute the backward signal path of the encoder, wherein the backward signal path of the encoder 20 corresponds to the signal path of the decoder (see Figure 3 The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded image buffer 230, the inter-frame prediction unit 244 and the intra-frame prediction unit 254 also constitute a “built-in decoder” of the video encoder 20.

[0122] Images and image segmentation (images and blocks)

[0123] The encoder 20 may be configured to receive, via an input 201 or the like, an image (or image data) 17, e.g., an image in a sequence of images forming a video or a video sequence. The received image or image data may also be a pre-processed image (or pre-processed image data) 19. For simplicity, the following description uses image 17. Image 17 may also be referred to as a current image or an image to be encoded (particularly when distinguishing the current image from other images in video encoding, such as previously encoded and / or decoded images in the same video sequence, i.e., a video sequence also including the current image).

[0124] A (digital) image is or can be considered as a two-dimensional array or matrix of pixels with intensity values. The pixels in the array can also be called pixels (pixel or pel) (short for picture elements). The number of pixels in the array or image in the horizontal and vertical directions (or axes) determines the size and / or resolution of the image. In order to represent color, three color components are usually used, that is, the image can be represented as or include three pixel arrays. In the RBG format or color space, the image includes corresponding red, green and blue pixel arrays. However, in video coding, each pixel is usually represented in a brightness / chrominance format or color space, such as YCbCr, including a brightness component indicated by Y (sometimes also represented by L) and two chrominance components represented by Cb and Cr. The brightness (luma) component Y represents the brightness or grayscale level intensity (for example, the two are the same in grayscale images), while the two chrominance (abbreviated as chroma) components Cb and Cr represent the chrominance or color information components. Accordingly, an image in YCbCr format includes a luminance pixel array of luminance pixel values ​​(Y) and two chrominance pixel arrays of chrominance values ​​(Cb and Cr). An image in RGB format can be converted or transformed into YCbCr format and vice versa, a process also referred to as color conversion or transformation. If the image is black and white, the image may include only a luminance pixel array. Accordingly, an image may be, for example, a luminance pixel array in a monochrome format or a luminance pixel array and two corresponding chrominance pixel arrays in 4:2:0, 4:2:2 and 4:4:4 color formats.

[0125] In one embodiment, an embodiment of the video encoder 20 may include an image segmentation unit ( Figure 2 ), for dividing the image 17 into a plurality of (usually non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC) or coding tree blocks (CTB), or coding tree units (CTU) in the H.265 / HEVC and VVC standards. The partitioning unit may be used to use the same block size for all images in a video sequence and a corresponding grid of defined block sizes, or to change the block size between images or subsets of images or groups of images, and to partition each image into corresponding blocks.

[0126] In other embodiments, the video encoder may be configured to directly receive a block 203 of the image 17, for example, one, several or all blocks constituting the image 17. The image block 203 may also be referred to as a current image block or an image block to be encoded.

[0127] Like image 17, image block 203 is also or can be considered as a two-dimensional array or matrix composed of pixels with intensity values ​​(pixel values), but image block 203 is smaller than image 17. In other words, block 203 may include one pixel array (e.g., a brightness array in the case of monochrome image 17 or a brightness array or chrominance array in the case of a color image) or three pixel arrays (e.g., one brightness array and two chrominance arrays in the case of a color image 17) or any other number and / or type of arrays according to the color format adopted. The number of pixels in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Accordingly, the block can be an M×N (M columns×N rows) pixel array, or an M×N transform coefficient array, etc.

[0128] In one embodiment, Figure 2 The video encoder 20 shown is used to encode the image 17 block by block, for example, encoding and prediction are performed on each block 203.

[0129] In one embodiment, Figure 2 The illustrated video encoder 20 may also be used to partition and / or encode an image using slices (also referred to as video slices), wherein an image may be partitioned or encoded using one or more slices (usually non-overlapping). Each slice may include one or more blocks (e.g., coding tree units CTUs) or one or more groups of blocks (e.g., coding tiles in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0130] In one embodiment, Figure 2 The video encoder 20 shown can also be used to partition and / or encode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), wherein an image can be partitioned or encoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTU) or one or more coding blocks, etc., wherein each coding block may be in a rectangular shape, etc., and may include one or more complete or partial blocks (e.g., CTU).

[0131] Residual calculation

[0132] The residual calculation unit 204 is used to calculate the residual block 205 (the prediction block 265 is described in detail later) based on the image block 203 and the prediction block 265 in the following manner: for example, the pixel value of the prediction block 265 is subtracted from the pixel value of the image block 203 pixel by pixel (pixel by pixel) to obtain the residual block 205 in the pixel domain.

[0133] Transform

[0134] The transform processing unit 206 is used to perform discrete cosine transform (DCT) or discrete sine transform (DST) on the pixel values ​​of the residual block 205 to obtain transform coefficients 207 in the transform domain. The transform coefficients 207 may also be called transform residual coefficients, representing the residual block 205 in the transform domain.

[0135] The transform processing unit 206 may be used to apply an integerized approximation of the DCT / DST, such as the transform specified for H.265 / HEVC. This integerized approximation is typically scaled by a factor compared to an orthogonal DCT transform. In order to maintain the norm of the residual block processed by the forward transform and the inverse transform, other scaling factors are used as part of the transform process. The scaling factor is typically selected based on certain constraints, such as whether the scaling factor is a power of 2 for the shift operation, the bit depth of the transform coefficients, a trade-off between accuracy and implementation cost, etc. For example, a specific scaling factor is specified for the inverse transform on the encoder 20 side by the inverse transform processing unit 212 (and for the corresponding inverse transform on the decoder 30 side by, for example, the inverse transform processing unit 312), and correspondingly, a corresponding scaling factor may be specified for the forward transform on the encoder 20 side by the transform processing unit 206.

[0136] In one embodiment, the video encoder 20 (correspondingly, the transform processing unit 206) may be used to output transform parameters such as one or more transform types, for example, directly output or output after encoding or compression by the entropy coding unit 270, for example, so that the video decoder 30 may receive and use the transform parameters for decoding.

[0137] Quantification

[0138] The quantization unit 208 is used to quantize the transform coefficient 207 by, for example, scalar quantization or vector quantization to obtain a quantized transform coefficient 209 . The quantized transform coefficient 209 may also be referred to as a quantized residual coefficient 209 .

[0139] The quantization process may reduce the bit depth associated with some or all of the transform coefficients 207. For example, an n-bit transform coefficient may be rounded down to an m-bit transform coefficient during quantization, where n is greater than m. The degree of quantization may be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, different degrees of scaling may be applied to achieve finer or coarser quantization. A smaller quantization step size corresponds to a finer quantization, while a larger quantization step size corresponds to a coarser quantization. A suitable quantization step size may be indicated by a quantization parameter (QP). For example, a quantization parameter may be an index of a predefined set of suitable quantization step sizes. For example, a smaller quantization parameter may correspond to fine quantization (a smaller quantization step size), a larger quantization parameter may correspond to coarse quantization (a larger quantization step size), and vice versa. Quantization may include dividing by a quantization step size, while a corresponding or inverse dequantization performed by an inverse quantization unit 210 or the like may include multiplying by a quantization step size. Embodiments according to some standards, such as HEVC, may be used to determine a quantization step size using a quantization parameter. In general, the quantization step size may be calculated using a fixed-point approximation of an equation containing division according to the quantization parameter. Other scaling factors may be introduced for quantization and dequantization to recover the norm of the residual block that may be modified due to the scaling used in the fixed-point approximation of the equations for the quantization step size and the quantization parameter. In one exemplary implementation, the scaling of the inverse transform and dequantization may be combined. Alternatively, a custom quantization table may be used and indicated from the encoder to the decoder in the bitstream, etc. Quantization is a lossy operation, where the larger the quantization step size, the greater the loss.

[0140] In one embodiment, the video encoder 20 (correspondingly, the quantization unit 208) may be configured to output a quantization parameter (QP), for example, directly output or output after being encoded or compressed by the entropy coding unit 270, for example, so that the video decoder 30 may receive and use the quantization parameter for decoding.

[0141] Dequantization

[0142] The inverse quantization unit 210 is used to perform inverse quantization of the quantization unit 208 on the quantized coefficients to obtain dequantized coefficients 211, for example, performing an inverse quantization scheme of the quantization scheme performed by the quantization unit 208 according to or using the same quantization step size as the quantization unit 208. The dequantized coefficients 211 may also be referred to as dequantized residual coefficients 211, corresponding to the transform coefficients 207, but due to the loss caused by quantization, the dequantized coefficients 211 are usually not completely the same as the transform coefficients.

[0143] Inverse Transform

[0144] The inverse transform processing unit 212 is used to perform an inverse transform of the transform performed by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), to obtain a reconstructed residual block 213 (or a corresponding dequantized coefficient 213) in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 213.

[0145] reconstruction

[0146] The reconstruction unit 214 (e.g., the summer 214) is used to add the transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265 to obtain the reconstructed block 215 in the pixel domain, for example, by adding the pixel point values ​​of the reconstructed residual block 213 and the pixel point values ​​of the prediction block 265.

[0147] Filtering

[0148] The loop filter unit 220 (or simply "loop filter" 220) is used to filter the reconstruction block 215 to obtain the filter block 221, or is generally used to filter the reconstructed pixel points to obtain the filtered pixel point value. For example, the loop filter unit is used to smoothly perform pixel conversion or improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process may also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Figure 2 2. Although shown as a loop filter in FIG. 2, in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filter block 221 may also be referred to as a filter reconstruction block 221.

[0149] In one embodiment, the video encoder 20 (correspondingly, the loop filter unit 220) may be used to output loop filter parameters (e.g., SAO filter parameters, ALF filter parameters, or LMCS parameters), for example, directly or after being entropy encoded by the entropy encoding unit 270, such that the decoder 30 may receive and use the same or different loop filter parameters for decoding.

[0150] Decoded Image Buffer

[0151] The decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference picture data for use by the video encoder 20 when encoding video data. The DPB 230 may be formed by any of a variety of memory devices, such as a dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of storage devices. The decoded picture buffer 230 may be used to store one or more filter blocks 221. The decoded picture buffer 230 may also be used to store other previous filter blocks of the same current picture or a different picture such as a previous reconstructed block, such as a previously reconstructed and filtered block 221, and may provide a complete previously reconstructed, i.e., decoded picture (and corresponding reference blocks and pixels) and / or a partially reconstructed current picture (and corresponding reference blocks and pixels), such as for inter-frame prediction. The decoded image buffer 230 may also be used to store one or more unfiltered reconstructed blocks 215, or generally to store unfiltered reconstructed pixels, for example, reconstructed blocks 215 that have not been filtered by the loop filter unit 220, or reconstructed blocks or reconstructed pixels that have not undergone any other processing.

[0152] Mode selection (segmentation and prediction)

[0153] The mode selection unit 260 includes a segmentation unit 262, an inter-frame prediction unit 244, and an intra-frame prediction unit 254, which are used to receive or obtain original image data such as the original block 203 (the current block 203 of the current image 17) and reconstructed block data from the decoded image buffer 230 or other buffers (e.g., column buffers, not shown in the figure), such as filtered and / or unfiltered reconstructed pixels or reconstructed blocks of the same (current) image and / or one or more previously decoded images. The reconstructed block data is used as reference image data required for prediction such as inter-frame prediction or intra-frame prediction to obtain a prediction block 265 or a prediction value 265.

[0154] The mode selection unit 260 may be used to determine or select a partitioning mode (eg, intra-frame or inter-frame prediction mode) for the current block (including no partitioning) and generate a corresponding prediction block 265 to calculate the residual block 205 and reconstruct the reconstruction block 215 .

[0155] In one embodiment, the mode selection unit 260 may be used to select a segmentation and prediction mode (e.g., from prediction modes supported or available by the mode selection unit 260) that provides the best match or minimum residual (minimum residual means better compression in transmission or storage), or provides minimum signaling overhead (minimum signaling overhead means better compression in transmission or storage), or considers or balances both of the above. The mode selection unit 260 may be used to determine the segmentation and prediction mode based on rate distortion optimization (RDO), i.e., select the prediction mode that provides minimum rate distortion optimization. The terms "best", "lowest", "optimal", etc. herein do not necessarily mean "best", "lowest", "optimal" in general, but may also refer to situations where termination or selection criteria are met, for example, values ​​exceeding or falling below a threshold or other restrictions may result in a "suboptimal selection" but reduce complexity and processing time.

[0156] In other words, the partitioning unit 262 may be used to partition an image in a video sequence into a sequence of coding tree units (CTUs), and the CTU 203 may be further partitioned into smaller block portions or sub-blocks (again forming blocks), for example, by iteratively using quad-tree partitioning (QT), binary-tree partitioning (BT) or triple-tree partitioning (TT) or any combination thereof, and for performing prediction on each of the block portions or sub-blocks, for example, wherein the mode selection comprises selecting a tree structure of the partition block 203 and selecting a prediction mode to be applied to each of the block portions or sub-blocks.

[0157] The segmentation (eg, performed by segmentation unit 262) and prediction processes (eg, performed by inter-prediction unit 244 and intra-prediction unit 254) performed by video encoder 20 are described in detail below.

[0158] segmentation

[0159] The partitioning unit 262 may partition (or divide) a coding tree unit 203 into smaller parts, such as small blocks of square or rectangular shapes. For an image with three pixel arrays, a CTU consists of N×N luminance pixel blocks and two corresponding chrominance pixel blocks.

[0160] The H.265 / HEVC video coding standard divides a frame into non-overlapping CTUs, and the size of the CTU can be set to 64×64 (the size of the CTU can also be set to other values, such as the CTU size in the JVET reference software JEM is increased to 128×128 or 256×256). The 64×64 CTU contains a rectangular pixel matrix with 64 columns and 64 pixels in each column, and each pixel contains a luminance component and / or a chrominance component.

[0161] H.265 uses a QT-based CTU partitioning method, taking CTU as the root node of QT. According to the QT partitioning method, CTU is recursively divided into several leaf nodes. A node corresponds to an image area. If the node is not divided, the node is called a leaf node, and the image area corresponding to it is a CU. If the node continues to be divided, the image area corresponding to the node can be divided into four areas of the same size (the length and width are half of the divided area), each area corresponds to a node, and it is necessary to determine whether these nodes will be divided. Whether a node is divided is indicated by the split flag split_cu_flag corresponding to the node in the bitstream. A node A is divided once to obtain 4 nodes Bi, i=0~3, Bi is called the child node of A, and A is called the parent node of Bi. The QT level (qtDepth) of the root node is 0, and the QT level of the node is the four QT levels of the parent node of the node plus 1.

[0162] In the H.265 / HEVC standard, for images in YUV4:2:0 format, a CTU contains a luma block and two chroma blocks. The luma block and the chroma block can be divided in the same way, which is called a luma-chroma joint coding tree. In VVC, if the current frame is an I frame, when a CTU is a node of a preset size (such as 64×64) in an intra-frame coded frame (I frame), the luma block contained in the node is divided into a group of coding units containing only luma blocks through the luma coding tree, and the chroma block contained in the node is divided into a group of coding units containing only chroma blocks through the chroma coding tree; the division of the luma coding tree and the chroma coding tree are independent of each other. This kind of luma block and chroma block use independent coding trees, which are called separate trees. In H.265, CU contains luma pixels and chroma pixels; in H.266, AVS3 and other standards, in addition to CUs containing both luma pixels and chroma pixels, there are also luma CUs containing only luma pixels and chroma CUs containing only chroma pixels.

[0163] As described above, the video encoder 20 is operable to determine or select the best or optimal prediction mode from a (predetermined) set of prediction modes. The set of prediction modes may include, for example, an intra prediction mode and / or an inter prediction mode.

[0164] Intra-frame prediction

[0165] The intra prediction mode set may include 35 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and plane mode, or directional modes as defined in HEVC, or may include 67 different intra prediction modes, for example, non-directional modes like DC (or mean) mode and plane mode, or directional modes as defined in VVC. For example, several traditional angle intra prediction modes are adaptively replaced with wide-angle intra prediction modes for non-square blocks defined in VVC. For another example, in order to avoid the division operation of DC prediction, only the longer side is used to calculate the average value of non-square blocks. In addition, the intra prediction result of the plane mode can also be modified using the position-dependent intra prediction combination (PDPC) method.

[0166] The intra prediction unit 254 is configured to generate an intra prediction block 265 using reconstructed pixels of adjacent blocks of the same current image according to an intra prediction mode in the intra prediction mode set.

[0167] The intra-frame prediction unit 254 (or generally the mode selection unit 260) is also used to output intra-frame prediction parameters (or generally information indicating the selected intra-frame prediction mode of the block) in the form of syntax elements 266 to the entropy coding unit 270 for inclusion in the encoded image data 21, so that the video decoder 30 can perform operations, such as receiving and using the prediction parameters for decoding.

[0168] Inter prediction

[0169] In a possible implementation, the set of inter-prediction modes depends on the available reference picture (i.e., at least part of the previously decoded picture stored in the DBP 230 as mentioned above) and other inter-prediction parameters, for example, on whether to use the entire reference picture or only a part of the reference picture, such as a search window area around the area of ​​the current block, to search for the best matching reference block, and / or on whether to perform pixel interpolation such as half-pixel, quarter-pixel and / or 1 / 16 interpolation, for example.

[0170] In addition to the above prediction modes, skip mode and / or direct mode may also be employed.

[0171] For example, extended merge prediction, the merge candidate list of this mode consists of the following five candidate types in order: spatial MVP from spatially adjacent CUs, temporal MVP from collocated CUs, history-based MVP from FIFO tables, pairwise average MVP, and zero MV. Decoder side motion vector refinement (DMVR) based on bilateral matching can be used to increase the accuracy of the MV of the merge mode. Merge mode with MVD (MMVD) is derived from the merge mode with motion vector difference. The MMVD flag is sent immediately after the skip flag and merge flag are sent to specify whether the CU uses the MMVD mode. The CU-level adaptive motion vector resolution (AMVR) scheme can be used. AMVR supports encoding the MVD of the CU with different precisions. The MVD of the current CU is adaptively selected according to the prediction mode of the current CU. When the CU is encoded in merge mode, the combined inter / intra prediction (CIIP) mode can be applied to the current CU. The CIIP prediction is obtained by weighted averaging the inter-frame and intra-frame prediction signals. For affine motion compensated prediction, the affine motion field of the block is described by the motion information of 2 control points (4 parameters) or 3 control points (6 parameters) motion vectors. Subblock-based temporal motion vector prediction (SbTMVP) is similar to the temporal motion vector prediction (TMVP) in HEVC, but the motion vector of the sub-CU within the current CU is predicted. Bidirectional optical flow (BDOF), formerly known as BIO, is a simplified version that reduces calculations, especially in terms of the number of multiplications and the size of the multipliers. In the triangle partitioning mode, the CU is evenly divided into two triangular parts in two ways: diagonal partitioning and anti-diagonal partitioning. In addition, the bidirectional prediction mode is extended on the basis of simple averaging to support weighted averaging of two prediction signals.

[0172] The inter-frame prediction unit 244 may include a motion estimation (ME) unit and a motion compensation (MC) unit (both in Figure 2). The motion estimation unit may be configured to receive or acquire the image block 203 (the current image block 203 of the current image 17) and the decoded image 231, or at least one or more previously reconstructed blocks, for example, reconstructed blocks of one or more other / different previously decoded images 231, for motion estimation. For example, a video sequence may include the current image and the previously decoded image 231, or in other words, the current image and the previously decoded image 231 may be part of or form a sequence of images forming the video sequence.

[0173] For example, the encoder 20 may be configured to select a reference block from a plurality of reference blocks of the same or different images in a plurality of other images, and provide the reference image (or reference image index) and / or the offset (spatial offset) between the position (x, y coordinates) of the reference block and the position of the current block as an inter-frame prediction parameter to the motion estimation unit. The offset is also referred to as a motion vector (MV).

[0174] The motion compensation unit is used to obtain, for example, receive, inter-frame prediction parameters, and perform inter-frame prediction based on or using the inter-frame prediction parameters to obtain an inter-frame prediction block 246. The motion compensation performed by the motion compensation unit may include extracting or generating a prediction block based on a motion / block vector determined by motion estimation, and may also include performing interpolation with sub-pixel accuracy. Interpolation filtering can generate pixel points of other pixels from pixel points of known pixels, thereby potentially increasing the number of candidate prediction blocks that can be used to encode the image block. Once a motion vector corresponding to a PU of a current image block is received, the motion compensation unit can locate the prediction block pointed to by the motion vector in one of the reference image lists.

[0175] The motion compensation unit may also generate syntax elements associated with blocks and video slices for use by the video decoder 30 when decoding image blocks of the video slice. In addition, or as an alternative to slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be generated or used.

[0176] Entropy Coding

[0177] The entropy coding unit 270 is used to apply an entropy coding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context adaptive VLC scheme (CALVC), an arithmetic coding scheme, a binarization algorithm, a context adaptive binary arithmetic coding (CABAC), a syntax-based context-adaptive binary arithmetic coding (SBAC), a probability interval partitioning entropy (PIPE) coding, or other entropy coding methods or techniques) to the quantized residual coefficients 209, inter-frame prediction parameters, intra-frame prediction parameters, loop filter parameters, and / or other syntax elements to obtain coded image data 21 that can be output in the form of a coded bit stream 21 through an output terminal 272, so that the video decoder 30, etc. can receive and use the parameters for decoding. The coded bit stream 21 can be transmitted to the video decoder 30, or it can be stored in a memory for later transmission or retrieval by the video decoder 30.

[0178] Other structural variations of the video encoder 20 may be used to encode the video stream. For example, a non-transform based encoder 20 may directly quantize the residual signal without a transform processing unit 206 for certain blocks or frames. In another implementation, the encoder 20 may have a quantization unit 208 and an inverse quantization unit 210 combined into a single unit.

[0179] Decoder and decoding method

[0180] Figure 3 3 is an exemplary block diagram of a video decoder 30 according to an embodiment of the present application. The video decoder 30 is used to receive, for example, coded image data 21 (for example, coded bitstream 21) encoded by an encoder 20, and obtain a decoded image 331. The coded image data or bitstream includes information for decoding the coded image data, such as data representing image blocks of a coded video slice (and / or a coding block group or coding block) and related syntax elements.

[0181] exist Figure 3In the example of FIG. 3 , decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., a summer 314), a loop filter 320, a decoded picture buffer (DBP) 330, a mode application unit 360, an inter-frame prediction unit 344, and an intra-frame prediction unit 354. Inter-frame prediction unit 344 may be or include a motion compensation unit. In some examples, video decoder 30 may perform substantially the same as reference Figure 2 The decoding process of the video encoder 100 is the reverse of the encoding process described above.

[0182] As described for encoder 20, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer DPB 230, inter prediction unit 344, and intra prediction unit 354 also constitute a “built-in decoder” of video encoder 20. Accordingly, inverse quantization unit 310 may be functionally identical to inverse quantization unit 110, inverse transform processing unit 312 may be functionally identical to inverse transform processing unit 122, reconstruction unit 314 may be functionally identical to reconstruction unit 214, loop filter 320 may be functionally identical to loop filter 220, and decoded picture buffer 330 may be functionally identical to decoded picture buffer 230. Therefore, the explanation of the corresponding units and functions of video encoder 20 applies accordingly to the corresponding units and functions of video decoder 30.

[0183] Entropy decoding

[0184] The entropy decoding unit 304 is used to parse the bit stream 21 (or generally the encoded image data 21) and perform entropy decoding on the encoded image data 21 to obtain the quantization coefficients 309 and / or the decoded encoding parameters ( Figure 3 The entropy decoding unit 304 may be used to provide the inter-frame prediction parameters, intra-frame prediction parameters and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 may receive syntax elements at the video slice and / or video block level. In addition, or as an alternative to slices and corresponding syntax elements, coding block groups and / or coding blocks and corresponding syntax elements may be received or used.

[0185] Dequantization

[0186] The inverse quantization unit 310 may be configured to receive a quantization parameter (QP) (or generally information related to inverse quantization) and a quantization coefficient from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304), and inverse quantize the decoded quantization coefficient 309 based on the quantization parameter to obtain an inverse quantization coefficient 311, which may also be referred to as a transform coefficient 311. The inverse quantization process may include using the quantization parameter calculated by the video encoder 20 for each video block in the video slice to determine a degree of quantization, and also determine a degree of inverse quantization that needs to be performed.

[0187] Inverse Transform

[0188] The inverse transform processing unit 312 may be configured to receive the dequantized coefficients 311, also referred to as transform coefficients 311, and apply a transform to the dequantized coefficients 311 to obtain a reconstructed residual block 213 in the pixel domain. The reconstructed residual block 213 may also be referred to as a transform block 313. The transform may be an inverse transform, such as an inverse DCT, an inverse DST, an inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 may also be configured to receive transform parameters or corresponding information from the encoded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304) to determine the transform applied to the dequantized coefficients 311.

[0189] reconstruction

[0190] The reconstruction unit 314 (eg, the summer 314 ) is used to add the reconstructed residual block 313 to the prediction block 365 to obtain the reconstructed block 315 in the pixel domain, for example, by adding the pixel point values ​​of the reconstructed residual block 313 and the pixel point values ​​of the prediction block 365 .

[0191] Filtering

[0192] The loop filter unit 320 (in or after the encoding loop) is used to filter the reconstruction block 315 to obtain a filter block 321, so as to smoothly perform pixel conversion or improve video quality, etc. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample-adaptive offset (SAO) filter, or one or more other filters, such as an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination. For example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be a deblocking filter, an SAO filter, and an ALF filter. For another example, a process called luma mapping with chromascaling (LMCS) (i.e., an adaptive in-loop shaper) is added. This process is performed before deblocking. For another example, the deblocking filtering process may also be applied to internal sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra sub-partition (ISP) edges. Figure 3 3. Although shown as a loop filter in FIG. 3, in other configurations, the loop filter unit 320 may be implemented as a post-loop filter.

[0193] Decoded Image Buffer

[0194] The decoded video blocks 321 in one picture are then stored in a decoded picture buffer 330 which stores the decoded pictures 331 as reference pictures for subsequent motion compensation of other pictures and / or for respective output display.

[0195] The decoder 30 is used to output the decoded image 311 through the output terminal 312, etc., for display to the user or for the user to view.

[0196] predict

[0197] The inter-frame prediction unit 344 may be functionally the same as the inter-frame prediction unit 244 (particularly the motion compensation unit), and the intra-frame prediction unit 354 may be functionally the same as the inter-frame prediction unit 254, and may determine the division or segmentation and perform prediction based on the segmentation and / or prediction parameters or corresponding information received from the coded image data 21 (e.g., parsed and / or decoded by the entropy decoding unit 304). The mode application unit 360 may be used to perform prediction (intra-frame or inter-frame prediction) of each block according to the reconstructed block, block or corresponding pixel point (filtered or unfiltered), and obtain a prediction block 365.

[0198] When the video slice is encoded as an intra coded (I) slice, the intra prediction unit 354 in the mode application unit 360 is used to generate a prediction block 365 for the image block of the current video slice based on the indicated intra prediction mode and data from the previously decoded block of the current image. When the video image is encoded as an inter coded (i.e., B or P) slice, the inter prediction unit 344 (e.g., motion compensation unit) in the mode application unit 360 is used to generate a prediction block 365 for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 304. For inter prediction, these prediction blocks can be generated from one of the reference images in one of the reference image lists. The video decoder 30 can construct reference frame list 0 and list 1 using a default construction technique based on the reference images stored in the DPB 330. The same or similar processes may be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks) in addition to or as an alternative to slices (e.g., video slices), for example, a video may be encoded using I, P or B coding block groups and / or coding blocks.

[0199] The mode application unit 360 is used to determine prediction information for a video block of a current video slice by parsing motion vectors and other syntax elements, and use the prediction information to generate a prediction block for the current video block being decoded. For example, the mode application unit 360 uses some of the received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction) for encoding the video block of the video slice, the inter-frame prediction slice type (e.g., B slice, P slice, or GPB slice), the construction information of one or more reference picture lists for the slice, the motion vector for each inter-frame coded video block of the slice, the inter-frame prediction state for each inter-frame coded video block of the slice, and other information to decode the video block in the current video slice. In addition to slices (e.g., video slices) or as an alternative to slices, the same or similar process can be applied to embodiments of coding block groups (e.g., video coding block groups) and / or coding blocks (e.g., video coding blocks), for example, the video can be encoded using I, P, or B coding block groups and / or coding blocks.

[0200] In one embodiment, Figure 3 The illustrated video encoder 30 may also be used to partition and / or decode an image using slices (also referred to as video slices), where an image may be partitioned or decoded using one or more slices (usually non-overlapping). Each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., coding blocks in the H.265 / HEVC / VVC standard and bricks in the VVC standard).

[0201] In one embodiment, Figure 3The video decoder 30 shown can also be used to segment and / or decode an image using slices / coding block groups (also called video coding block groups) and / or coding blocks (also called video coding blocks), wherein an image can be segmented or decoded using one or more slices / coding block groups (usually non-overlapping), each slice / coding block group may include one or more blocks (e.g., CTU) or one or more coding blocks, etc., wherein each coding block may be in a rectangular shape, etc., and may include one or more complete or partial blocks (e.g., CTU).

[0202] Other variations of the video decoder 30 may be used to decode the encoded image data 21. For example, the decoder 30 may generate an output video stream without the loop filter unit 320. For example, a non-transform based decoder 30 may directly dequantize the residual signal without the inverse transform processing unit 312 for certain blocks or frames. In another implementation, the video decoder 30 may have the dequantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0203] It should be understood that the processing result of the current step can be further processed in the encoder 20 and the decoder 30 and then output to the next step. For example, after interpolation filtering, motion vector derivation or loop filtering, the processing result of interpolation filtering, motion vector derivation or loop filtering can be further operated, such as clipping or shifting operation.

[0204] It should be noted that the derived motion vector of the current block (including but not limited to the control point motion vector of the affine mode, the sub-block motion vector of the affine, plane, ATMVP mode, the temporal motion vector, etc.) can be further operated. For example, the value of the motion vector is limited to a predefined range according to the representation bit of the motion vector. If the representation bit of the motion vector is bitDepth, the range is -2^(bitDepth-1) to 2^(bitDepth-1)-1, where "^" represents the power. For example, if bitDepth is set to 16, the range is -32768 to 32767; if bitDepth is set to 18, the range is -131072 to 131071. For example, the value of the derived motion vector (e.g., the MV of 4 4×4 sub-blocks in an 8×8 block) is limited so that the maximum difference between the integer parts of the 4 4×4 sub-block MVs does not exceed N pixels, for example, not more than 1 pixel. Two methods of limiting motion vectors according to bitDepth are provided here.

[0205] Although the above embodiments mainly describe video coding and decoding, it should be noted that the embodiments of the decoding system 10, the encoder 20 and the decoder 30 and other embodiments described herein can also be used for still image processing or coding and decoding, that is, the processing or coding and decoding of a single image in video coding and decoding that is independent of any previous or consecutive images. In general, if the image processing is limited to a single image 17, the inter-frame prediction unit 244 (encoder) and the inter-frame prediction unit 344 (decoder) may not be available. All other functions (also called tools or techniques) of the video encoder 20 and the video decoder 30 can also be used for still image processing, such as residual calculation 204 / 304, transformation 206, quantization 208, inverse quantization 210 / 310, (inverse) transformation 212 / 312, segmentation 262 / 362, intra-frame prediction 254 / 354 and / or loop filtering 220 / 320, entropy coding 270 and entropy decoding 304.

[0206] Figure 4 4 is an exemplary block diagram of a video decoding device 400 according to an embodiment of the present application. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be a decoder, such as Figure 1A The video decoder 30 in the embodiment may also be an encoder, for example Figure 1A The video encoder 20 in.

[0207] The video decoding device 400 includes: an input port 410 (or input port 410) and a receiving unit (Rx) 420 for receiving data; a processor, a logic unit or a central processing unit (CPU) 430 for processing data; a transmitter unit (Tx) 440 and an output port 450 (or output port 450) for transmitting data; and a memory 460 for storing data. The video decoding device 400 may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to the input port 410, the receiving unit 420, the transmitting unit 440 and the output port 450 for the output or output of optical signals or electrical signals.

[0208] The processor 430 is implemented by hardware and software. The processor 430 can be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the input port 410, the receiving unit 420, the sending unit 440, the output port 450, and the memory 460. The processor 430 includes a decoding module 470. The decoding module 470 implements the embodiments disclosed above. For example, the decoding module 470 performs, processes, prepares, or provides various encoding operations. Therefore, the decoding module 470 provides substantial improvements to the functions of the video decoding device 400 and affects the switching of the video decoding device 400 to different states. Alternatively, the decoding module 470 is implemented with instructions stored in the memory 460 and executed by the processor 430.

[0209] The memory 460 includes one or more disks, tape drives, and solid-state hard disks, and can be used as an overflow data storage device for storing such programs when such programs are selected for execution, and for storing instructions and data read during program execution. The memory 460 can be volatile and / or non-volatile, and can be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random-access memory (SRAM).

[0210] To facilitate understanding, the terms used in the embodiments of the present application are first explained below, and the terms are also considered as part of the content of the invention.

[0211] YUV: usually refers to the representation of color format images in the YCbCr color space, which contains three color channels: Y, U, and V. The Y channel is the brightness component channel, which represents the brightness information of the image; the U and V channels are the chrominance component channels, which represent the color information of the image. There are three main sampling methods for YUV images, YUV4:4:4, YUV4:2:2, and YUV4:2:0. Among them, the sampling rates of the Y, U, and V components of YUV4:4:4 are the same, the sampling rates of the U and V components of YUV4:2:2 are half of the Y component, and the sampling rates of the U and V components of YUV4:2:0 are one-quarter of the Y component. Among them,

[0212] The sampling rates of the Y, U, and V components of YUV4:4:4 are the same. According to every 2 rows of pixels, the brightness component (Y) takes 4 sampling points and the chrominance component (UV) also takes 4 sampling points. Figure 5a As shown, in the figure, ○ represents the brightness sampling point, and × represents the sampling point of a chromaticity component;

[0213] The sampling rate of the U and V components of YUV4:2:2 is half of that of the Y component. For every 2 rows of pixels, the brightness component (Y) takes 4 sampling points and the chrominance component (UV) takes 2 sampling points. For example, the chrominance component takes 2 sampling points in the first row and 2 sampling points in the second row. The chrominance sampling point can be the same as or different from the position of the brightness sampling point. It can be interpolated from the chrominance sampling pixels at the corresponding position, such as Figure 5b As shown;

[0214] The sampling rate of the U and V components of YUV4:2:0 is one-fourth of that of the Y component. For every 2 rows of pixels, the brightness component (Y) takes 4 sampling points and the chrominance component (UV) takes 1 sampling point. For example, the chrominance component takes 2 sampling points in the first row and 0 sampling points in the second row. Figure 5c shown.

[0215] Video segment (group of picture, GOP): A video segment is divided into multiple video segments of the same length. Any GOP includes multiple image frames, and multiple GOPs include the same number of image frames.

[0216] IPPP: An inter-frame reference mode for video coding. In any GOP, only the first image frame is encoded using intra-frame prediction, i.e., an I-frame. Each subsequent image frame is encoded using inter-frame prediction based on the previous image frame, i.e., a P-frame.

[0217] Motor vehicles, non-motor vehicles and pedestrians: an abbreviation for motor vehicles, non-motor vehicles and pedestrians.

[0218] Video segment (group of picture, GOP)

[0219] The embodiment of the present application provides an image encoding method and device, and the technical solution of the present application will be described below.

[0220] Figure 6 600 is a flowchart of the process 600 of the image encoding method of the present application. The process 600 may be performed by the video encoder 20 (or encoder). The process 600 is described as a series of steps or operations, and it should be understood that the process 600 may be performed in various orders and / or occur simultaneously, not limited to Figure 6 The execution order shown. Process 600 includes the following steps:

[0221] Step 601: Obtain a preset background frame.

[0222] The background frame is required to include as few foreground objects as possible, such as machines rather than people. For example, in a video shot by a security camera with a fixed viewing angle, image frames without dynamic foreground objects such as machines rather than people can be used as background frames.

[0223] The image content of the background frame is the background targets that remain unchanged for a long time within the camera's field of view, such as walls, curtains, bookshelves, beds, tables, etc. These objects are in fixed positions for a long time. When there are no cameras or non-human objects in the frame, multiple consecutive image frames (for example, all image frames or most image frames of the video) in the video shot by the camera will contain these background targets, and their positions, orientations, etc. are unchanged, and their corresponding pixel values ​​are also unchanged; even when there are cameras or non-human objects in the frame, at least one image frame in the video shot by the camera still contains these background targets at a position where they are not blocked by cameras or non-human objects.

[0224] The reason why non-human targets such as machines are called dynamic foreground targets is that in a GOP or a long video, these targets do not appear in all or most of the image frames, and may only appear in some image frames. Compared with the above-mentioned background targets, non-human targets such as machines only appear briefly in the video. Therefore, non-human targets such as machines can be considered dynamic foreground targets.

[0225] In a possible implementation, the background frame may be specified by the user, who, by watching a video, specifies an image frame that meets the above requirements as the background frame.

[0226] In a possible implementation, the background frame can be determined by a target recognition or target segmentation method. The encoder performs target recognition or target segmentation on the image frames in the video to obtain whether the target contained in the image frames has changed. If the targets contained in multiple consecutive image frames are the same, one of the multiple image frames can be determined as the background frame (for example, the first image frame among the multiple image frames is determined as the background frame).

[0227] It should be noted that the embodiments of the present application may also use other methods to determine the background frame, such as a neural network, etc., so there is no specific limitation on the method for obtaining the background frame.

[0228] In an embodiment of the present application, the background frame can be the first frame of the video currently being processed, or it can be a frame in a video shot within an initial period of time after the camera is turned on, or it can be an image frame in any video shot by the camera. Therefore, the embodiment of the present application does not specifically limit which image frame is used as the background frame.

[0229] Optionally, the encoder may periodically acquire a background frame. Whenever a background frame is updated, subsequent image frames are processed based on the latest background frame.

[0230] Step 602: Get the current frame.

[0231] The current frame is an image frame to be encoded in the current GOP, and the image frame may be the first frame of the current GOP or a non-first frame of the current GOP. The previous frame and the background frame may or may not be adjacent to each other, and this is not specifically limited.

[0232] Step 603: Acquire a foreground frame corresponding to the current frame according to the background frame and the current frame.

[0233] The foreground frame can reflect the difference between the current frame and the background frame. For example, if the image content of the current frame includes a foreground object such as a machine but not a person, then the pixel value at the position corresponding to the foreground object such as a machine but not a person in the current frame is different from the pixel value at the same position in the background frame, and the foreground frame can represent the difference in the aforementioned pixel values.

[0234] In a possible implementation, the method for acquiring the foreground frame may include: acquiring a foreground template according to a background frame and a current frame; and then calculating the difference between the current frame and the foreground template to obtain the foreground frame.

[0235] The foreground template can be obtained by the following formula:

[0236]

[0237] Among them, t (x,y) Represents the pixel value at the (x,y) position in the foreground template, e (x,y) Represents the pixel value at the (x,y) position in the background frame, y (x,y) Represents the pixel value at position (x,y) in the current frame.

[0238] That is, taking the position (x, y) in the image frame as an example, the pixel value in the background frame is e (x,y) , whose pixel value in the current frame is y (x,y) , in the foreground template, its pixel value is t (x,y) When the pixel value in the current frame is the same as the pixel value in the background frame, that is, e (x,y) =y (x,y) , the pixel value in the current frame can be assigned to the foreground template; when the pixel value in the current frame is different from the pixel value in the background frame, that is, e (x,y) ≠y (x,y) , the foreground template can be assigned a value of 0. In the obtained foreground template, the pixel value in the first area is not 0, which is taken from the first area of ​​the current frame, and the pixel value in the second area is 0, where the first area is the same as the pixel value of the background frame (i.e., e (x,y) =y (x,y) ) area, and the second area is the area where the pixel values ​​of the current frame and the background frame are different (i.e., e (x,y) ≠y (x,y) ) area.

[0239] The foreground frame can be obtained by the following formula:

[0240] f (x,y) =y (x,y) -t (x,y)

[0241] Among them, f (x,y) Represents the pixel value at the (x,y) position in the foreground frame, y (x,y) Represents the pixel value at the (x,y) position in the current frame, t (x,y) Represents the pixel value at the (x,y) position in the foreground template.

[0242] The foreground frame obtained through the above calculation satisfies the following formula:

[0243]

[0244] Among them, f (x,y) Represents the pixel value at the (x,y) position in the foreground frame, e (x,y) Represents the pixel value at the (x,y) position in the background frame, y (x,y) Represents the pixel value at position (x,y) in the current frame.

[0245] That is, taking the position (x, y) in the image frame as an example, the pixel value in the background frame is e (x,y) , whose pixel value in the current frame is y (x,y) , whose pixel value in the foreground frame is f (x,y) When the pixel value in the current frame is the same as the pixel value in the background frame, that is, e (x,y) =y (x,y) , the foreground frame can be assigned a value of 0; when the pixel value in the current frame is different from the pixel value in the background frame, that is, e (x,y) ≠y (x,y) , the pixel values ​​in the current frame can be assigned to the foreground template. In the obtained foreground frame, the pixel values ​​in the first area are 0, and the pixel values ​​in the second area are not 0, which are taken from the second area of ​​the current frame.

[0246] The above difference calculation is similar to the principle of residual calculation. The pixel values ​​that are the same as the background frame (located in the first area) can be changed to 0, and only the pixel values ​​in the current frame that are different from the background frame (located in the second area) are retained. Such a background frame may contain a large number of zero-valued pixel values, so the amount of data can be greatly reduced in the subsequent encoding of the background frame. Only encoding the foreground frame that is different from the background frame (also called foreground encoding) can effectively reduce the redundant encoding of background information in the video and further reduce the size of the bitstream.

[0247] Step 604: Encode the foreground frame to obtain a first code stream.

[0248] In a possible implementation, when the current frame is the first frame of the current video segment, the foreground frame is encoded as an I frame to obtain a first bitstream. Alternatively, when the current frame is a non-first frame of the current video segment, the foreground frame is encoded as a P frame to obtain a first bitstream.

[0249] As described in step 602 above, the current frame can be the first frame of the current video segment (GOP) or a non-first frame of the current GOP. Then according to the IPPP inter-frame reference mode, the first frame of a GOP is encoded as an I frame according to the intra-frame prediction method, and each image frame (non-first frame) after the first frame of the GOP is encoded as a P frame based on the previous image frame according to the inter-frame prediction method. Therefore, the encoding method for the foreground frame can refer to the corresponding current frame. If the current frame is the first frame, the foreground frame is encoded as an I frame. If the current frame is not the first frame, the foreground frame is encoded as a P frame. In this scenario, the image frames in multiple GOPs obtain foreground frames based on the background frames, respectively, which has reduced the amount of encoded data (i.e., background information). Therefore, even when the first frame of each GOP is encoded as an I frame, compared with the related technology that encodes the complete image frame as an I frame, the purpose of reducing redundant encoding and improving encoding efficiency has been achieved.

[0250] It should be noted that other encoding methods may be used for the foreground frame, and reference may be made to existing video encoding and decoding standards, which are not specifically limited in the embodiments of the present application.

[0251] In a possible implementation manner, the background frame is encoded as an I frame to obtain a second code stream.

[0252] According to the above description, the background frame is the basis for converting subsequent image frames into foreground frames. Therefore, in order for the decoder to have a basis, the complete image information of the background frame needs to be retained. Therefore, the encoder can encode the background frame as an I frame, and the processing of the background frame needs to be before all image frames that refer to it.

[0253] In the embodiment of the present application, by calculating the difference between the current frame and the background frame, the values ​​of the positions where the pixel values ​​of the two are the same can be set to 0, and then the values ​​of the positions where the pixel values ​​are different can be encoded, which can greatly reduce the amount of data. Only encoding the pixel values ​​that are different from the background frame can effectively reduce the redundant encoding of the background information in the video and further reduce the size of the bitstream.

[0254] Figure 7 Flowchart of process 700 of the image decoding method of the present application. Process 700 may be performed by video decoder 20 (or decoder). Process 700 is described as a series of steps or operations, and it should be understood that process 700 may be performed in various orders and / or occur simultaneously, not limited to Figure 7 The execution order shown. Process 700 includes the following steps:

[0255] Step 701: Get code stream.

[0256] Step 702: parse the code stream to obtain a background frame and a current foreground frame.

[0257] Corresponding to the encoder, the decoder can perform I frame decoding on the bitstream to obtain a background frame. The bitstream is decoded by I frame decoding or P frame decoding to obtain a current foreground frame. The method of decoding the bitstream by the decoder can be determined based on the syntax elements carried in the bitstream.

[0258] Step 703: Acquire a reconstructed frame corresponding to the current foreground frame according to the background frame and the current foreground frame.

[0259] Corresponding to the encoder, the current foreground frame obtained after decoding the bitstream can be referenced Figure 6 The description of the illustrated embodiment reflects the difference between the current frame and the background frame.

[0260] In the foreground frame, the pixel value in the first area is 0, indicating that the pixel values ​​of the image frame and the background frame in the area are the same; the pixel value in the second area is not 0, indicating that the pixel values ​​of the image frame and the background frame in the area are different, and the pixel value of the second area that is not 0 is taken from the image frame at the same position, not the background frame.

[0261] Therefore, when obtaining the reconstructed frame, based on its corresponding foreground frame, in the first area, the pixel value of any position can be taken as the pixel value of the same position in the background frame, and in the second area, the pixel value of any position can be taken as the pixel value of the same position in the foreground frame.

[0262] Based on this, the reconstructed frame satisfies the following formula:

[0263]

[0264] in, represents the pixel value at the (x,y) position in the reconstructed frame, Represents the pixel value at the (x,y) position in the current foreground frame, Represents the pixel value at position (x,y) in the background frame.

[0265] In the embodiment of the present application, the decoded bit stream can obtain a foreground frame and a background frame. The foreground frame only contains the values ​​of the pixel values ​​​​of the image frame and the background frame that are different, and the other positions are all 0, which effectively reduces the redundant encoding of background information and further reduces the bit rate. The separate processing of the background frame and the foreground frame can effectively reduce the computational loss caused by quantization, and the decoder can achieve lossless restoration of the reconstructed frame after encoding.

[0266] The technical solution of the present application is described below through several specific embodiments.

[0267] Figure 8 This is a flowchart of the image encoding method according to an embodiment of the present application. As shown in FIG8 , the method includes three steps:

[0268] 1) Select an image frame from the video as the background frame by using target recognition, target segmentation, etc. The background frame should include as few foreground targets as possible, such as machines but not people;

[0269] 2) The Nth image frame in the GOP and the background frame are calculated to obtain the foreground frame. The above calculation process can refer to Figure 6 Step 603 of the illustrated embodiment;

[0270] 3) The foreground frame is input into the encoder, which is encoded according to the IPPP inter-frame reference mode (including inter-frame prediction, transformation, quantization and entropy coding), and the encoder outputs the Nth frame foreground code stream (corresponding to the foreground frame of the Nth image frame). The background frame can be I-frame encoded as the first frame of the video.

[0271] Fig. 9 is a schematic diagram of an encoding method according to an embodiment of the present application, such as Fig. 9 As shown, the video (such as a YUV video source) is encoded foreground (refer to Figure 6 The image encoding method shown in the figure) is used, wherein the background frame (Bg) is encoded as an I frame, the first frame of a GOP is encoded as an I frame, and the non-first frames of the GOP are encoded as P frames, the first frame of the next GOP is encoded as an I frame, and the non-first frames of the GOP are encoded as P frames, and so on.

[0272] Fig.10 is a schematic diagram of a decoding method according to an embodiment of the present application, such as Fig.10 As shown, the code stream corresponds to Bg frame, I frame, P frame, ..., I frame, P frame, P frame. The aforementioned code stream is decoded to obtain decoded frames, wherein Bg is decoded to obtain background frames, and the remaining code streams are decoded to obtain foreground frames. The foreground frame of the first I frame is combined with the background frame to obtain the reconstructed 1st frame; the foreground frame of the next P frame is combined with the background frame to obtain the reconstructed 2nd frame; ...; the foreground frame of the subsequent I frame is combined with the background frame to obtain the reconstructed n-1th frame; the foreground frame of the next P frame is combined with the background frame to obtain the reconstructed nth frame; the foreground frame of the next P frame is combined with the background frame to obtain the reconstructed n+1th frame.

[0273] As described above, the technical solution of the embodiments of the present application can be used on front-end cameras, back-end servers and other devices with video encoding functions to provide efficient video encoding functions for cameras, servers, etc.

[0274] By obtaining the encoded bitstream, performing bitrate analysis on it, and analyzing the decoded reconstructed video stream:

[0275] 1) Determine the structure of the coded bitstream: Use a bitstream analysis tool (such as StreamEye) to analyze the bitstream. If the structure after analysis is as shown in 11( Fig.11 As shown in the code stream structure diagram, the first frame and the second frame in the code stream are both I frames, and the overall code stream structure is as follows Fig. 9 shown.

[0276] 2) Determine the decoded reconstructed video stream: Decode the code stream according to the video codec standard. Only the first image frame is a normal picture, and the subsequent image frames only contain foreground objects, such as Fig.12 ( Fig.12 (a schematic diagram of the decoded image) is shown in FIG.

[0277] Fig.13 1 is a schematic diagram of the structure of an image encoding device 1300 according to an embodiment of the present application. The image encoding device 1300 can be used in a video encoder 20 (or encoder), and includes: an acquisition module 1301, a processing module 1302 and an encoding module 1303.

[0278] An acquisition module 1301 is used to acquire a pre-set background frame; acquire a current frame; a processing module 1302 is used to acquire a foreground frame corresponding to the current frame based on the background frame and the current frame, wherein the pixel value of the first area in the foreground frame is 0, and the pixel value of the second area in the foreground frame is not 0, the first area is an area where the current frame and the background frame have the same pixel value, and the second area is an area where the current frame and the background frame have different pixel values; an encoding module 1303 is used to encode the foreground frame to obtain a first code stream.

[0279] In a possible implementation, the foreground frame satisfies the following formula:

[0280]

[0281] Among them, f (x,y) represents the pixel value at the (x, y) position in the foreground frame, e (x,y) Represents the pixel value at the (x, y) position in the background frame, y (x,y) Represents the pixel value at the (x, y) position in the current frame.

[0282] In a possible implementation, the processing module 1302 is specifically configured to obtain a foreground template according to the background frame and the current frame; and obtain the foreground frame by subtracting the current frame from the foreground template.

[0283] In a possible implementation, the foreground template is obtained by the following formula:

[0284]

[0285] Among them, t (x,y) represents the pixel value at the (x, y) position in the foreground template, e (x,y) Represents the pixel value at the (x, y) position in the background frame, y (x,y) Represents the pixel value at the (x, y) position in the current frame.

[0286] In a possible implementation, the foreground frame is obtained by the following formula:

[0287] f (x,y) =y (x,y) -t (x,y)

[0288] Among them, f (x,y) Represents the pixel value at the (x, y) position in the foreground frame, y (x,y) represents the pixel value at the (x, y) position in the current frame, t (x,y) Represents the pixel value at the (x, y) position in the foreground template.

[0289] In a possible implementation, the encoding module 1303 is specifically used to, when the current frame is the first frame of the current video segment, perform I frame encoding on the foreground frame to obtain the first code stream; or, when the current frame is not the first frame of the current video segment, perform P frame encoding on the foreground frame to obtain the first code stream.

[0290] In a possible implementation, the encoding module 1303 is further configured to perform I-frame encoding on the background frame to obtain a second code stream.

[0291] In a possible implementation manner, the background frame is specified by a user; or, the background frame is determined by a target recognition or target segmentation device.

[0292] The device of this embodiment can be used to perform Figure 6 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0293] Fig.14 14 is a schematic diagram of the structure of an image decoding device 1400 according to an embodiment of the present application. The image decoding device 1400 can be used in a video decoder 30 (or decoder), and includes: an acquisition module 1401, a decoding module 1402 and a processing module 1403.

[0294] The acquisition module 1401 is used to acquire a code stream; the decoding module 1402 is used to parse the code stream to obtain a background frame and a current foreground frame; the processing module 1403 is used to acquire a reconstructed frame corresponding to the current foreground frame according to the background frame and the current foreground frame.

[0295] In a possible implementation, the reconstructed frame satisfies the following formula:

[0296]

[0297] in, represents the pixel value at the (x, y) position in the reconstructed frame, represents the pixel value at the (x, y) position in the current foreground frame, Represents the pixel value at the (x, y) position in the background frame.

[0298] In a possible implementation, the decoding module 1402 is specifically configured to perform I frame decoding on the code stream to obtain the background frame; and perform I frame decoding or P frame decoding on the code stream to obtain the current foreground frame.

[0299] The device of this embodiment can be used to perform Figure 7 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.

[0300] In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The processor can be a general-purpose processor, a digital signal processor (digital signal processor, DSP), an application-specific integrated circuit (application-specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware coding processor to be executed, or the hardware and software modules in the coding processor are combined to be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0301] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0302] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0303] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0304] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0305] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0306] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0307] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0308] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image coding method, characterized in that: include: Get the preset background frame; Get the current frame; Acquire a foreground frame corresponding to the current frame according to the background frame and the current frame, wherein the pixel value of a first area in the foreground frame is 0, and the pixel value of a second area in the foreground frame is not 0, the first area is an area where the current frame and the background frame have the same pixel value, and the second area is an area where the current frame and the background frame have different pixel values; The foreground frame is encoded to obtain a first code stream.

2. The method according to claim 1, characterized in that The foreground frame satisfies the following formula: Among them, f (x,y) represents the pixel value at the (x, y) position in the foreground frame, e (x,y) Represents the pixel value at the (x, y) position in the background frame, y (x,y) Represents the pixel value at the (x, y) position in the current frame.

3. The method according to claim 1, characterized in that The step of acquiring a foreground frame corresponding to the current frame according to the background frame and the current frame includes: Acquire a foreground template according to the background frame and the current frame; The current frame is subtracted from the foreground template to obtain the foreground frame.

4. The method according to claim 3, characterized in that The foreground template is obtained by the following formula: Among them, t (x,y) represents the pixel value at the (x, y) position in the foreground template, e (x,y) Represents the pixel value at the (x, y) position in the background frame, y (x,y) Represents the pixel value at the (x, y) position in the current frame.

5. The method according to claim 3 or 4, characterized in that: The foreground frame is obtained by the following formula: f (x,y) =y (x,y) -t (x,y) Among them, f (x,y) Represents the pixel value at the (x, y) position in the foreground frame, y (x,y) represents the pixel value at the (x, y) position in the current frame, t (x,y) Represents the pixel value at the (x, y) position in the foreground template.

6. The method according to any one of claims 1 to 5, characterized in that The encoding of the foreground frame to obtain a first code stream includes: When the current frame is the first frame of the current video segment, I-frame encoding is performed on the foreground frame to obtain the first bitstream; or, When the current frame is a non-first frame of the current video segment, P frame encoding is performed on the foreground frame to obtain the first code stream.

7. The method according to any one of claims 1 to 6, characterized in that Also includes: The background frame is encoded as an I frame to obtain a second code stream.

8. The method according to any one of claims 1 to 7, characterized in that The background frame is specified by a user; or, the background frame is determined by a target recognition or target segmentation method.

9. An image decoding method, characterized in that: include: Get the code stream; Parsing the code stream to obtain a background frame and a current foreground frame; A reconstructed frame corresponding to the current foreground frame is acquired according to the background frame and the current foreground frame.

10. The method according to claim 9, characterized in that The reconstructed frame satisfies the following formula: in, represents the pixel value at the (x, y) position in the reconstructed frame, represents the pixel value at the (x, y) position in the current foreground frame, Represents the pixel value at the (x, y) position in the background frame.

11. The method according to claim 9 or 10, characterized in that: The parsing of the code stream to obtain a background frame and a current foreground frame includes: Performing I frame decoding on the code stream to obtain the background frame; The code stream is subjected to I frame decoding or P frame decoding to obtain the current foreground frame.

12. An image encoding device, characterized in that: include: An acquisition module, used for acquiring a preset background frame; Get the current frame; A processing module, configured to obtain a foreground frame corresponding to the current frame according to the background frame and the current frame, wherein the pixel value of a first area in the foreground frame is 0, and the pixel value of a second area in the foreground frame is not 0, the first area is an area where the current frame and the background frame have the same pixel value, and the second area is an area where the current frame and the background frame have different pixel values; The encoding module is used to encode the foreground frame to obtain a first code stream.

13. The device according to claim 12, characterized in that The foreground frame satisfies the following formula: Among them, f (x,y) represents the pixel value at the (x, y) position in the foreground frame, e (x,y) Represents the pixel value at the (x, y) position in the background frame, y (x,y) Represents the pixel value at the (x, y) position in the current frame.

14. The device according to claim 12, characterized in that The processing module is specifically used to obtain a foreground template according to the background frame and the current frame; and to obtain the foreground frame by subtracting the current frame from the foreground template.

15. The device according to claim 14, characterized in that The foreground template is obtained by the following formula: Among them, t (x,y) represents the pixel value at the (x, y) position in the foreground template, e (x,y) Represents the pixel value at the (x, y) position in the background frame, y (x,y) Represents the pixel value at the (x, y) position in the current frame.

16. The device according to claim 14 or 15, characterized in that The foreground frame is obtained by the following formula: f (x,y) =y (x,y) -t (x,y) Among them, f (x,y) Represents the pixel value at the (x, y) position in the foreground frame, y (x,y) represents the pixel value at the (x, y) position in the current frame, t (x,y) Represents the pixel value at the (x, y) position in the foreground template.

17. The device according to any one of claims 12 to 16, characterized in that The encoding module is specifically used to perform I frame encoding on the foreground frame to obtain the first code stream when the current frame is the first frame of the current video segment; or, when the current frame is not the first frame of the current video segment, perform P frame encoding on the foreground frame to obtain the first code stream.

18. The device according to any one of claims 12 to 17, characterized in that The encoding module is further used to perform I frame encoding on the background frame to obtain a second code stream.

19. The device according to any one of claims 12 to 18, characterized in that The background frame is specified by a user; or, the background frame is determined by a target recognition or target segmentation device.

20. An image decoding device, characterized in that: include: Acquisition module, used to obtain code stream; A decoding module, used for parsing the code stream to obtain a background frame and a current foreground frame; A processing module is used to obtain a reconstructed frame corresponding to the current foreground frame according to the background frame and the current foreground frame.

21. The device according to claim 20, characterized in that The reconstructed frame satisfies the following formula: in, represents the pixel value at the (x, y) position in the reconstructed frame, represents the pixel value at the (x, y) position in the current foreground frame, Represents the pixel value at the (x, y) position in the background frame.

22. The device according to claim 20 or 21, characterized in that The decoding module is specifically used to perform I frame decoding on the code stream to obtain the background frame; and perform I frame decoding or P frame decoding on the code stream to obtain the current foreground frame.

23. An encoder, characterized in that: include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the encoder to perform the method according to any one of claims 1-8.

24. A decoder, characterized in that include: one or more processors; A non-transitory computer-readable storage medium, coupled to the processor and storing a program executed by the processor, wherein the program, when executed by the processor, causes the decoder to perform the method according to any one of claims 9-11.

25. A computer-readable storage medium, characterized in that: The method comprises a program code, which, when executed by a computer device, is used to perform the method according to any one of claims 1 to 11.

26. A computer-readable storage medium, characterized in that: A bit stream comprising the bit stream encoded according to the method according to any one of claims 1-9.

Images