LTR frame updates in video coding

Abstract

Relates to LTR frame updating in video coding. Techniques for LTR frame update in a video coding process are provided. As part of a video encoding process, a method is performed by an image processing apparatus. The method includes encoding a first LTR frame. The method includes encoding a plurality of frames that directly or indirectly refer to a first LTR frame. The method includes sequentially updating the first LTR frame by evaluating a cost for encoding a block of image data in one of the plurality of frames and by updating an image region in the first LTR frame when the cost meets a cost criterion. The image region is updated based on the block of image data in at least one of the plurality of frames. The method includes encoding the sequentially updated first LTR frame as a second LTR frame.

Description

Technical Field

[0001] The embodiments presented herein relate to a method, an image processing apparatus, a computer program and a computer program product for long-term reference frame updating in a video encoding process. Background Art

[0002] In general, in order to improve video coding capabilities, a challenge is to improve compression efficiency while maintaining the original video quality. Motion compensated prediction can be used as part of achieving this goal. Motion compensated prediction generally involves applying a prediction model to generate new frames by shifting macroblocks or subblocks from other frames. These frames are called reference frames. In this regard, short-term (STR) frames and long-term reference frames (LTR) can be defined. LTR frames are frames that are marked as references for future frames, even for long time distances. LTR frames can be saved and referenced until explicitly deleted by the application.

[0003] exist Figure 1 Two examples of how LTR frames can be used are illustrated in FIG. Figure 1 Starting with the first example illustrated in (a) of the first sequence of frames 100a, reference is made to an LTR frame from a group of P frames (where P is an abbreviation for prediction). In some examples, the LTR frame and the P frame constitute a group of pictures (GOP). Figure 1 The second example illustrated in (b) of the second sequence of frames 100b references LTR frames from two groups of P frames, where the P frames in each group also reference corresponding I frames, and where each of the I frames also references an LTR frame (where I is an abbreviation for Internal). In some examples, the LTR frames, I frames, and P frames constitute a GOP. In other examples, each I frame and its referenced P frame constitute a GOP, which means that in Figure 1 In the example of (b), there are two GOPs.

[0004] A reference frame encoded with higher quality may improve the image quality of subsequent frames. If the video scene has a stable image scene, the reference frame can be kept longer while decoding more frames, which will help avoid transmitting another reference frame in the case of a stable scene and thus save transmission bandwidth. LTR frames have the advantage of being controlled by the encoding process at the application layer. This increases flexibility for improving coding efficiency.

[0005] The decoded images designated as LTR frames can be saved to the frame memory for a long period of time. Accordingly, it is possible to provide a long-term reference to the decoded frames designated as LTR frames in subsequent decoding iterations.

[0006] In some cases, the appropriate reference frame may not necessarily be included in the video to be encoded. In this case, it is difficult to refer to the appropriate reference image for inter-frame prediction. Therefore, the encoding efficiency may be reduced.

[0007] US20180007378 A1 discloses that the LTR frame can be updated based in part on multiple captured images. A moving object in the captured image can be identified, and only the background portion, not the area of ​​the moving object, is used to update the LTR frame.

[0008] However, there still exists a need to improve the updating of LTR frames. Summary of the invention

[0009] It is an aim of embodiments herein to provide efficient updating of LTR frames.

[0010] A specific object is to provide cost-effective updating of LTR frames.

[0011] One specific purpose is to avoid unnecessary updating of LTR frames.

[0012] According to a first aspect, a method for LTR frame updating in a video encoding process is proposed. The method is performed by an image processing device. As part of the video encoding process, the method includes encoding a first LTR frame. As part of the video encoding process, the method includes encoding multiple frames that directly or indirectly reference the first LTR frame. As part of the video encoding process, the method includes sequentially updating the first LTR frame by evaluating the cost for encoding an image data block in one of the multiple frames and by updating the image area in the first LTR frame when the cost meets a cost criterion. The image area is updated based on the image data block in at least one frame of the multiple frames. As part of the video encoding process, the method includes encoding the sequentially updated first LTR frame as a second LTR frame.

[0013] According to a second aspect, an image processing device for LTR frame updating in a video encoding process is proposed. The image processing device includes a processing circuit. The processing circuit is configured to cause the image processing device to encode a first LTR frame as part of the video encoding process. The processing circuit is configured to cause the image processing device to encode multiple frames that directly or indirectly refer to the first LTR frame as part of the video encoding process. The processing circuit is configured to cause the image processing device to sequentially update the first LTR frame as part of the video encoding process by evaluating the cost for encoding an image data block in one of the multiple frames and by updating the image region in the first LTR frame when the cost meets a cost criterion. The image region is updated based on the image data block in at least one frame of the multiple frames. The processing circuit is configured to cause the image processing device to encode the sequentially updated first LTR frame as a second LTR frame as part of the video encoding process.

[0014] According to a third aspect, a computer program for LTR frame updating in a video encoding process is proposed. The computer program includes computer code, which, when executed on a processing circuit of an image processing device, causes the image processing device to perform actions. One action includes the image processing device encoding a first LTR frame as part of the video encoding process. The processing circuit is configured to cause the image processing device to encode multiple frames that directly or indirectly reference the first LTR frame as part of the video encoding process. One action includes the image processing device sequentially updating the first LTR frame as part of the video encoding process by evaluating the cost for encoding an image data block in one of the multiple frames and by updating the image region in the first LTR frame when the cost meets a cost criterion. The image region is updated based on the image data block in at least one frame of the multiple frames. One action includes the image processing device encoding the sequentially updated first LTR frame as a second LTR frame as part of the video encoding process.

[0015] According to a fourth aspect, a computer program product is proposed, comprising the computer program according to the third aspect and a computer readable storage medium on which the computer program is stored. The computer readable storage medium may be a non-transitory computer readable storage medium.

[0016] Advantageously, these aspects provide efficient updating of LTR frames.

[0017] Advantageously, these aspects provide for cost-effective updating of LTR frames.

[0018] Advantageously, these aspects avoid unnecessary updating of LTR frames.

[0019] Advantageously, these aspects allow only blocks of image data to be updated at a time, rather than entire LTR frames, which in turn improves storage efficiency since less image data needs to be processed for the update.

[0020] Advantageously, these aspects thus reduce the need for data that needs to be stored in memory in order to update the LTR frame.

[0021] Other objectives, features and advantages of the enclosed embodiments will become apparent from the following detailed description, the attached dependent claims as well as the accompanying drawings.

[0022] In general, unless explicitly defined herein, all terms used in the claims should be interpreted according to their ordinary meaning in the art. Unless explicitly stated otherwise, all references to "a / an / the element, device, part, means, module, step, etc." should be openly interpreted as referring to at least one instance of the element, device, part, means, module, step, etc. Unless explicitly stated otherwise, the steps of any method disclosed herein do not need to be performed in the order disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present inventive concept will now be described by way of example with reference to the accompanying drawings, in which:

[0024] Figure 1 schematically illustrates a frame sequence according to an embodiment;

[0025] Figure 2 is a schematic diagram illustrating a system according to an embodiment;

[0026] Figure 3 is a flow chart of a method according to an embodiment;

[0027] Figure 4 schematically illustrates a frame sequence used in a process of updating an LTR frame according to an embodiment;

[0028] Figure 5 schematically illustrates a frame sequence according to an embodiment;

[0029] Figure 6 is a schematic diagram showing functional units of an image processing apparatus according to an embodiment;

[0030] Figure 7 An example of a computer program product including a computer-readable storage medium according to an embodiment is shown. DETAILED DESCRIPTION

[0031] The inventive concept will now be more fully described with reference to the accompanying drawings in which certain embodiments of the inventive concept are shown. However, the inventive concept can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein; on the contrary, these embodiments are provided by way of example to make the disclosure thorough and complete, and these embodiments will fully convey the scope of the inventive concept to those skilled in the art. Throughout the specification, the same number refers to the same element. Any step or feature illustrated by a dotted line should be considered optional.

[0032] Figure 2 2 is a schematic diagram illustrating a system 200 to which the embodiments proposed herein can be applied. The system 200 includes a first image processing apparatus 210 and a second image processing apparatus 230 operatively connected to each other via a network 220. The network 220 may be wired, wireless, or partially wired and partially wireless.

[0033] Next, aspects of the first image processing device 210 will be disclosed. The first image processing device 210 includes a camera device 212. The camera device 212 is configured to capture image frames. In some examples, the camera device 212 is a digital camera device and / or is capable of pan, tilt and zoom (PTZ), and can therefore be regarded as a (digital) PTZ camera device. In addition, the first image processing device 210 is configured to encode the image so that it can decode the image using any known video coding standard, such as any of the following standards (just to name a few examples): High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2; Advanced Video Coding (AVC), also known as H.264 and MPEG-4 Part 20; Versatile Video Coding (VVC), also known as H.266, MPEG-I Part 3 and Future Video Coding (FVC); VP9, ​​VP10 and AOMedia Video 2 (AV1). In this regard, encoding may be performed directly in conjunction with the camera device 212 that captures the image frames or at another entity (e.g., in a dedicated image encoder 216) and then at least temporarily stored in a database. The camera device 212 and the image encoder 216 are operably connected to the network 220 via the first interface entity 214. In some examples, the camera device 212, the image encoder 216, and the first interface entity 214 are provided in the same device.

[0034] Next, aspects of the second image processing device 230 will be disclosed. The second image processing device 230 includes an image decoder 234. The image decoder 234 is configured to decode the video stream received from the first image processing device 210. Therefore, the image decoder 234 is operably connected to the network 220 via the second interface entity 232. In addition, the second image processing device 230 is operably connected to a user interface device 236 with which the user 240 can interact via the second interface entity 232. In some examples, the user interface device 240 is a display device such as a computer monitor or screen or a television device. In other examples, the user interface device 240 is a handheld portable device such as a laptop computer, a tablet computer or a mobile phone (also referred to as a user device). In some examples, the user interface device 240, the image decoder 234 and the second interface entity 232 are provided in the same device.

[0035] As mentioned above, there still exists a need to improve the updating of LTR frames.

[0036] In more detail, updating LTR frames in a video encoding process based on multiple frames has relatively high processing requirements. Therefore, in the present disclosure, a technique is proposed to logically divide a single LTR frame into multiple regions, where only those regions that need to be updated are actually updated, rather than updating the entire LTR frame. The update is performed based on image data of the corresponding region in a frame group from an acquired video sequence.

[0037] Therefore, the embodiments disclosed herein particularly relate to techniques for LTR frame updating during video encoding. To achieve such techniques, an image processing device 210, 600, a method performed by the image processing device 210, 600, and a computer program product including code, such as in the form of a computer program, are provided, which, when run on the image processing device 210, 600, causes the image processing device 210, 600 to perform the method.

[0038] Which image areas are to be updated in the LTR frame is based on information about how much the cost is currently used to encode corresponding image data blocks in frames that directly or indirectly reference the LTR frame.

[0039] Figure 3 is a flow chart illustrating an embodiment of a method for LTR frame updating in a video encoding process. The method is performed by an image processing device 210, 600. Advantageously, the method is provided as a computer program 720. The method as part of a video encoding process comprises steps S102 to S108. Reference is also made here in parallel to Figure 4 , Figure 4 A frame sequence for updating a first LTR frame, denoted LTR1 , using image frames I1 , I2 , . . . , IK2 to produce an updated second LTR frame denoted LTR2 is schematically illustrated at reference numeral 400 .

[0040] S102: The image processing device 210, 600 encodes the first LTR frame. Figure 4 As in , the first LTR frame is denoted as LTR1. Here, note that the term "first" does not imply that the LTR frame is the first frame, or even the first LTR frame. The term "first" is used here to distinguish the LTR frame from the subsequent LTR frame, which is denoted as the second LTR frame. In some aspects, the second LTR frame can therefore be considered as the current LTR frame, while the first LTR frame can be considered as the previous LTR frame, and more accurately, the LTR frame that most recently appeared before the second LTR frame. That is, the next time the LTR frame is to be updated (resulting in the third LTR frame), the second LTR frame is used as the reference frame, and so on.

[0041] S104: The image processing apparatus 210, 600 encodes a plurality of frames. These frames are denoted as I1, I2, ..., IK, and they directly or indirectly refer to the first LTR frame LTR1.

[0042] There may be different examples of such frames. Generally speaking, these frames are inter-frames (and therefore have the notation "I1" etc.). Intermediate frames may be so-called P-frames or B-frames (where B is short for "bidirectional").

[0043] The first LTR frame is updated sequentially to adapt to the changing conditions in the scene so that the next LTR frame actually encoded (ie, the second LTR frame) is useful, for example, for encoding of the next GOP.

[0044] S106: The image processing device 210, 600 sequentially updates the first LTR frame LTR1, resulting in intermediate LTR frames represented as LTR1Update1, LTR1Update2, ..., LTR1UpdateK. Figure 4 As shown in , LTR1Update0 is updated, and LTR1Update0 corresponds to the first LTR frame LTR1, but LTR1Update0 is not encoded.

[0045] The first LTR frame LTR1 is updated sequentially according to steps S106-2 and S106-4. In this regard, steps S106-2 and S106-4 are therefore repeatedly performed. These steps correspond to a process of updating part of the first LTR frame, but not encoding (or transmitting) the LTR frame after each update. In some examples, steps S106-2 and S106-4 are performed once for each of K frames I1, I2, ..., IK in a plurality of frames. Therefore, in some examples, steps S106-2 and S106-4 are performed K times before entering step S108. However, in other examples, not all K frames I1, I2, ..., IK are considered, and therefore steps S106-2 and S106-4 are performed less than K times before entering step S108.

[0046] S106-2: The image processing device 210, 600 evaluates the cost for encoding an image data block denoted as ID1, ID2, ..., IDK in one of the plurality of frames I1, I2, ..., IK. That is, data block ID1 belongs to image frame I1, and so on. However, the embodiments disclosed herein do not exclude the possibility of encoding more than one data block for each frame for evaluation.

[0047] In some examples, each image data block ID1, ID2, ..., IDK corresponds to one or more macroblocks.

[0048] S106-4: When the cost meets the cost criterion, the image processing device 210, 600 updates the image area represented as IA1, IA2, ..., IAK in the first LTR frame LTR1. The image area IA1, IA2, ..., IAK is updated based on the image data block ID1, ID2, ..., IDK in at least one of the multiple frames I1, I2, ..., IK. That is, the image area IA1 is updated based on the image data block ID1, and so on. As above, the embodiments disclosed herein do not exclude the possibility of updating more than one image area based on different data blocks from the same frame I1, I2, ..., IK.

[0049] In general, updating to a more accurate and up-to-date appearance in an image region may improve the usability of the image region, allowing the image region to be referenced to a greater extent in encoding, thereby reducing encoding costs. Sequentially updating the first LTR frame LTR1 produces an updated first LTR frame denoted as LTR1 update K. Image data blocks that do not meet the cost criteria are reused from a previous LTR frame (i.e., from the first LTR frame LTR1).

[0050] S108: The image processing device 210, 600 encodes the first LTR frame, which is represented as LTR1 update K and is sequentially updated, into a second LTR frame. Figure 4 As shown in , the second LTR frame corresponding to the LTR 1 update K is therefore also denoted as LTR2, and wherein LTR2 is encoded accordingly.

[0051] There are benefits of the method on both the encoder side and the decoder side. On the encoder side, memory and processing efficiency are improved because less data needs to be processed at one time. On the decoder side, relevant LTR frames are retained for decoding, which can be beneficial because only parts of the LTR frames need to be updated for each GOP, rather than the entire LTR frame. This benefit on the decoder side can be achieved if only information about the image data blocks to be updated is sent from the image encoder 216 to the image decoder 234. In addition, one benefit on the decoder side is that better LTR frames are generated faster, resulting in lower bit rates and therefore (usually) higher decoding performance. If the LTR frames are updated too slowly, then slow scene changes may eliminate most of the LTR gain. As an illustrative example, referring to a scene depicting a parking lot, according to an embodiment disclosed herein, vehicles parked in the parking lot during the day can be quickly incorporated into the LTR frame, while for the default algorithm, it may take a whole day to include the vehicle, and it may not be included until the vehicle finally leaves the parking lot.

[0052] Embodiments relating to further details of LTR frame updating in a video encoding process performed by the image processing apparatus 210 , 600 will now be disclosed.

[0053] There may be different examples of cost criteria. Different embodiments related thereto will now be described in turn.

[0054] In some aspects, the cost criterion considers costs evaluated for several image frames. That is, in some embodiments, the cost criterion is related to costs evaluated for at least two of the plurality of frames I1, I2, ..., IK. In this way, trends in costs for, for example, encoding image data blocks ID1, ID2, ..., IDK can be evaluated. In this way, image data blocks ID1, ID2, ..., IDK for which costs are being increased can be prioritized.

[0055] In particular, in some aspects, the cost criterion is met when the cost increases or decreases at a certain speed or amount. In more detail, image area blocks that are highly encoded using LTR frames can generally be encoded at low cost, while image area blocks that are encoded using intra-frame blocks rather than reference LTR frames result in high costs to a large extent. By identifying the areas in the LTR frames for which the encoding cost is being increased, it is concluded that the LTR frames are referenced to a lesser extent. Therefore, in some embodiments, the cost criterion is met when the cost is limited to a predetermined continuous range at a speed at which it increases or decreases from one evaluation of the cost to the next evaluation of the cost or in a series of adjacent evaluations of the cost. In this regard, the speed is non-zero (i.e., positive or negative). That is, in some embodiments, the predetermined continuous range is composed of points, and any point in the predetermined continuous range is at least greater than a predetermined distance from zero.

[0056] In some aspects, if the cost is continuously high or continuously low (i.e., the amount is high or low, and the speed is close to zero), the cost standard is not met. In more detail, in some examples, image data blocks whose encoding costs are continuously high are not given priority. These image data blocks may have objects that change and move therein, such as depicting a busy crosswalk, which makes them difficult to encode with reference to the LTR frame. Since these image data blocks of the LTR frame are not referenced, they do not need to be useful or up-to-date. In addition, since the pixel changes greatly, it may not be updated based on the image background part or the static image part. Therefore, the update may not only be unnecessary, but also may degrade the LTR frame. Therefore, in some embodiments, when the amount of the cost is higher than a first threshold or lower than a second threshold, wherein the second threshold is lower than the first threshold, and the cost is within a predetermined distance to zero at the speed at which it rises or decreases from one evaluation of the cost to the next evaluation of the cost or in the evaluation of a series of adjacent costs, the cost standard is not met. That is, in some aspects, if the cost of encoding is continuously high, the cost standard is not met. In other words, in some embodiments, when the cost on the evaluation of a series of adjacent costs is higher than the first threshold, the cost standard is not met.

[0057] In some aspects, the cost criterion is not met if the cost flickers between a high level and a low level. That is, in some embodiments, the cost criterion is not met when the cost flickers between a first value and a second value at a rate that increases or decreases from one evaluation of the cost to the next evaluation of the cost or over a series of adjacent evaluations of the cost, wherein the first value and the second value are greater than a predetermined distance from each other. By having the first value and the second value be greater than a predetermined distance from each other, this ensures that the flickering behavior of the cost is captured.

[0058] There can be different ways to calculate the cost for encoding the image data blocks ID1, ID2, ..., IDK. Different embodiments related thereto will now be described in turn. In general, the encoding cost of the image data blocks ID1, ID2, ..., IDK represents how efficiently the image data blocks ID1, ID2, ..., IDK can be encoded.

[0059] In some aspects, the cost for encoding the image data blocks ID1, ID2, ..., IDK is based on the extent to which the LTR frames can be used. That is, in some embodiments, the cost for encoding the image data blocks ID1, ID2, ..., IDK is related to how much of the first LTR frame LTR1 can be used to encode the image data blocks ID1, ID2, ..., IDK. Then, the higher the amount of the first LTR frame LTR1 that can be used to encode the image data blocks ID1, ID2, ..., IDK, the lower the cost for encoding the image data blocks ID1, ID2, ..., IDK.

[0060] In some aspects, the cost for encoding image data blocks ID1, ID2, ..., IDK is based on the priority of image data blocks ID1, ID2, ..., IDK. An example of priority involves determining the priority of adjacent zones for updating. In particular, in some aspects, the cost for encoding image data blocks ID1, ID2, ..., IDK is based on whether adjacent image data blocks ID1, ID2, ..., IDK have been updated recently. Image data blocks ID1, ID2, ..., IDK for which adjacent image data blocks ID1, ID2, ..., IDK have been updated recently are given priority. By selecting such image data blocks ID1, ID2, ..., IDK, the time when adjacent image data blocks ID1, ID2, ..., IDK will be updated is very close, thereby reducing the risk of sharp changes occurring at the boundaries between adjacent image data blocks ID1, ID2, ..., IDK. That is, in some embodiments, the cost for encoding the image data blocks ID1, ID2, ..., IDK is related to whether the first LTR frame LTR1 has been updated with the image data ID1, ID2, ..., IDK corresponding to another image data block ID1, ID2, ..., IDK adjacent to the image data block. Then, the more adjacent image data blocks ID1, ID2, ..., IDK with which the first LTR frame LTR1 has been updated, the higher the priority of the image data blocks ID1, ID2, ..., IDK to be preferentially encoded.

[0061] Next, there is the aspect of how often the evaluation in step S106 - 2 and the updating in step S106 - 4 are to be performed.

[0062] Here, note that not every data block in each frame I1, I2, ..., IK needs to be evaluated. The data block does not need to correspond to the coding block structure. The data block can have any size and shape. Larger data blocks will provide a higher effect for each update, and smaller data blocks can have the benefit of finding better candidates for updating. For example, it will be obvious that if a car is parked in a part of the road and the data block is so small that it is covered by the car, the data block covering the part needs to be updated. In this case, since the cost of encoding the data block will obviously rise, the data block can be identified as a priority update. For larger data blocks, since the data block also covers the part next to the car, the increase in cost may not be so obvious.

[0063] In some aspects, the evaluation in step S106-2 and the updating in step S106-4 are performed according to each individual frame. However, the cost criterion can still be based on the cost evaluated for several frames. Therefore, in some embodiments, the evaluation and updating are performed according to each sequence of multiple frames I1, I2, ..., IK. In some aspects, the evaluation and updating are performed simultaneously on a group of frames. Therefore, in some embodiments, the evaluation and updating are performed according to each group sequence of multiple frames I1, I2, ..., IK using a sliding window, and for each occurrence of the evaluation and updating, the sliding window covers a corresponding group of frames. The size of the sliding window can depend on the size of the GOP. In some illustrative examples, the sliding window covers at least two frames, but is not longer than half the size of the GOP.

[0064] Furthermore, in this regard, there may be different relationships between LTR and a GOP. In some examples, there is one LTR frame per GOP. That is, in some embodiments, a first LTR frame LTR1 and a plurality of frames I1, I2, ..., IK that directly or indirectly reference the first LTR frame LTR1 constitute a first GOP, and a second LTR frame LTR2 constitutes the beginning of a second GOP. However, in other examples, there are several GOPs per LTR frame.

[0065] Next reference Figure 5 , Figure 5 Schematically illustrates an example sequence 500a, 500b, 500c, 500d of LTR frames, P frames, B frames and F frames according to an embodiment. In this regard, F frames can be considered as P frames referring to LTR frames. Therefore, F frames provide a shortcut to LTR frames. Other P frames can also refer to LTR frames.

[0066] During encoding of non-LTR frames of a GOP, the method according to the invention sequentially updates LTR frames to adapt to changing conditions in the scene so that the next LTR frame sent is useful for encoding of the next GOP.Selected image areas of LTR frames can be updated at the decoder side.

[0067] In at least some of these example sequences 500a:500d, the LTR frame is an I frame with a tag indicating that it is an LTR frame. The tag informs the image decoder 234 that since the LTR frame will be used as a reference during decoding, the image decoder 234 should retain the frame until the next LTR frame is received. In contrast, a normal I frame (i.e., an I frame that is not marked as an LTR frame) is only retained for decoding the next P frame or B frame.

[0068] According to sequence 500a, P frames P1 and P4 reference an LTR frame (here, referred to as LTR1), while the remaining P frames only reference their respective previous P frames. The second LTR frame may follow the last P frame (i.e., after P6) and is based on the image data of LTR1 and any or all of P1 to P6.

[0069] According to sequence 500b, P frame P1 refers to an LTR frame (herein, referred to as LTR1), while P frame P2 refers only to P1 and B frame B1 refers to both P1 and P2. A second LTR frame may follow the last P frame (i.e., after P2) and be based on image data of LTR1 and image data of any one or all of P1, P2, and B1.

[0070] According to sequence 500c, P frame P1 and B frames B1 and B2 refer to an LTR frame (herein, referred to as LTR1). In addition, B1 refers to P1, and B2 refers to B1. The second LTR frame may follow the last B frame (i.e., after B2) and is based on the image data of LTR1 and any or all of P1, B1, and B2.

[0071] According to sequence 500d, P frame P1 and F frames F1 and F2 refer to an LTR frame (herein, referred to as LTR1), while the remaining P frames refer only to their respective previous frames (which are P frames or F frames). The second LTR frame may follow the last F frame (i.e., after F2) and is based on the image data of LTR1 and any or all of F1, F2, P1, P2, P3, and P4.

[0072] Figure 6 The components of the image processing apparatus 600 according to the embodiment are schematically illustrated from the perspective of a plurality of functional units. Figure 7 The processing circuit 610 is provided by any combination of one or more of a suitable central processing unit (CPU), a multiprocessor, a microcontroller, a digital signal processor (DSP), etc., which executes software instructions in the form of a processor (such as in the example of FIG. 1 ). The processing circuit 610 may further be provided as at least one application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0073] In particular, the processing circuit 610 is configured to cause the image processing apparatus 600 to perform a set of operations or steps as disclosed above. For example, the storage medium 630 may store the set of operations, and the processing circuit 610 may be configured to retrieve the set of operations from the storage medium 630 to cause the image processing apparatus 600 to perform the set of operations. The set of operations may be provided as a set of executable instructions.

[0074] Thus, the processing circuit 610 is thereby arranged to perform the method as disclosed herein. The storage medium 630 may also include any single or combination of permanent storage, which may be, for example, magnetic storage, optical storage, solid-state storage, or even remotely mounted storage. Figure 1 As in the above, the image processing device 600 may further include a communication (comm.) interface 620 configured at least for communicating with other entities, functions, nodes and devices. Therefore, the communication interface 620 may include one or more transmitters and receivers including analog and digital components. The processing circuit 610 controls the general operation of the image processing device 600, for example, by sending data signals and control signals to the communication interface 620 and the storage medium 630, by receiving data and reports from the communication interface 620, and by retrieving data and instructions from the storage medium 630. Other components of the image processing device 600 and related functions are omitted to avoid confusing the concepts presented in this article.

[0075] The image processing device 210, 600 may be provided as a stand-alone device or as part of at least one additional device. A first portion of the instructions executed by the image processing device 210, 600 may be executed in a first device, and a second portion of the instructions executed by the image processing device 210, 600 may be executed in a second device; the embodiments disclosed herein are not limited to any particular number of devices on which the instructions executed by the image processing device 210, 600 may be executed. Therefore, the method according to the embodiments disclosed herein is suitable for execution by the image processing device 210, 600 residing in a cloud computing environment. Therefore, although in Figure 6 A single processing circuit 610 is illustrated in FIG, but the processing circuit 610 may be distributed among multiple devices or nodes. The same applies to Figure 7 Computer program 720.

[0076] Figure 7 An example of a computer program product 710 including a computer-readable storage medium 730 is shown. On the computer-readable storage medium 730, a computer program 720 can be stored, which can cause the processing circuit 610 and entities and devices operably connected thereto (e.g., the communication interface 620 and the storage medium 630) to perform methods according to the embodiments described herein. Therefore, the computer program 720 and / or the computer program product 710 can provide means for performing any steps disclosed herein.

[0077] exist Figure 7In the example of , the computer program product 710 is illustrated as an optical disc such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-ray disc. The computer program product 710 may also be implemented as a memory such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or an electrically erasable programmable read-only memory (EEPROM), and more specifically, as a non-volatile storage medium of the device in an external memory such as a USB (universal serial bus) memory or a flash memory such as a compact flash. Therefore, although the computer program 720 is schematically shown here as a track on the depicted optical disc, the computer program 720 can be stored in any manner suitable for the computer program product 710.

[0078] The inventive concept has mainly been described above with reference to a few embodiments. However, as readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept as defined by the appended patent claims.

Claims

1. A method for updating a long-term reference (LTR) frame in a video encoding process, wherein: The method is performed by an image processing device (210, 600), and wherein, as part of the video encoding process, the method comprises: Encoding (S102) the first LTR frame (LTR1); encoding (S104) a plurality of frames (I1, I2, ..., IK) directly or indirectly referencing the first LTR frame (LTR1); The first LTR frame (LTR1) is updated (S106) in sequence through the following steps: evaluating (S106-2) a cost for encoding an image data block (ID1, ID2, ..., IDK) in one of the plurality of frames (I1, I2, ..., IK); and When the cost meets the cost criterion, updating (S106-4) the image area (IA1, IA2, ..., IAK) in the first LTR frame (LTR1), wherein the image area (IA1, IA2, ..., IAK) is updated based on the image data block (ID1, ID2, ..., IDK) in at least one frame of the plurality of frames (I1, I2, ..., IK); and The sequentially updated first LTR frame (LTR1 Update K) is encoded (S108) as a second LTR frame (LTR2).

2. The method according to claim 1, wherein: The cost criterion is related to the costs evaluated for at least two of the plurality of frames (I1, I2, . . . , IK).

3. The method according to claim 1, wherein: The cost criterion is met when the rate at which the cost increases or decreases from one evaluation of the cost to the next evaluation of the cost or over a series of adjacent evaluations of the cost is constrained within a predetermined continuous range.

4. The method according to claim 3, wherein: The predetermined continuous range consists of points, and wherein any point in the predetermined continuous range is at least at a distance from zero greater than a predetermined distance.

5. The method according to claim 1, wherein: The cost criterion is failed to be met when the amount of the cost is above a first threshold or below a second threshold, wherein the second threshold is lower than the first threshold, and the rate at which the cost increases or decreases from one assessment of the cost to the next assessment of the cost or in a series of adjacent assessments of the cost is within a predetermined distance of zero.

6. The method according to claim 1, wherein: The cost criterion is failed to be met when the cost flickers between a first value and a second value at a rate that increases or decreases from one assessment of the cost to the next assessment of the cost or in a series of adjacent assessments of the cost, wherein the first value and the second value are greater than a predetermined distance from each other.

7. The method according to claim 1, wherein: When the costs in a series of adjacent evaluations of the costs are all above a first threshold, the cost criterion is not met.

8. The method according to claim 1, wherein: The cost for encoding the image data blocks (ID1, ID2, ..., IDK) is related to how much of the first LTR frame (LTR1) can be used to encode the image data blocks (ID1, ID2, ..., IDK), and wherein, the higher the amount of the first LTR frame (LTR1) that can be used to encode the image data blocks (ID1, ID2, ..., IDK), the lower the cost for encoding the image data blocks (ID1, ID2, ..., IDK).

9. The method according to claim 1, wherein: The costs for encoding the image data blocks (ID1, ID2, ..., IDK) are based on the priority order of the image data blocks (ID1, ID2, ..., IDK).

10. The method according to claim 9, wherein: The cost for encoding the image data block (ID1, ID2, ..., IDK) is related to whether the first LTR frame (LTR1) has been updated with image data (ID1, ID2, ..., IDK) corresponding to another image data block (ID1, ID2, ..., IDK) adjacent to the image data block, and wherein, the more adjacent image data blocks (ID1, ID2, ..., IDK) having image data (ID1, ID2, ..., IDK) with which the first LTR frame (LTR1) has been updated, the higher the priority with which the image data block (ID1, ID2, ..., IDK) is encoded.

11. The method according to claim 1, wherein: The evaluating and the updating are sequentially performed according to each of the plurality of frames (I1, I2, . . . , IK).

12. The method according to claim 1, wherein: Said evaluating and said updating are sequentially performed according to each group of said plurality of frames (I1, I2, ..., IK) using a sliding window, said sliding window covering, for each occurrence of said evaluating and said updating, a corresponding group of said frames.

13. The method according to claim 1, wherein: The first LTR frame (LTR1) and the plurality of frames (I1, I2, ..., IK) directly or indirectly referring to the first LTR frame (LTR1) constitute a first group of pictures GOP, and wherein the second LTR frame (LTR2) constitutes the beginning of a second GOP.

14. An image processing device (210, 600) for updating a long-term reference (LTR) frame in a video encoding process, the image processing device (210, 600) comprising a processing circuit (610), the processing circuit being configured to enable the image processing device (210, 600), as part of the video encoding process: Encode the first LTR frame (LTR1); encoding a plurality of frames (I1, I2, ..., IK) directly or indirectly referring to the first LTR frame (LTR1); The first LTR frame (LTR1) is updated sequentially through the following steps: evaluating a cost for encoding a block of image data (ID1, ID2, ..., IDK) in one of the plurality of frames (I1, I2, ..., IK); and When the cost meets the cost standard, the image area (ID1, ID2, ..., IDK) in the first LTR frame (LTR1) is updated, wherein, updating the image area (IA1, IA2, ..., IAK) based on the image data block (ID1, ID2, ..., IDK) in at least one frame of the plurality of frames (I1, I2, ..., IK); and The sequentially updated first LTR frame (LTR1 update K) is encoded as a second LTR frame (LTR2).

15. A computer program (720) for long term reference (LTR) frame updating in a video encoding process, the computer program comprising computer code which, when executed on a processing circuit (610) of an image processing device (210, 600), causes the image processing device (210, 600), as part of the video encoding process: Encoding (S102) the first LTR frame (LTR1); encoding (S104) a plurality of frames (I1, I2, ..., IK) directly or indirectly referencing the first LTR frame (LTR1); The first LTR frame (LTR1) is updated (S106) in sequence through the following steps: evaluating (S106-2) a cost for encoding an image data block (ID1, ID2, ..., IDK) in one of the plurality of frames (I1, I2, ..., IK); and When the cost meets the cost standard, the image areas (IA1, IA2, ..., IAK) in the first LTR frame (LTR1) are updated (S106-4), wherein: updating the image area (IA1, IA2, ..., IAK) based on the image data block (ID1, ID2, ..., IDK) in at least one frame of the plurality of frames (I1, I2, ..., IK); and The sequentially updated first LTR frame (LTR1 Update K) is encoded (S108) as a second LTR frame (LTR2).

Images