Adaptive intra refresh coding of video streams

Abstract

Adaptive intra-frame refresh coding for video streams is provided. Specifically, a method (500) for intra-frame refresh coding of video streams is provided. The method includes: encoding the video stream using a first intra-frame refresh mode (S510); determining (S520) that the time variation of the number of bits per frame for encoding the video stream using a second intra-frame refresh mode different from the first intra-frame refresh mode is lower than the time variation of the number of bits per frame for encoding the video stream using the first intra-frame refresh mode; and switching (S530) to encoding the video stream using the second intra-frame refresh mode. A corresponding apparatus configured to perform the method, as well as a corresponding computer program and computer program product, are also provided.

Description

Technical Field

[0001] This disclosure generally relates to the field of video coding. More specifically, this disclosure relates to adaptive intra-frame refresh coding of video streams. Background Technology

[0002] In modern video compression / coding techniques, different types of video frames are used to take advantage of the fact that, for example, the data needed to decode a video frame is often found, at least partially, in one or more other already decoded video frames. Examples of such video frame types, such as those used in the H.264 coding standard, are predictive frames (P-frames) and bidirectional predictive frames (B-frames), also known as inter-frame frames. Additionally, there are so-called intra-frame frames (I-frames), which are self-contained and do not reference any other video frames. I-frames are typically inserted into the encoded video stream at regular intervals to, for example, prevent errors introduced by multiple inter-frame frames referencing each other (in a chain-like manner) from growing indefinitely, and to provide a way to "reset" the video stream on a regular basis. The use of I-frames also helps prevent errors introduced due to, for example, packet loss or similar situations, from remaining in the decoded video stream for extended periods.

[0003] However, a common problem with the mixing of inter-frames and intra-frames in a video stream is that each I-frame can cause bitrate spikes because each I-frame requires more bits to transmit compared to, for example, P-frames and / or B-frames. For low-latency video streams, these bitrate spikes can be particularly undesirable, and encoders used for such applications are therefore forced to avoid periodically introducing I-frames into their output encoded video streams. This can, over time, worsen issues such as no-boundary errors due to inter-frame coding and artifacts remaining on the screen, such as those caused by missing data tables.

[0004] One solution to this problem is to instead encode only a portion of each video frame (e.g., only one or a few macroblocks) as self-contained (called I-blocks), distributing each I-frame across multiple video frames and changing the portion of the I-block encoded as consecutive video frames. This technique is called "intra-frame refresh coding," where the intra-frame refresh pattern is used to define how the portion of a video frame encoded as an I-block moves over time. The pattern is typically periodic, causing the same portion of the frame image to reappear every [time period]. Each video frame will be encoded again into an I-block, where... This can be referred to as the period of intra-frame refresh. Since each video frame will then include both I-blocks and blocks encoded using prediction (e.g., P-blocks and / or B-blocks), the bit rate remains more constant over time. US 2020 / 228823 A1 discloses a solution in which intra-frame refresh coding is used in conjunction with Overlapping Block Motion Compensation (OBMC). WO 2020 / 188149 A1 discloses a solution for intra-frame refresh coding of images using a diagonal motion pattern. CN 11294923 A discloses a solution for intra-frame refresh coding of different sub-image groups.

[0005] However, especially for low-latency applications, even with this intra-frame refresh coding, there may still be oscillating behavior in terms of bit rate (e.g., the number of bits per frame). Summary of the Invention

[0006] This disclosure aims to further develop contemporary technology and provide solutions that at least partially overcome the aforementioned problems.

[0007] According to a first aspect of this disclosure, a (computer-implemented) method for intra-frame refresh coding of a video stream is provided. The method includes encoding the video stream using a first intra-frame refresh mode. The method includes determining that the time variation in the number of bits per frame for encoding the video stream using a second intra-frame refresh mode different from the first intra-frame refresh mode is lower than the time variation in the number of bits per frame for encoding the video stream using the first intra-frame refresh mode. The method also includes switching to encoding the video stream using the second intra-frame refresh mode (instead of the first intra-frame refresh mode).

[0008] The proposed solution offers the advantage of allowing the intra-frame refresh mode to be changed based on the current scenario, thus providing a more flexible solution in which an intra-frame refresh mode that is superior in terms of bit rate variation can be selected and used, making the final number of bits required for each encoded frame more constant over time. In other words, the proposed solution helps reduce oscillations in the number of bits per frame and is therefore particularly suitable for low-latency applications such as low-latency video streaming, where it may be of great interest to keep the time elapsed between capturing an event and making the same event visible on the screen to, for example, an operator (or making the same event available to computer systems configured to analyze the video stream and, for example, detect the event) as small as possible.

[0009] In one or more embodiments of the method, the first intra-frame refresh mode and the second intra-frame refresh mode can be a horizontal intra-frame refresh mode and a vertical intra-frame refresh mode, respectively, and vice versa. In other words, the first mode can be a horizontal mode and the second mode can be a vertical mode, or the first mode can be a vertical mode and the second mode can be a horizontal mode. Switching between horizontal and vertical modes can be useful, for example, for cameras that are sometimes expected to rotate 90 degrees, such as wearable cameras, drone cameras, or similar cameras mounted on non-stationary objects, where the camera is tilted and / or rotated, for example, due to the tilt and / or rotation of the object on which the camera is mounted, and / or the camera is tilted and / or rotated relative to the object, such as by using a suitable camera gimbal. A 90-degree rotation may be caused, for example, by changing from landscape mode to portrait mode.

[0010] In one or more embodiments of the method, the first intra-frame refresh mode and the second intra-frame refresh mode can be two orthogonal intra-frame refresh modes. Here, "orthogonal" means that if one mode involves scanning a region across (some) images in a first direction, then a mode orthogonal to this mode instead includes scanning that region in a direction oriented orthogonal (i.e., perpendicular) to the first direction. Using a set of orthogonal modes can provide a large difference in oscillatory behavior, especially in scenarios, for example, where the complexity (in terms of coding efficiency) differs only in one direction, where the improvement in switching to the new mode can be significant if the new mode is orthogonal to the previously used mode.

[0011] In one or more embodiments of the method, the first intra-frame refresh mode and the second intra-frame refresh mode can be two opposite diagonal intra-frame refresh modes. For example, the first mode may extend from, for example, the lower left corner of the image to the upper right corner of the image, and the second mode may extend from, for example, the upper left corner of the image to the lower right corner of the image, and so on. Using a set of such modes can be advantageous, for example, in scenarios where the complexity (in terms of coding efficiency) remains more constant along one diagonal of the image than the other.

[0012] In one or more embodiments of the method, a first intra-frame refresh mode and a second intra-frame refresh mode may form part of a plurality of different intra-frame refresh modes, and the determination operation may include evaluating the final temporal variation of the number of bits per frame for each of the (different) plurality of intra-frame refresh modes. Using multiple modes to select from can increase the chance of finding the best mode for the current scene.

[0013] In one or more embodiments of the method, a second intra-frame refresh mode can be determined as (among a plurality of different intra-frame refresh modes) an intra-frame refresh mode having the minimum temporal variation in the number of bits per frame used to encode the video stream. Given a set of different available modes, this can provide the maximum possible reduction in the variation of the number of bits per frame over time.

[0014] In one or more embodiments of the method, the temporal variation of the number of bits per frame can be defined based on the bit variance per frame, the standard deviation of the number of bits per frame, and / or the oscillation amplitude of the number of bits per frame.

[0015] The switching (to the second intra-frame refresh mode) is performed in response to this determination. In other words, the method involves first determining that the second mode is better than the first mode, and then switching to use the second mode. This determination can be based, for example, on statistics, on digital models / predictions, and / or, for example, by evaluating the second mode in parallel with the first mode using a separate encoder, to see if the second mode might perform better for the current scene captured in the video stream to be encoded.

[0016] In one or more embodiments of the method, determination and switching may be performed in response to detecting that a temporal variation in the number of bits per frame used to encode the video stream using a first intra-frame refresh mode exceeds a threshold. In other words, the method may cause the determination of a second mode to lead to improvement and switching to the second mode in response to detecting that the first mode is being performed at a lower-than-expected level. The threshold may correspond, for example, to a threshold variation in the number of bits per frame and packet size, and a first mode exceeding such a threshold variation may trigger a switch to the second mode. Of course, as contemplated herein, if the metric used to evaluate the performance of the mode is instead formulated such that it decreases with increasing performance of the mode and increases with decreasing performance of the mode, then "exceeding the threshold" also includes / covers "below the threshold."

[0017] In one or more embodiments of the method, the determination may include, in response to the detection (exceeding a threshold), evaluating the temporal variation of the number of bits per frame for each of a first intra-frame refresh mode and a second intra-frame refresh mode over one or more finite time intervals. This determination (preferably the second mode) may be performed by detecting that, during at least one of the one or more finite time intervals, the temporal variation of the number of bits per frame for the second intra-frame refresh mode is lower than the temporal variation of the number of bits per frame for the first intra-frame refresh mode. For example, the determination may include, for instance, encoding X seconds, X minutes, etc., of the video stream using the first mode and encoding Y seconds, Y minutes, etc., of the video stream using the second mode, and then comparing which mode results in the lowest temporal variation. As contemplated herein, X may be equal to Y or may not be equal to Y, and the portion of the video stream encoded using the first mode may or may not be the same portion of the video stream encoded using the second mode. For example, the method may include first encoding a first portion of the video stream using the first mode and then encoding a second portion of the video stream (different from the first portion) using the second mode, or vice versa. If, for example, multiple encoders are accessible, or an encoder capable of encoding multiple streams in parallel is accessible, then the same portion of the video stream can be analyzed / evaluated for both modes, and so on.

[0018] According to a second aspect of this disclosure, an apparatus is provided. The apparatus includes processing circuitry. The processing circuitry is configured to encode a video stream using a first intra-frame refresh mode. The processing circuitry is configured to determine that the time variation in the number of bits per frame for encoding the video stream using a second intra-frame refresh mode different from the first intra-frame refresh mode is lower than the time variation in the number of bits per frame for encoding the video stream using the first intra-frame refresh mode. The processing circuitry is further configured to switch to encoding the video stream using the second intra-frame refresh mode. The apparatus is further configured to perform the operation of the method of the first aspect disclosed herein or any example embodiment thereof. As used herein and as illustrated above, encoding a video stream using the first mode and the second mode may include encoding the same portion of the video stream using each of the modes or encoding different portions of the video stream using these modes.

[0019] In one or more embodiments of the device, the device may be a surveillance camera, such as a camera for monitoring and / or surveillance of a scene. The camera may be configured, for example, to be mounted on a stationary object such as a building. The camera may be “static” in the sense that the field of view (FOV) resulting from the orientation of the mounted camera is fixed, or the camera may be “dynamic” in the sense that the orientation and / or FOV of the camera can be changed as required, for example, by changing the lens arrangement, rotating, tilting and / or zooming the camera.

[0020] In one or more embodiments of the device, the device may be a wearable camera or a drone camera. The envisioned solution may be particularly useful for this type of camera, which is expected to move and change its orientation / FOV over time, because the movement and / or orientation / FOV of the camera can influence which intra-frame refresh mode is most effective in terms of temporal variation in the number of bits per frame.

[0021] According to a third aspect of this disclosure, a computer program is provided. The computer program includes computer code that, when executed on processing circuitry of an apparatus (e.g., the apparatus of the second aspect disclosed herein or any example embodiment thereof), causes the apparatus to: encode a video stream using a first intra-frame refresh mode; determine that the time variation in the number of bits per frame for encoding the video stream using a second intra-frame refresh mode different from the first intra-frame refresh mode is lower than the time variation in the number of bits per frame for encoding the video stream using the first intra-frame refresh mode; and switch to encoding the video stream using the second intra-frame refresh mode. The computer program further causes the apparatus to perform the operations of the methods of the first aspect disclosed herein or any example embodiment thereof.

[0022] According to a fourth aspect of this disclosure, a computer program product is provided. This computer program product includes a computer-readable storage medium on which the computer program of the third aspect is stored. As used herein, the computer-readable storage medium may be, for example, non-transitory and is provided as, for example, a hard disk drive (HDD), a solid-state drive (SSD), a USB flash drive, an SD card, a CD / DVD, and / or any other storage medium capable of non-transitory storage of data. In other embodiments, the computer-readable storage medium may be transient and corresponds, for example, to a signal (electrical, optical, mechanical, or similar) present on, for example, a communication link, a wire, or a similar signal transmission device, in which case the computer-readable storage medium is, of course, more of a data carrier than a data storage entity.

[0023] Other objects and advantages of this disclosure will become apparent from the following detailed description, drawings, and claims. Within the scope of this disclosure, it is contemplated that all features and advantages described with reference to, for example, the method of the first aspect are relevant to, applicable to, and can be used in conjunction with the apparatus of the second aspect, the computer program of the third aspect, and the computer program product of the fourth aspect, and vice versa. Attached Figure Description

[0024] Exemplary embodiments will now be described with reference to the accompanying drawings, in which: Figure 1A The schematic map illustrates an example of an intra-frame refresh process using the vertical scan intra-frame refresh mode; Figure 1B The schematic map illustrates an example of an intra-frame refresh process using the horizontal scan intra-frame refresh mode; Figure 2A The schematic map illustrates example images of regions with varying degrees of complexity in terms of coding efficiency, as envisioned in this paper. Figure 2B Schematic map shows that Figure 2A An example of an image being divided into multiple rows / columns of macroblocks; Figure 3A The schematic map illustrates the use of vertical scan intra-frame refresh mode for... Figure 2A / Figure 2B Example of time-varying packet size (bits per frame) for encoding images; Figure 3B The schematic map illustrates the use of horizontal scan intra-frame refresh mode for... Figure 2A / Figure 2B Example of time-varying packet size (bits per frame) for encoding images; Figure 4 The schematic map illustrates an example of how to define the orientation of the scan direction in an intra-frame refresh mode as envisioned in this article; Figure 5A , Figure 5B and Figure 5C The schematic diagram illustrates flowcharts of various example embodiments of methods for intra-frame refresh coding of video streams as envisioned herein; Figure 6A and Figure 6B The schematic diagram illustrates the components and functional blocks of various example embodiments of the device as envisioned herein; and Figure 7 The schematic diagrams illustrate example embodiments of computer programs, computer program products, and computer-readable storage media as envisioned herein.

[0025] In the accompanying drawings and figures therein, unless otherwise stated, the same reference numerals will be used for the same elements. Unless explicitly stated to the contrary, the drawings only show elements necessary to illustrate exemplary embodiments, while other elements may be omitted or may be implied for clarity. As illustrated, for illustrative purposes, the (absolute or relative) dimensions of elements and regions may be exaggerated or underestimated relative to their true values, and are therefore provided to illustrate the general structure of the embodiments. Detailed Implementation

[0026] Now refer to Figure 1A and Figure 1B Here are some examples of intra-frame refresh encoding for video streams that will be described in more detail.

[0027] Figure 1A The schematic diagram illustrates video frames 110-1, 110-2, 110-3, 110-4, ... of video stream 100. Video stream 100 may include a fixed number of such video frames, for example, a total of A single frame, or it could be a video stream where the total number of video frames increases infinitely as more images are generated, and therefore more and more video frames to be encoded are added over time.

[0028] Each video frame 110-1 to 110-4 is divided into multiple macroblocks 120, where a macroblock can be, for example, a set of a predetermined number of image pixels such as 2×2, 4×4, 8×8, and 16×16 pixels. In this particular example, the macroblocks do not overlap and together form a grid of macroblocks spanning the entire image. In the first video frame 110-1, the first (i.e., top) row 130-1 of the macroblocks is encoded as an I-block, while other macroblocks not belonging to row 130-1 are encoded as inter-frame blocks, such as P-blocks and / or B-blocks. The portion of each video frame / image encoded as an I-block can be referred to as an intra-frame refresh region, and in video frame / image 110-1, the intra-frame refresh region thus corresponds to the macroblock in row 130-1.

[0029] In the second video frame 110-2, the intra-frame refresh region instead corresponds to the second row 130-2; that is, macroblocks belonging to row 130-2 are encoded as I-blocks, while other blocks in the second video frame 110-2 are encoded as inter-frame blocks. Similarly, in the third video frame 110-3, this region is row 130-3 (third from the top). In the fourth video frame 110-4, this region is row 130-4 (fourth from the top). For the next video frame (not shown) of video stream 100, encoding continues similarly, and the region is scanned so that it moves down one line per video frame. After the last line of the video frame, i.e., if there are [a certain number of lines] in each video frame / image... Then the macro block reaches the first In the video frame, the region moves to the top line again, even though the first... This region of a video frame corresponds to the macroblock in the top row of that video frame / image. In other words, each The pattern repeats in each video frame, and therefore the region appears in every pair of video frames. and video frames The same applies in [the context]. Figure 1A In the frame refresh mode, the scanning is vertical because during each cycle of the repeat mode, the position of the I-block region moves down one line. Figure 1A The mode can be called "vertical scan mode", "vertical mode" or similar modes.

[0030] Figure 1BThe schematic map illustrates another possible intra-frame refresh pattern for the same or similar video stream 101, wherein the pattern is replaced by such that the region corresponds to a column of macroblocks 120 for each video frame / image 110-1 to 110-4 (or 110-M). For example, the region of the first video frame 110-1 corresponds to the leftmost column 132-1 of the macroblock, the region of the second video frame 110-2 corresponds to the next column 132-2 of the macroblock, the region of the third video frame 110-3 corresponds to the next column 132-3, and the region of the fourth video frame 110-4 corresponds to column 132-4, and so on. Figure 1B In the example, the intra-frame refresh mode therefore causes the area to be horizontally shifted; that is, the mode is "horizontal scan" and is "horizontal mode," etc. If there are [various parameters] in each video frame / image... Line macroblock, then each The pattern repeats in the video frames, for example, the first... This region of the video frame will be combined with the first Columns with the same columns in the video frame correspond to each other.

[0031] Figure 2A A schematic map illustrates example image 200 of an example scene, where the example scene takes the form of a highway in front of a city. In the scene depicted in image 200, the highway extends horizontally, as does the city in the background. The scene and image 200 can be divided into horizontal regions of varying complexity (in terms of coding efficiency). For example, such a first region 210 mainly comprises static content (e.g., grass) and can be assumed to be less complex in terms of coding efficiency due to its static / homogeneous content. The second region 212 includes the highway, which can be expected to be more complex to encode effectively due to both the presence of the highway's roads / lanes and multiple vehicles, where the positions of the vehicles may change between subsequent video frames. The third region 214 corresponds to a portion of the scene in image 200 capturing a distant city and can be expected to have moderate complexity in terms of coding efficiency because, although the city and its surrounding environment may not change dynamically, they can be intricate. The fourth region 216 corresponds to the sky above the city and can be considered less complex to encode due to its predominantly static and highly homogeneous nature. Of course, clouds and similar phenomena may exist, but it can be assumed, for example, that the shape and / or position of such clouds remain fairly constant over at least a finite number of consecutive video frames. To avoid having to insert complete I-frames at regular intervals, the encoding of image 200 (and the video stream of the scene depicted in image 200) can be performed using intra-frame refresh coding as described earlier in this document.

[0032] Figure 2BThe schematic map illustrates how image 200 (and any other images in the video stream) is divided into macroblocks 220, where, in this particular example, the macroblocks form a grid of macroblocks, allowing macroblocks to be defined with multiple rows and / or multiple columns. Figure 2B In this specific example, for illustrative purposes only, a total of six rows (R1 to R6) and eight columns (C1 to C8) of macroblocks are available. Of course, in other examples, the number of rows and / or columns may differ. Figure 2B The number shown can depend, for example, on the size of each macroblock, the size of the image, and the aspect ratio / scale, etc.

[0033] Figure 3A The schematic map illustrates an attempt to use a vertical scan intra-frame refresh mode (e.g.) Figure 1A The example graph 300 illustrates how the packet size (i.e., bits per frame) generated by encoding image 200 using the vertical scan intra-frame refresh mode changes over time, i.e., how the packet size depends on the number of (video) frames considered. For the first video frame, the intra-frame refresh region corresponds to the first row R1, and results in a relatively low packet size due to the relatively low complexity of region 216 corresponding to the first row R1. For the second video frame, the region has moved to the second row R2, corresponding to the medium-complexity region 214. The same applies to the second video frame where the region has moved to the third row R3. Therefore, the complexity of the second and third video frames is slightly higher than that of the first video frame. For the fourth and fifth video frames, the region corresponds to the fourth row R4 and the fifth row R5, respectively, which in turn correspond to the more complex region 212 of image 200 depicting a highway. Therefore, the packet sizes of video frames four and five are larger than those of the second and third video frames. Finally, for the sixth video frame where the region has moved to the sixth row R6, the complexity is again low due to the low complexity of the corresponding low-complexity region 210 of image 200 / scene, and therefore, the final packet size is again at the level of the first video frame (or even lower). Since there are a total of six rows of macroblocks, the intra-frame refresh mode in this example has... The period, because for the seventh frame the region will move back to the top line R1 and for each Video frames and The video frame area will be the same. This is visible in Figure 300 because the packet size, as a function of the video frame number, exhibits periodicity and has... The period length of each video frame. Due to the considerably different complexities in terms of coding efficiency between different rows of image 200 / scene and their corresponding regions, the amplitude of the oscillating packet size (over time) This is quite large and could result in non-constant (and often quite large) latency, as each video frame will provide a different size data packet due to the varying complexity of encoding the refresh area within each frame.

[0034] This disclosure envisions a solution to this problem in which the existence of a temporal variation (i.e., a smaller magnitude) leading to (or potentially leading to) a lower number of bits per frame is evaluated and determined. One or more other intra-frame refresh modes, and when encoding the video stream, this intra-frame refresh mode is selected instead of (in this example) the vertical intra-frame refresh mode.

[0035] Figure 3B The schematic map illustrates such a second mode (i.e., horizontal scan intra-frame refresh mode, e.g., Figure 1B The graph 301 shows the packet size (i.e., bits per frame) as a function of the video frame number in the horizontal scan intra-frame refresh mode. Here, the region is not moved row by row, but column by column. For the first video frame, this region corresponds to the first column C1. The complexity of the first column C1 is average because, as described earlier in this document, the first column C1 includes image content belonging to all the different (horizontal) regions 210, 212, 214, and 216. For example, column C1 includes a portion of the sky / clouds (region 216), a portion of the city (region 214), a portion of the highway (212), and a portion of the (static) foreground (region 210). This also applies to each of the other columns C2 through C8. Therefore, although the complexity for each column may not be exactly the same due to, for example, one column including more vehicles, stronger contrast between buildings and clouds, more different buildings, etc. than one or more other columns, the difference in complexity in terms of coding efficiency between columns C1 through C8 may be less than the difference in complexity between rows R1 through R6. Therefore, as can be seen in Figure 301, the horizontal scan intra-frame refresh mode also provides some degree of oscillation behavior, namely, non-zero time variation in the number of bits per frame, but this oscillation behavior is more pronounced than the reference. Figure 3A The amplitude of the described vertical scan intra-frame refresh mode is small. It can also be seen that, since there are a total of eight columns, the period length of the oscillating behavior is... That is, for each of the first Video frames and the ( The data packet size, as a function of the video frame number, is approximately [missing information]. In summary, for the specific scene depicted in image 200, the time variation resulting from the switch from the first (vertical) intra-frame refresh mode to the second (horizontal) intra-frame refresh mode (i.e., with a lower magnitude) leads to a reduction in the number of bits per frame. This also results in more uniform packet sizes and more stable latency over time, leading to a better experience for operators, such as those watching decoded video streams.

[0036] Figure 4 The schematic map illustrates how the intra-frame refresh mode is defined based on the direction in which the region is scanned / moved between consecutive video frames, as envisioned herein. In the image 400 to be encoded, the region of the I-macroblock (I-block) 410 is oriented such that it has a primary extension direction (here, from the lower left to the upper right). Perpendicular to this primary extension direction, a scan direction 420 is defined, forming an angle with the vertical direction of the image 400. Direction 420 can be referred to as the scanning direction and propagation direction of the intra-frame refresh mode, etc. Of course, as generally envisioned in this paper, region 410 can also be scanned in direction 422, opposite to direction 420. As envisioned in this paper, the first... Angle of intra-frame refresh mode It can be used to define one or more other patterns. The comparison patterns, because these other patterns will have other perspectives. .For example, Figure 1A and Figure 3A The vertical scan intra-frame refresh mode can be defined as Degree or by Degree definition, and Figure 1B and Figure 3B The horizontal scan intra-frame refresh mode can be defined as degree (or (Radians). Its angle. and Two modes that differ by 90 degrees can be referred to as two orthogonal intra-frame refresh modes, and so on. For this... This mode can be called the vertical intra-frame refresh mode, and for it... The mode can be referred to as a horizontal intra-frame refresh mode, etc. As envisioned in this article, the two modes (e.g., mode one and mode two) do not necessarily have to differ by 90 degrees, but can use any finite angle difference, as long as... .

[0037] Figure 5AThe schematic diagram illustrates a flowchart of an example embodiment of a method 500 for intra-frame refresh coding of a video stream, as contemplated herein, utilizing the observations made above. As part of operation S510, method 500 includes encoding the video stream using a first intra-frame refresh mode. As part of operation S520, method 500 includes determining that the temporal variation in the number of bits per frame encoded using a second intra-frame refresh mode is lower than the temporal variation in the number of bits per frame encoded using the first intra-frame refresh mode. As part of operation S530, method 500 includes switching to encoding the video stream using the second intra-frame refresh mode. As contemplated herein, operation S530 is performed in response to operation S520. For example, it is first determined that switching to the second mode will reduce the temporal variation, and in response to such determination, the second mode is switched.

[0038] As envisioned in this paper, one or more suitable metrics can be used to determine / evaluate the temporal variation of the number of bits per frame for a particular intra-frame refresh pattern. One example of such a metric can be obtained by performing a frequency analysis of the number of bits per frame as a function of time (e.g., packet size) and by studying one or more amplitudes of one or more dominant frequency components. For example, in Figure 3A In this context, Figure 300 can be approximated as a sine curve of the fundamental frequency, with some additional high-frequency components. The time variation can be assumed to be represented by the amplitude of the frequency components corresponding to this fundamental frequency, and the amplitude can be provided, for example... The instructions. For Figure 3B In Chart 301, the frequency can be different (because there are eight columns instead of six rows), but the amplitude corresponding to this fundamental frequency will likely be lower than that in Chart 300, and therefore used as... Figure 3B The (second) mode is better than the mode used in terms of temporal variation in the number of bits per frame. Figure 3A The (first) mode is a good instruction.

[0039] In other examples, statistical analysis can be used to obtain a suitable metric. For instance, the average packet size and the standard deviation / variance of this average can be obtained by collecting data on how packet size changes over time, and the standard deviation (or variance) can be used as a measure of the time variation in the number of bits per frame. If used again... Figure 3A and Figure 3B Using Figures 300 and 301 as examples, the average packet size in Figures 300 and 301 can be roughly the same. However, the variance (or standard deviation) in Figure 301 may be smaller because the packet size values ​​in Figure 301 oscillate around the average less than those in Figure 300. Based on this, the appropriate method can be determined. Figure 3BThe (second) mode is better than the one used for (low) time variation. Figure 3A The mode performs well. Other examples of how to define and compare the temporal variation of the number of bits per frame for different intra-frame refresh modes are of course possible, and all of these examples are envisioned to be usable in the context of this disclosure.

[0040] For example, the packet rate as a function of the video frame number. A set of measurements / values Variance can be calculated as in, It is the average data packet rate, which is obtained as Furthermore, the corresponding (biased) standard deviation is obtained as follows: In other examples, unbiased variance can be used instead, which is corrected using Bessel. replace Items in the expression This can be obtained using the corrected sample standard deviation (obtained by applying a Bézier correction to the standard deviation of the sample). (obtained by taking the square root), or by applying contemporary knowledge based on... The unbiased sample standard deviation is defined by the assumed probability distribution of the value.

[0041] Figure 5B The schematic diagram illustrates additional example embodiments of method 501, such as method 500 which includes operations S510, S520, and S530, but adds an embodiment of determination (S520) and switching (S530) that can be performed in response to operation S512, which first detects (e.g., established by any suitable metric as described above) that a temporal change in the refresh mode within a first frame exceeds a threshold. For example, the threshold may be related to an amplitude value. Correspondingly, operation S512 may include detection. (in, For example, it can be achieved by executing values. (This can be obtained through frequency analysis such as Fourier analysis, Laplace analysis, Z-transform analysis, etc.). If, for example, standard deviation and / or variance are used as a measure of time variation, the threshold could be, for example, the standard deviation value. Furthermore, operation S512 may include detecting the standard deviation of the data packet size. Make Similarly, if variance is used as a measure of time variation, then the threshold can be defined, for example, as... And operating S512 may include detection .

[0042] Figure 5C The schematic diagram illustrates additional example embodiments of method 502, such as embodiments of method 500 or method 501, wherein operation S520 includes a sub-operation S522 of selecting a second intra-frame refresh mode by evaluating a plurality of different intra-frame refresh modes (based on their temporal variation performance). For example, operation S522 may include evaluating the same or different portions of the video stream for different modes. For example, the same portion of the video stream may be encoded using different modes. As another example, different portions of the video stream may be encoded using different modes. For example, if multiple encoders are accessible or at least one encoder capable of encoding multiple streams in parallel is accessible, the same portion of the video stream may be encoded using each of the plurality of different modes. In other examples, different modes may be used to encode each of several different (e.g., consecutive) portions of the video stream. After the different modes have been evaluated, the second mode may, for example, be selected as the mode (among the available modes) that results in the lowest temporal variation in the number of bits per frame. In other examples, other preferences may be used. For example, if a pattern simultaneously results in an average packet size higher than the desired average packet size, then choosing the pattern that causes the lowest time variation may not be optimal. In this case, there may be one or more other patterns that cause, for example, slightly similar (but higher similarity) time variations but result in a lower average packet size than the pattern causing the lowest time variation. In this case, it may be desirable to choose one of these one or more patterns instead as the second intra-frame refresh mode.

[0043] Method 502 may also optionally include operation S523, which includes evaluating the intra-frame refresh mode (among multiple modes) during one or more finite intervals. For example, each mode may be for a total The evaluation is performed using individual video frames, among which... It is the index of the corresponding pattern. In some examples, Possibly for all They are all equal, but in other examples, Possibly for different They are different. Similarly, as described earlier in this document, all modes can be evaluated based on the same portion of the video stream, or these modes can be evaluated based on different portions of the video stream. For example, in some embodiments, it is envisioned that each mode can be evaluated over a period of, for example, X seconds or X minutes (for the same or different portions of the video stream), and then compared to see, for example, which of several modes resulted in the lowest time variation, the lowest combined time variation, and the average packet size (or bits per frame), and then, if that mode performs better than the first mode already tried, it can be selected as the second mode. In other examples, multiple test runs can be performed, for example, such that each mode can be evaluated for different lengths and / or portions of the video stream, and then compared to determine which mode (if any) will be selected as the second mode.

[0044] As already described herein, operation S520 of method 502 can be triggered by operation S512 (i.e., by detecting that the performance of the first mode is lower than the minimum expected value, for example, a certain measure of the time variation of the first mode exceeds a threshold).

[0045] This document also envisions providing apparatus capable of performing at least the above-described method 500 (and optionally one or more of method 501 and method 502), as well as computer programs and computer program products for distributing and performing such methods.

[0046] Figure 6A The schematic diagram illustrates an apparatus 600 for performing the methods envisioned herein (i.e., configured to perform reference). Figure 5A (and / or Figure 5B and / or Figure 5CThis refers to one or more examples of an apparatus (e.g., a camera) that performs the methods 500 (and / or 501 and / or 502) described herein. Apparatus 600 includes at least a processor (or “processing circuitry”) 610 and optionally a memory 612. As used herein, a “processor” or “processing circuitry” can be, for example, any combination of one or more suitable central processing units (CPUs), multiprocessors, microcontrollers (µCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), etc., capable of executing software instructions stored in memory 612. Memory 612 may be external to processor 610 or internal to processor 610. As used herein, “memory” can be any combination of random access memory (RAM) and read-only memory (ROM) or any other type of memory capable of storing instructions. Memory 612 contains (i.e., stores) instructions that, when executed by processor 610, cause apparatus 600 to perform the methods described herein (i.e., method 500 or any embodiment thereof). The apparatus 600 may further include one or more additional items 614 that may be useful in some cases for performing the method. In some example embodiments, the apparatus 600 may be, for example, a (video) camera such as a (video) surveillance camera, a wearable camera, a drone camera, etc., and (some) additional items 614 may further include, for example, an image sensor and one or more lenses, for example, for focusing light from the scene onto the image sensor so that the surveillance camera can capture an image of the scene (as part of performing the contemplated method). (Some) additional items 614 may also include, for example, various other electronic components required for capturing the scene (e.g., for the proper operation of the image sensor and / or lenses as desired). Performing the method in a surveillance camera can be useful because processing is moved to the "edge," i.e., closer to where the actual scene is captured compared to performing, for example, image analysis elsewhere (e.g., at a more centralized processing server, etc.).

[0047] Device 600 may be connected to a network, for example, to transmit the results of performing the method to, for example, a user / operator and / or to another device such as a server. For this purpose, device 600 may include a network interface 616, which may be, for example, a wireless network interface (as defined in any of the IEEE 802.11 or subsequent standards supporting, for example, Wi-Fi) or a wired network interface (as defined in any of the IEEE 802.3 or subsequent standards supporting, for example, Ethernet). Network interface 616 may also support, for example, any other wireless standard capable of transmitting encoded video, such as Bluetooth. Various components 610, 612, 614, and 616 (if present) may be connected via one or more communication buses 620, enabling these components to communicate with each other and exchange data as needed.

[0048] Device 600 may be, for example, a surveillance camera mounted or installable on a building or other supporting structure, such as a PTZ camera or fisheye camera capable of providing a wider field of view of the scene, or any other type of surveillance / monitoring camera. Device 600 may be, for example, a body camera, action camera, or dashcam suitable for mounting on people, animals, and / or various vehicles, etc. Device 600 may be, for example, a drone or drone camera capable of acquiring images from above. Device 600 may be, for example, a smartphone or tablet that a user can carry and photograph a scene. In any such example of device 600, it is contemplated that device 600 may include all necessary components (if any) other than those already known herein, provided that device 600 is still capable of performing method 500 or any embodiment thereof as contemplated herein. In some examples, the various components of device 600 may be further configured to implement the method operation as described herein (e.g., at least S510, S520, and S530). In other examples, device 600 may be distributed across multiple physical and / or logical entities to form, for example, a computer system, wherein two or more operations (and / or two or more different sub-operations of the same operation) may be executed on or by different physical and / or logical entities, for example, as part of a distributed computing process.

[0049] Figure 6B One or more embodiments of the device 600 are schematically illustrated based on multiple functional / computational blocks 610a, 610b, 610c. Figure 5AAs shown in the flowchart, each of these blocks 610a to 610c is responsible for performing a function according to a specific operation of method 500. For example, one such functional block 610a can be configured to encode a video stream using a specific mode (e.g., in operation S510). Block 610a can be referred to as an encoding block / encoding module and encoder, etc. Another block 610b can be configured to perform (as in operation S520) a determination that the second mode performs better than the first mode. Block 610b can be referred to as a determination block / determination module and determiner, etc. Another block 610c can be configured to switch the first mode to the second mode (as in operation S530), for example, by providing instructions to block 610a. The apparatus 600 may optionally include, for example, one or more additional functional blocks such as one or more of blocks 610d and 610e, such as blocks for detecting performance below the expected level of the first mode (i.e., for implementing operation S520 of method 501) and / or (in the absence of these sub-operations in block 610b) for performing sub-operations S522 and / or S523, etc.

[0050] Generally, each functional block 610a to 610e can be implemented in hardware or software. Preferably, one or more or all functional blocks 610a to 610e can be implemented by processing circuitry 610 (possibly in cooperation with storage medium / memory 612 and / or communication interface 616). Thus, processing circuitry 610 can be arranged to retrieve instructions provided by functional blocks 610a to 610e from memory 612 and execute those instructions, thereby performing any operation of method 500 or any embodiment thereof performed by / in the apparatus 600 disclosed herein.

[0051] Figure 7 The schematic diagram illustrates a computer program product 710 including a computer-readable device / storage medium 730. On the computer storage medium 730, a computer program 720 (including computer code) may be stored. The computer program 720 (when the code is executed) can cause the processor 610 and entities and devices of the device 600 operatively coupled to the device 600, such as a communication interface 616 and a memory 612, to perform the functions described herein, for example, as referenced herein. Figure 3A , Figure 3B , Figure 5A , Figure 5B and Figure 5C The methods 500 and / or their embodiments 501, 502 are described. Therefore, computer program 720 and / or computer program product 710 can provide tools for performing any operation of method 500 (or any of its embodiments such as 501 and / or 502) performed by the apparatus 600 disclosed herein.

[0052] exist Figure 7 In the example, computer program product 710 and computer-readable storage medium 730 are illustrated as optical discs such as CDs (optical discs), DVDs (digital versatile optical discs), or Blu-ray discs. Computer program product 710 and computer-readable storage medium 730 can also be implemented as memories such as random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM), and more specifically, as non-volatile storage media in external memory such as USB (Universal Serial Bus) memory or flash memory (e.g., compact flash memory). Therefore, although computer program 720 is schematically shown herein as tracks on the depicted optical disc, computer program 720 can be stored in any manner suitable for computer program product 710 and computer-readable storage medium 730.

[0053] In summary, this disclosure improves upon contemporary techniques by providing a solution for adaptive intra-refresh encoding of video streams, in which the encoding is not locked to any particular intra-refresh mode, but rather the mode can dynamically change depending on the current configuration and the complexity distribution of the scene. The envisioned solution proposes evaluating whether an alternative mode is performing better than the currently used mode and switching to that mode to improve the temporal variation in bits per frame, which is important for applications such as low-latency applications / video surveillance. In some examples, this evaluation and switching can be triggered by first determining that the current mode used for intra-refresh encoding of the video stream is performing at a lower-than-expected level in terms of the temporal variation in bits per frame.

[0054] Although features and elements may have been described above in specific combinations, each feature or element may be used alone without other features and elements or in various combinations with or without other features and elements. Furthermore, by studying the accompanying drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the claimed invention.

[0055] In the claims, the words "comprising" and "including" do not exclude other elements, and the word "a" does not exclude a plurality. The fact that certain features are described in different dependent claims does not mean that combinations of these features cannot be used beneficially. List of reference numerals Video streams for scenes 100 and 101 Video frames of video streams 110-1 to 110-4 120 macroblocks Lines of macroblocks 130-1 to 130-4 Columns of macroblocks 132-1 to 132-4 Example images of 200 scenes Scenes / image regions of varying complexity: 210, 212, 214, 216 220 macroblocks A chart showing the relationship between 300 and 301 data packet sizes and video frame numbers. 400 images to be encoded 410 Intra-frame refresh area Scanning direction in 420 and 422 frame refresh modes Methods 500, 501, and 502 for adaptive intra-frame refresh coding S510, S520, and S530 method operations S512, S522, S523 Optional operation methods 600 device 610 Processor / Processing Circuit 612 memory 614 Optional Additional Entities 616 Communication Interface 620 communication bus Function blocks 610a to 610e 710 Computer Program Products 720 Computer Program 730 Computer-readable storage media Pattern scanning angle Lines of macroblocks R1 to R6 Columns of macroblocks C1 to C8 P is the period length of the oscillation. A. Amplitude of oscillation / time variation in the number of bits per frame

Claims

1. A method (500) for intra-frame refresh coding of a video stream, wherein, The method includes: The video stream is encoded using the first frame refresh mode (S510). (S520) It is determined that the time variation of the number of bits per frame used to encode the video stream using a second intra-frame refresh mode different from the first intra-frame refresh mode is lower than the time variation of the number of bits per frame used to encode the video stream using the first intra-frame refresh mode; and In response to the determination, the process switches (S530) to encode the video stream using the second intra-frame refresh mode.

2. The method according to claim 1, wherein, The first intra-frame refresh mode and the second intra-frame refresh mode are respectively the horizontal intra-frame refresh mode and the vertical intra-frame refresh mode, or respectively the vertical intra-frame refresh mode and the horizontal intra-frame refresh mode.

3. The method according to claim 1, wherein, The first intra-frame refresh mode and the second intra-frame refresh mode are two opposite diagonal intra-frame refresh modes.

4. The method according to claim 1, wherein, The first intra-frame refresh mode and the second intra-frame refresh mode are two orthogonal intra-frame refresh modes.

5. The method according to claim 1, wherein, The time variation of the number of bits per frame is defined based on the variance of the number of bits per frame, the standard deviation of the number of bits per frame, and / or the oscillation amplitude of the number of bits per frame.

6. The method according to claim 1, wherein, The first intra-frame refresh mode and the second intra-frame refresh mode form part of a plurality of different intra-frame refresh modes, and wherein the determination includes evaluating (S522) the final time change of the number of bits per frame for each of the plurality of intra-frame refresh modes.

7. The method according to claim 6, wherein, The second intra-frame refresh mode is determined to be the intra-frame refresh mode among the different plurality of intra-frame refresh modes that has the minimum time variation in the number of bits per frame used to encode the video stream.

8. The method according to claim 1, wherein, The determination and the switching are executed in response to detecting (S512) that the time change of the number of bits per frame used to encode the video stream using the first intra-frame refresh mode exceeds a threshold.

9. The method according to claim 8, wherein, The determination includes evaluating (S523) the time variation of the number of bits per frame for each of the first intra-frame refresh mode and the second intra-frame refresh mode during one or more finite time intervals, and wherein the determination is performed by detecting that during at least one of the one or more finite time intervals, the time variation of the number of bits per frame for the second intra-frame refresh mode is lower than the time variation of the number of bits per frame for the first intra-frame refresh mode.

10. An apparatus (600) including a processing circuit (610), said processing circuit (610) being configured to: The video stream is encoded using the first-frame refresh mode; The time variation in the number of bits per frame used to encode the video stream using a second intra-frame refresh mode different from the first intra-frame refresh mode is determined to be lower than the time variation in the number of bits per frame used to encode the video stream using the first intra-frame refresh mode; and In response to the determination, the encoding of the video stream is switched to the second intra-frame refresh mode.

11. The apparatus according to claim 10, wherein, The device is a surveillance camera.

12. The apparatus according to claim 10, wherein, The device is a wearable camera or a drone camera.

13. A non-transitory computer-readable storage medium (730) comprising a computer program (720) including computer code that, when executed on a processing circuitry (610) of a device (600), causes the device to: The video stream is encoded using the first-frame refresh mode; The time variation in the number of bits per frame used to encode the video stream using a second intra-frame refresh mode different from the first intra-frame refresh mode is determined to be lower than the time variation in the number of bits per frame used to encode the video stream using the first intra-frame refresh mode; and In response to the determination, the encoding of the video stream is switched to the second intra-frame refresh mode.

14. A computer program product (710) comprising a non-transitory computer-readable storage medium (730) according to claim 13.

Images