Encoding a video stream while changing the camera field of view

Abstract

To provide a method, a device, and a program of encoding a video stream recorded by a camera.SOLUTION: The method of encoding a video stream includes: obtaining a first image with a first FOV; encoding the first image as part of a first encoded video frame; obtaining a second image with a second FOV different from the first FOV; generating a first additional video frame referencing the first video frame, including motion vectors transforming an image content of the first image to an FOV closer to the second FOV than the first FOV, in which the motion vectors are formed based on a difference between the first and second FOVs; inserting the first additional video frame into the encoded video stream as a no-display frame; and encoding the second image as part of a second video frame of the encoded video stream referencing the first additional video frame.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This disclosure relates to encoding a video stream recorded by a camera, and in particular to encoding a video stream during times when the field of view of the camera changes. [Background technology]

[0002] Modern video coding algorithms often use predictive coding to take advantage of the fact that much scene detail may remain static between one image frame and the next. Furthermore, so-called "motion vectors" may also be used to account for objects that move between image frames, in order to instruct the decoder in which areas of one or more other image frames to look for image data belonging to the object when decoding a current image frame containing the object.

[0003] However, with some types of movement and scaling of objects between image frames, caused, for example, by changing the camera's field of view (FOV) between capturing two image frames, using such motion vectors to efficiently encode / decode a video stream can be difficult. One such situation is when a camera performs a zooming operation, causing motion vectors to become longer as time progresses and also to change for different parts of the image frame. As the length of the ideal motion vector increases, the encoder will be forced to, for example, expand the search area in the previous image frame to find such a motion vector. This can therefore lead to an increase in the required calculation time, which can often make the use of motion vectors impractical and sometimes cumbersome.

[0004] As a result, there is therefore a need for an improved way of encoding a video stream using image frames captured during a camera's varying FOV. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Annex G extension of the H.264 / MPEG-4 AVC video compression standard [Non-patent document 2] Annex H extension of H.265 / MPEG-H HEVC video compression standard Summary of the Invention

[0006] To at least partially meet the above-identified needs, the present disclosure provides improved methods, devices, computer programs and computer program products for encoding a video stream as defined in and by the accompanying independent claims. Various embodiments of the methods, devices, computer programs and computer program products are defined in and by the accompanying dependent claims.

[0007] According to a first aspect of the present disclosure, there is provided a method for encoding a video stream. The method includes (e.g., as step “a”) acquiring a first image captured by a video camera having a first field of view (FOV). The method includes (e.g., as step “b”) encoding the first image as part of a first video frame of an encoded video stream. The method includes (e.g., as step “c”) acquiring a second image captured by a video camera having a second FOV that is different from the first FOV but at least partially overlaps with the first FOV. The method also includes (e.g., as step “d”) generating a first additional video frame that references the first video frame and includes a first motion vector that transforms image content of the first image to an FOV closer to the second FOV than the first FOV, where the first motion vector is formed based on a difference between the second FOV and the first FOV. The method includes inserting (e.g., as step "e") a first additional video frame into the encoded video stream and marking the first additional video frame as a no-display frame. The method further includes encoding (e.g., as step "f") a second image as part of a second video frame of the encoded video stream, the second video frame referencing the first additional video frame.

[0008] As used herein, the FOV of an image being transformed to be “closer to another FOV” also includes situations in which the FOV of an image is transformed to exactly match the other FOV. The FOVs of two image frames “at least partially overlap” should be understood as having at least one object, or at least a portion of an object, visible in both image frames, so that motion vectors can be used to locate (parts of) that object in the other image frame when encoding one image frame, etc. A “hidden frame” should be understood as a frame that should not be rendered as part of the decoded video stream, but that is flagged in some way to instruct a decoder that the frame is still available, so that information can be obtained from the frame and used for decoding one or more other image frames that should be displayed (i.e., not marked as a hidden frame). It is envisioned that any video coding standard that supports the above concepts of hidden frames, motion vectors, and the possibility of one frame referencing another (previous or subsequent) frame when encoding / decoding can be used to implement the disclosed method. Examples of such standards include (but are not necessarily limited to) High Efficiency Video Coding (HEVC) H.265, Advanced Video Coding (AVC) H.264, VP8, VP9, ​​AV1, and Versatile Video Coding (VVC) H.266.

[0009] As envisioned herein, the first motion vector need not be calculated by performing a motion vector search, but may be derived / calculated based on knowledge of the first and second FOVs of each of the cameras. For example, such knowledge may include data indicating, for example, the orientation (e.g., defined by pan / tilt / roll angles) and, for example, the focal length of the camera's lens (e.g., as changed during zooming operations), or the like, at times that include both the time when the first FOV was present and the time when the second FOV was present.

[0010] As will be described in more detail later in this specification, the contemplated method of the first aspect improves on currently available coding techniques in that the method provides a more bitrate-efficient way of encoding a video stream during a changing FOV of a camera, such as during a zooming operation. In particular, the need to search for lengthy and changing motion vectors as part of encoding a second video frame is eliminated because the (first) motion vector needed to define a transformation from a first FOV to (or closer to) a second FOV can instead be calculated, for example, from knowledge of the camera direction and focal length (i.e., zoom level), and without, for example, requiring a computationally intensive motion vector search to be performed. When the second video frame references a first additional video frame (for which, at least after decoding, a transformed view of, for example, a particular zoomed-in-on object is available), finding the motion vector for such reference is more trivial and can be performed using conventional methods of searching. For example, in the case of a zooming operation, the first additional video frame would encode a scaled and cropped version of the first image in the first video frame, thus reducing the bit rate as the motion vector search becomes more trivial for finding a match in the first additional video frame to encode the second video frame. Furthermore, the first additional video frame need not contain new image data itself, but only motion vectors calculated based on the difference in FOV and required to efficiently perform a transformation such as scaling. The envisioned method can also be applied for transformations other than scaling, such as when a camera pans, tilts, and / or rolls. A panning or tilting operation may require a perspective transformation that can be accounted for in the additional video frame, and a rolling operation may require a rotation transformation that can also be accounted for in the additional video frame.

[0011] In one or more embodiments of the method, encoding the video stream may include using a layered type of coding, such as, for example, “scalable video coding” (SVC) or “scalable high-efficiency video coding” (SHVC). A first additional video frame may be inserted into a base layer of the encoded video stream, and a second video frame may be inserted into an enhancement layer of the encoded video stream. An example of scalable video coding may be, for example, that defined in Annex G extensions of the H.264 / MPEG-4 AVC video compression standard. Another example of scalable video coding may be, for example, that defined in Annex H extensions of the H.265 / MPEG-H HEVC video compression standard. As an example, the base layer may be coded to provide a lower resolution (e.g., full HD resolution or the like), while the enhancement layer may be coded to provide a higher resolution (e.g., 4K resolution or the like). In a layered type of coding, there may of course also be one or more additional enhancement layers, and it may be envisioned that, for example, a second video frame may be encoded as part of any such enhancement layer. Using layered coding such as SVC may help, for example, to maintain a desired frames per second (FPS) rate, since the first additional video frame and the second video frame may be inserted in different layers, thus avoiding that the (hidden) first additional video frame would have to be inserted, for example, in the middle of two video frames that should be displayed.

[0012] In one or more embodiments of the method, the first additional video frame may be one of a predicted frame (“P frame”) and a bidirectionally predicted frame (“B frame”). As used herein, these various types of video frames are defined as typically done in modern coding standards. A P frame may use / reference data from one or more previous video frames to be decoded, while a B frame may use / reference data from both one or more previous video frames and one or more future / future video frames to be decoded. Because such a video frame includes references to one or more other video frames, as opposed to, for example, an “intra frame” or “intra-coded frame” (“I frame”), which does not include references to one or more other video frames, such a video frame may also be referred to as an “inter frame” or “inter-coded frame.”

[0013] In one or more embodiments of the method, the method may include proceeding to generate and insert one or more additional video frames while the camera's FOV is changing (i.e., during an ongoing zooming or similar operation). This may correspond to a method that further includes (e.g., as additional step “g”) acquiring a third image captured by a video camera having a third FOV that is different from the second FOV but at least partially overlaps with the second FOV. The method may further include (e.g., as additional step “h”) generating a second additional video frame including a second motion vector that transforms image content of the second image to an FOV closer to the third FOV than the second FOV, the second motion vector being formed based on a difference between the third FOV and the second FOV. The method may further include (e.g., as additional step “i”) inserting the second additional video frame into the encoded video stream, including marking the second additional video frame as a hidden frame. The method may further include (e.g., as additional step "j") encoding a third image as part of a third video frame of the encoded video stream, where the third video frame references a second additional video frame. In other words, if more and more additional video frames continue to be inserted into the encoded video stream, the method may include repeating steps c-f described above, but with replacing "first and second" with "second and third," etc.

[0014] In other words, additional video frames may thus be inserted, for example, as long as a zooming operation is ongoing. Instead of attempting to encode the actual images captured during an ongoing zooming operation (such as the second and third images) by searching for motion vectors in previously captured frames, images captured by the camera after the FOV change has begun are instead encoded (as the second and third video frames, etc.) by referencing the respective additional video frames (such as the first and second additional video frames).

[0015] In one or more embodiments of the method, the difference between the first FOV and the second FOV (or, for example, the difference between the second FOV and the third FOV, etc.) may be due to a zooming operation performed by the camera. The zooming operation may be ongoing while the method is performed, for example, when the encoded video stream represents a live stream of a captured scene. The envisioned method may also be used for transcoding cases, where the method is applied to already recorded video material, for example, to optimize (reduce) the size required to store such video material in storage (e.g., on a server). In such cases, it may be envisioned that data related to the various FOVs of the camera during conversion (e.g., camera orientation angle, focal length defining the zoom level of the camera at each time, mounting position of the camera relative to the scene, etc.) is stored as metadata or the like in the recorded video frames, for example, as part of the video frame header.

[0016] In one or more embodiments of the method, the zooming operation may be a zooming-in operation, such that the second FOV is an expansion of the first FOV (and, for example, the third FOV is an expansion of the second FOV, and so on). Applying the method while performing a zooming-in operation may be advantageous, for example, in that a larger portion of each image may be referenced by the next image in the video stream. It should be noted that the envisioned method may also be applied, for example, during a zooming-out operation, such as, for example, where the second FOV is instead a reduction / contraction of the first FOV, but with less benefit, since only a smaller portion of each image may be referenced by the next image in the video stream.

[0017] In one or more embodiments of the method, the first and second images may be, for example, successive images of a video stream captured by a camera, such that adding an additional (non-displayed) video frame is performed for each image of the video stream while the camera transformation is ongoing, and such that, for example, each encoded (to be displayed) video frame references a corresponding additional (non-displayed) video frame.

[0018] In one or more embodiments of the method, the method may be implemented in a camera used to capture at least the first and second images. This may, for example, allow the encoded video stream to be generated at the "edge" without requiring additional intermediate video processing equipment.

[0019] Generally, it is assumed herein that a video frame of the encoded video stream that is to be displayed (e.g., not marked as a hidden frame) references at least its corresponding additional (hidden) video frame, and that each additional (hidden) video frame references at least a previous (hidden) video frame in the encoded video stream. Of course, it may also be the case, for example, that a (to-be-displayed) video frame also references, for example, a previous (to-be-displayed) video frame in the encoded video stream. When using layered coding such as SVC, each video frame in the enhancement layer therefore references a corresponding additional video frame in the base layer, but may also, if desired, for example, reference one or more previous (or further future / future) video frames in the enhancement layer.

[0020] According to a second aspect of the present disclosure, there is provided a corresponding device for encoding a video stream, the device including a processor and a memory, the memory storing instructions that, when executed by the processor, cause the device to perform the method according to the first aspect. This includes causing a device to acquire a first image captured by a video camera having a first FOV, encoding the first image as part of the first video frame of the encoded video stream, acquiring a second image captured by the video camera having a second FOV that is different from the first FOV but at least partially overlaps with the first FOV, generating a first additional video frame that references the first video frame and includes a first motion vector that transforms image content of the first image to an FOV that is closer to the second FOV than the first FOV, where the first motion vector is formed / calculated based on a difference between the second FOV and the first FOV, inserting the first additional video frame into the encoded video stream, where the first additional video frame is marked as a hidden frame, and encoding a second image as part of the second video frame of the encoded video stream, where the second video frame references the first additional video frame.

[0021] In one or more embodiments of the device, the instructions stored in the memory of the device may further be such that, when executed by a processor, the instructions cause the device to perform any embodiment of the method (of the first aspect) as disclosed and contemplated herein.

[0022] In one or more embodiments of the device, the device may be a camera for capturing at least a first image and a second image.

[0023] According to a third aspect of the present disclosure, there is provided a corresponding computer program for encoding a video stream, which computer program is configured (i.e. comprises instructions / computer code that cause the device to perform the method of the first aspect) when executed by a processor of a device (such as the device of the second aspect). This includes causing a device to acquire a first image captured by a video camera having a first FOV, encoding the first image as part of the first video frame of the encoded video stream, acquiring a second image captured by the video camera having a second FOV that is different from the first FOV but at least partially overlaps with the first FOV, generating a first additional video frame that references the first video frame and includes a first motion vector that transforms image content of the first image to an FOV that is closer to the second FOV than the first FOV, where the first motion vector is formed / calculated based on a difference between the second FOV and the first FOV, inserting the first additional video frame into the encoded video stream, where the first additional video frame is marked as a hidden frame, and encoding a second image as part of the second video frame of the encoded video stream, where the second video frame references the first additional video frame.

[0024] In one or more embodiments of the computer program, the computer program (i.e., instructions / computer code provided therewith) may be further configured, when executed by a processor, to cause the device to perform any embodiment of the method (of the first aspect) as disclosed and discussed herein.

[0025] According to a fourth aspect of the present disclosure, a corresponding computer program product is provided. The computer program product includes a computer-readable storage medium storing a computer program according to the third aspect (or according to any of the herein described and discussed embodiments of the computer program product). As used herein, a computer-readable storage medium may be, for example, non-transitory and may be provided as, for example, a hard disk drive (HDD), a solid-state drive (SDD), a USB flash drive, an SD card, a CD / DVD, and / or any other storage medium capable of non-transitory storage of data. In other embodiments, the computer-readable storage medium may be transitory and may, for example, correspond to (electrical, optical, mechanical, or similar) signals present on, for example, a communications link, wire, or similar means of signal transfer.

[0026] Other objects and advantages of the present disclosure will be apparent from the following detailed description, drawings, and claims. Within the scope of the present disclosure, it is contemplated that, for example, all features and advantages described with respect to the method of the first aspect are also appropriate to, apply to, and can be used in combination with the device of the second aspect, the computer program of the third aspect, and the computer program product of the fourth aspect, and vice versa.

[0027] Exemplary embodiments will now be described below with reference to the accompanying drawings. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 shows a schematic diagram of an example video stream in which the camera's FOV changes during image capture. [Figure 2] 2 is a diagram illustrating an example of an improved method for encoding a video stream that is applied to the video stream of FIG. 1 according to an embodiment of the present disclosure. [Figure 3] FIG. 1 shows a schematic flow chart of an improved method for encoding a video stream according to an embodiment of the present disclosure. [Figure 4] 2 is a diagram schematically illustrating another example of an improved method for encoding a video stream applied to the video stream of FIG. 1 according to an embodiment of the present disclosure. [Figure 5] FIG. 10 shows a schematic diagram of an example of how motion vectors should be obtained based on knowledge of how the FOV of the camera has changed, according to an embodiment of the improved method of the present disclosure. [Figure 6] FIG. 1 is a diagram illustrating an example of a device for encoding a video stream, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0029] In the drawings, like reference numerals are used for like elements unless otherwise noted. Unless explicitly stated to the contrary, the drawings show only those elements necessary to illustrate the exemplary embodiments, while other elements may be omitted or only suggested for clarity. As shown in the figures, the sizes (absolute or relative) of elements and regions may be exaggerated or understated relative to their true values ​​for illustrative purposes, and thus are provided to illustrate the general structure of the embodiments.

[0030] 1 shows a schematic diagram of one typical example of a video stream 100 captured by a (video) camera, where the FOV of the camera changes during capture. In this particular example, the change in FOV is due to the camera performing a zooming operation, more specifically a zoom-in operation that involves slight panning and tilting of the camera to move the object of interest to the center of the camera's FOV.

[0031] In this particular example, the scene is assumed to include a single object of interest 110, which is assumed to be a flower in a pot, but could, of course, be any combination of one or more other objects. The video sequence 100 includes a first image 120a captured by a camera in a first FOV, where the first FOV is such that the object 110 occupies only the lower left corner of the first image 120a. As time progresses, the camera then begins to zoom in on the object 110, e.g., by adjusting the camera lens and / or by using digital processing that emulates such physical zooming, and also pans slightly left and tilts slightly down to center the object 110 in the middle of the FOV.

[0032] When a subsequent second image 120b is captured by the camera, the camera therefore has a second FOV that provides a magnification of the object 110 compared to the first FOV. The object 110 is slightly larger in size when shown in the second image 120b compared to when shown in the first image 120a, and has also begun to move towards the center of the image due to the panning and tilting of the camera.

[0033] As time progresses further, the camera continues to zoom, capturing more images of the scene so that object 110, for example, becomes larger and larger, occupying a larger and larger portion of each image. In the particular example shown in FIG. 1 , this involves capturing a third image 120c using a camera having a third FOV and a fourth image 120d using a camera having a fourth FOV. After capturing fourth image 120d, in which object 110 now occupies nearly all of the available image space and is roughly centered within the FOV, the camera stops zooming. In successive frames, such as fifth image 120e and sixth image 120f, the camera's FOV remains unchanged and equals the fourth FOV used to capture fourth image 120d.

[0034] To encode the video stream 100, various encoding schemes may be applied, e.g., to conserve bitrate / bandwidth. A conventional such scheme may begin by encoding the first image 120a as an intra-predicted frame (I-frame) or the like, without reference to one or more additional video frames. The second image 120b may then be encoded as an inter-predicted frame (P-frame), providing a reference to the I-frame encoding the first image 120a, e.g., in the form of a motion vector. However, to find such a motion vector, the encoder must first perform a motion vector search, in which the encoder attempts to identify, for each portion of the second image 120b, from which portion of the first image 120a image data should be retrieved for later decoding the P-frame encoding the second image 120b. This is often achieved by defining a limited search area around each portion of the second image 120b and then searching through this area in the first image 120a to find a suitable match. If the change in camera FOV did not involve a zooming operation, but, for example, only a slow translational movement of the camera (such as panning), this would be the most potentially efficient way of encoding the P frame (and thereby the second image 120b), as the required search area would remain small and the motion vectors would be approximately the same for all parts of the second image 120b.

[0035] However, in the case of a zooming operation such as that shown in Figure 1, the search for such a motion vector would likely be more difficult and would require the search area to be enlarged. Furthermore, the motion vectors would also be different for many or all parts of the second image 120b, such that, for example, the movement of pixels belonging to the flower pot would be smaller than, for example, the movement of pixels belonging to the top of the flower, and such that, for example, pixels belonging to the left part of the object 110 would have moved approximately only upward in the second image 120b compared to those in the first image 120a, while, for example, pixels belonging to the right part of the object 110 would have instead moved both upward and right. As zooming progresses and different parts of the object 110 start to move further and faster away from each other with each newly captured image (and each new P-frame must, for example, refer to a previously encoded P-frame), the motion vectors will become both longer and longer and will also become increasingly different (in direction) for different parts of the second image 120b and the P-frame encoding the second image 120b. As a result, the processing power required to perform such a motion vector search will risk, for example, exceeding the capabilities of the entity responsible for the encoding, which in a live situation may, for example, be the camera itself.

[0036] How the method envisaged herein can improve this situation will now be explained in more detail, for example with reference to Figures 2 and 4. Reference is also made to Figure 3, which shows schematically a flow chart 300 of the various steps of such a method.

[0037] 2 schematically illustrates the encoding of various video frames performed as part of method 200. As envisioned herein, improved method 200 includes obtaining (as step S301) a first image (e.g., first image 120a) captured by a video camera having a first FOV. Method 200 further includes encoding (as step S302) first image 120a as part of first video frame 220a of encoded video stream 210. This may be performed as conventionally done, e.g., by encoding first video frame 220a as an I-frame or the like, such that first video frame 220a contains all necessary image data needed to decode first image 120a again at a decoder side (not shown). The encoding may be lossless or produce at least some loss of image quality when decoding first video frame 210a, depending, for example, on the exact method used for such encoding of first video frame 220a.

[0038] Method 200 includes (as step S303) receiving a second image (e.g., second image 120b) captured by a video camera having a second FOV. For example, as previously described herein with reference to FIG. 1, the first FOV is different from the second FOV, but the second FOV is such that the second FOV at least partially overlaps with the first FOV. In other words, for method 200 to function as intended, there should preferably be at least some portion of, e.g., object 110, visible in both first image 120a and second image 120b. More preferably, all portions of object 110 that are visible in second image 120b should also be visible in first image 120a.

[0039] In contrast to conventional encoding methods, the contemplated improved method 200 does not proceed by attempting to directly encode the second image 120b, for example, as a regular P-frame using a motion vector search on the first image 120a. Instead, in step S304, the method 200 proceeds by generating a first additional video frame 230a that references the first video frame 220a, where the first video frame 220a includes first motion vectors that transform (in this case, scale and translate) the image content of the first image 120a to an FOV closer to the second FOV than the first FOV. These first motion vectors are not found by a computation-intensive motion vector search, but instead are calculated / formed based on knowledge of the second FOV and the first FOV, i.e., knowledge of how the camera's FOV changed between capturing the first image 120a and capturing the second image 120b. Such information may be received, for example, from the camera itself and / or from any other specific sensor(s) capable of detecting, for example, the camera orientation (such as pan / tilt / roll angles or the like and / or the focal length of the camera and its lens). For example, it may be known that the camera's FOV was changed between capturing the first image 120a and the second image 120b, for example, by increasing the lens focal length by a certain amount and tilting the camera slightly downward while panning it slightly to the left (to move the object 110 toward the center of the image while zooming). This information may then be used to directly obtain the first motion vector needed to perform scaling / translation that transforms the FOV of the first image 120a into an FOV that is closer to or equal to the second FOV used to capture the second image 120b.As used herein, "the FOV of an image is transformed to some other FOV" should be understood to mean that the transformation, when calculated, when applied to the image content of the image will render the resulting image to look similar (or more similar) to if the image had been captured by a camera having this other FOV. How a first motion vector may be formed / calculated based on knowledge of the various FOVs used to capture the two images is described in more detail later herein, for example, with reference to FIG. 5.

[0040] After the first motion vector is formed / calculated, the method 200 proceeds by inserting (at step S305) a first additional video frame 230a into the encoded video stream 210, including marking the first additional video frame 230a as a hidden frame (as indicated by the strike-through screen symbol shown in FIG. 2). The first additional video frame 230a will therefore not be shown during subsequent decoding of the encoded video stream 210, but will serve only as a container of information for decoding / encoding other video frames. Notably, in this example, the first additional video frame 230a does not contain actual image data, but need only contain a first motion vector that allows (during subsequent decoding of the first additional video frame 230a) to transform the FOV of the first image 120a into an FOV that more closely resembles the second FOV, i.e., by scaling, translating, and / or rotating the image content of the first image 120a as necessary.

[0041] The method then continues in step S306 with encoding the second image 120b as part of a second video frame 220b of the encoded video stream 210, the second video frame 220b being such that it references the first additional video frame 230a. The second video frame 220b may be, for example, a P-frame or the like, with a motion vector indicating which portion of the (decoded) first additional video frame 230a should be used to decode which portion of the second video frame 220b. Importantly, because the image FOV of the (to-be-decoded) first additional video frame 230a more closely resembles (or even is equal to) the FOV of the second image 120b, finding such a motion vector for decoding the second video frame 220b using a motion vector search becomes more straightforward than, for example, attempting to find such a motion vector directly based on the image content of the first image 120a. The motion vector for the second video frame 220b can be found, for example, by comparing the image content of the second image 120b with the image content of the translated image content of the first image 120a contained in (or generated from) the first additional video frame 230a after decoding. The envisaged method therefore provides an improved way of encoding video streams captured during changes in the camera's FOV.

[0042] 2, references from one coded video frame to another coded video frame are indicated by solid arrows. For example, solid arrow 240a indicates how the first additional video frame 230a references the first video frame 220a, solid arrow 240b indicates how the second video frame 220b references the first additional video frame 230a, solid arrow 240c indicates how the second additional video frame 230b references the first additional video frame 230a (if included, as discussed later herein), and so on. As used herein, the arrow points in the direction information is transferred such that, for example, arrow 240a pointing from the first video frame 220a to the first additional video frame 230a indicates that the first additional video frame 230a is the video frame that references the first video frame 220a. As envisioned herein, in some embodiments of the method, there may of course be other references included between various video frames. 2 by dashed arrows and may include, for example, the second video frame 220a also referencing the first video frame 220a, for portions that do not require motion vectors, such as a constant background or the like. Other such optional references may include, for example, one or more frames being bidirectionally predicted frames (B-frames), which also include references to one or more "future" video frames.

[0043] In addition to what has been described above, it is understood that in some embodiments, the contemplated method may also continue in a similar manner to encode one or more additional images captured while the camera's FOV is changing, e.g., as part of an ongoing zooming operation. The method contemplated herein and described by flowchart 300 may therefore include optional step S307, in which it is checked whether the camera's FOV is still changing. If so, the method may backtrack (as indicated by dashed arrow 310) and redo steps S303-S306 again, generating a new, hidden additional video frame that references the previous additional video frame each time a new image is acquired, and encoding the new image as part of the new video frame that references the new additional video frame. If it is not determined that a change in camera FOV is still in progress, the method may instead, for example, stop and return to encoding frames as conventionally done, e.g., as indicated by dashed arrow 312.

[0044] As shown in FIG. 2 , continuing method 200 may include, for example, acquiring a third image (e.g., third image 120c) captured by a camera having a third FOV, where the third FOV is different from the second FOV but also such that the third FOV at least partially overlaps with the second FOV. This may correspond to repeating step S303 above, except, for example, using “second and third” instead of “first and second.” The method may include, for example, generating a second additional video frame 230b that references the first additional video frame 230a and includes a second motion vector, which is used to transform the image content of second image 120b to an FOV closer to the third FOV than the second FOV, in a manner similar to the first motion vector described above. Here, the second motion vector is instead formed based on the difference between the third FOV and the second FOV. This may correspond, for example, to repeating step S304 above, but again using "second and third" instead of "first and second." The method may continue by inserting a second additional video frame 230b into the encoded video stream 210 and marking the second additional video frame 230b as a hidden frame (which may correspond to repeating step S305 above, using "second and third" instead of "first and second," etc.). The method may continue by encoding a third image 120c as part of a third video frame 220c, which may, for example, reference the second additional video frame 230b. The method may continue in a similar manner, in some embodiments, until the change in the camera's FOV is complete. In this example, this may include obtaining a fourth image, for example, fourth image 120d, generating a third additional video frame 230c that references the second additional video frame 230b, encoding the fourth image 120d in the encoded video stream 210 as part of the fourth video frame 230d that references the third additional video frame 230d, etc.Upon detecting that zooming has stopped, for example, by noting that the fourth FOV of the camera when capturing the fourth image 104d is the same as the fifth FOV of the camera when capturing the fifth image 104e, the method may, for example, in some embodiments, stop inserting additional video frames and instead return to encoding frames as would be done when zooming is not currently continuing. For example, the fifth image 104e may be encoded as part of the fifth video frame 220e that references the fourth additional video frame 230d, the next image 230f may be encoded as video frame 220f that references video frame 230e, etc. In particular, the methods envisioned herein apply primarily to encoding video frames corresponding to times during which the camera is changing its FOV and during which traditional coding using motion vectors is difficult due to the fact that such motion vectors are both long and changing due to the camera's changing FOV.

[0045] It should be noted that, while illustrated solely by a change in FOV due to a zooming-in operation (plus additional panning / tilting operations) performed by the camera, the contemplated method may also be used to encode a video stream during recording in which there are one or more additional changes in the camera's FOV and / or other causes for change in the camera's FOV. Such examples may include, for example, camera rotation, or other combinations of possible changes related to pan, tilt, roll, and / or focal length changes, or the like. It is contemplated that, to the extent that knowledge regarding the different FOVs used when recording different images is known (or can possibly be derived in some way), the motion vectors needed to generate the additional video frames may be obtained directly from this knowledge without requiring a motion vector search by a decoder, such as a camera. To the extent that the FOV of the additional video frames can thus be made to more closely or exactly match the FOV of the images captured by the camera, motion vector search may then be more efficiently performed when encoding the captured images by searching in and referencing the additional video frames.

[0046] Another embodiment of the envisaged method will now be described with reference to Figure 4. Here, it is assumed that method 400 still encodes video stream 100 captured while zooming in on object 110, as shown in Figure 1, but in contrast to method 200 described with reference to Figure 2, method 400 instead uses a layered type of coding, such as scalable video coding (SVC). In the following, only the differences between method 200 and method 400 will be highlighted.

[0047] The encoded video stream 410 has multiple layers, including at least a base layer 412 and at least one enhancement layer 414. The method 400 begins, as is conventional in such layered coding schemes, by obtaining a first image 120a and encoding this image as part of a first video frame 420a in the base layer 412 (e.g., using a lower resolution) and as part of a first video frame 440a in the enhancement layer 414 (using detail to obtain a higher resolution than that of the base layer 412).

[0048] After acquiring the second image 120b, a first additional video frame 430a is generated as described earlier in this specification and references the first video frame 420a of the base layer 412. The first additional video frame 430a is also encoded in the base layer 412 and marked as a hidden frame.

[0049] However, the second image 120b is not directly encoded as part of any video frame of the base layer 412, but instead is encoded only as part of a second video frame 440b of the enhancement layer 414, with the second video frame 440b referencing a first additional video frame 430a in the base layer 412.

[0050] Similarly, as previously described herein, the method 400 may also, in some embodiments, continue to generate and encode more video frames, as long as the camera's FOV is changing, of course. A third image 120c may be acquired, the third FOV used to capture the third image 120c may be compared to the second FOV used to capture the second image 120b, and a second motion vector may be calculated and used to generate a second additional video frame 430b that is encoded in the base layer 412. Just as previously described, the third image 120c is not directly encoded in the base layer 412, but instead is encoded in the enhancement layer 414 as a third video frame 440c and made to reference the second additional video frame 430b in the base layer 412. The same procedure may then, of course, be repeated to obtain, for example, a fourth image 120d, a fifth image 120e, etc., and corresponding frames may be generated and inserted into the base layer 412 and the enhancement layer 414 as described herein. In the present example, this includes generating a third additional video frame 430c (marked as hidden and referencing the second additional video frame 430b) in the base layer 412, encoding the fourth image 120d as part of a fourth video frame 440d in the enhancement layer 414 (wherein the fourth video frame 440d references the third additional video frame 430c in the base layer 412), etc. When zooming in on the object 110 stops, the method 400 may stop generating and inserting additional video frames and instead return to encoding the fifth image 120e and the sixth image 120f again in both the base layer 412 and the enhancement layer 414.For example, this may include encoding the fifth image 120e both in a fifth video frame 420e in the base layer 412 (using a lower resolution) and in a fifth video frame 440e in the enhancement layer 414 (using detail to obtain a resolution higher than that of the base layer 412), and may include encoding the sixth image 120 both in a sixth video frame 420f in the base layer 412 and in a sixth video frame 440f in the enhancement layer 414. As can be seen in FIG. 4, in method 400, the additional video frames 230a, 230b, and 230c thus replace what would potentially be the second, third, and fourth normally encoded video frames in the base layer 412.

[0051] The fifth video frame 420e in the base layer 412 may, for example, reference the third additional video frame 430c, etc., as indicated by the solid arrows provided in FIG. 4 and as already described with reference to FIG. 2. There may also be other references included between various video frames, as described herein. For example, as indicated by the dashed arrows in FIG. 5, each video frame in the enhancement layer 414 may also reference, for example, one or more previous video frames in the enhancement layer 414, and each "normal" (i.e., non-additional video frame) in the base layer 412 may reference one or more previous video frames in the base layer 412. Of course, there may be other references not shown, such as, for example, references to future frames or the like, if conceivable.

[0052] 2, method 400 has the advantage that when all the additional video frames (e.g., frames 430a-c) are inserted into base layer 412, the enhancement layer 414 still contains only one encoded video frame per image captured by the camera, so all the additional video frames do not affect the FPS rate. In situations where the enhancement layer 414 should not be decoded, for example, when the processing power of the encoder is insufficient for decoding, or if the enhancement layer 414 is also removed from the encoded video stream 410 before sending the encoded video stream 410, for example, over a network (to save bandwidth), it can also be envisioned that there may still be some use in retrieving and decoding only the base layer 412, so that the “FOV-transformed” image content of the various captured images generated using motion vectors for scaling, translation, and / or rotation found in the additional frames can still be used to render the image content.

[0053] FIG. 5 shows in more detail how the required (first) motion vector calculation can be obtained. Here, as an example, a camera 510 having a first FOV (“FOV1”) captures a first image 120a of an object 110. At another (e.g., later) time, the same camera 510 captures a second image 120b of the object 110 while having a different second FOV (“FOV2”). For illustrative purposes only, in the example provided in FIG. 5, the difference between the first and second FOVs (and thereby the difference between the first and second images 120a and 120b) is exaggerated in FIG. 5 compared to, for example, FIGS. 1 and 2.

[0054] When capturing a first image 120a, a first FOV (FOV1) is defined by the camera 510 having, for example, a first pan angle 520a, a first tilt angle 522a, and a first roll angle 524a, and a first focal length 526a. When capturing a second image 120b, a second FOV (FOV2) is defined by the camera 510 having a second pan angle 520b, a second tilt angle 522b, a second roll angle 524b, and a second focal length 526b. In this particular example, as discussed above, the roll angles 524a-b are the same (because the camera is zoomed in and panned / tilted slightly left / down in the second FOV than in the first FOV), but the second pan angle 520b is slightly smaller than the first pan angle 520a, the second tilt angle 522b is slightly larger than the first tilt angle 522a, and the second focal length 526b is larger than the first focal length 526a. The exact definitions of the various angles are not envisioned as important, as long as the definitions allow for measuring / detecting how the angles have changed between one FOV and the other.

[0055] By having knowledge of the first and second FOVs of camera 510 (e.g., by having knowledge of angles 520a-b, 522a-b, 524a-b, and focal lengths 526a-b), first motion vectors (e.g., 530a and 530b) can then be calculated as shown in overlay 540 of first image 120a and second image 120b, also shown in FIG. 5. As can be seen in overlay 540, first motion vectors 530a and 530b can be different for different portions of second image 120b (e.g., 550a and 550b) because the scaling and translation required to go from the first FOV closer to the second FOV can cause different portions of object 110 to move in different directions and at different speeds. Of course, there can be additional first motion vectors other than those shown in FIG. 5, for each portion of second image 120b.

[0056] It is also envisaged herein to provide a device for encoding a video stream, as will now be described in more detail with reference to FIG.

[0057] FIG. 6 schematically illustrates a device 600 for encoding a video stream as previously discussed herein, i.e., a device configured to perform, for example, the method 200 described with reference to FIG. 2 and / or the method 400 described with reference to FIG. 4. The device 600 includes at least a processor (or “processing circuitry”) 610 and a memory 612. A “processor” or “processing circuitry” as used herein may be any combination of one or more suitable central processing units (CPUs), multiprocessors, microcontrollers (μCs), digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), graphics processing units (GPUs), etc., capable of executing software instructions stored in the memory 612. The memory 612 may be external to the processor 610 or may be internal to the processor 610. A “memory” as used herein may be any combination of random access memory (RAM) and read-only memory (ROM), or any other type of memory capable of storing instructions. Memory 612 includes (i.e., stores) instructions that, when executed by processor 610, cause device 600 to perform a method described herein (i.e., method 200, 400, or any embodiment thereof). Device 600 may further include one or more additional items 614 that may, in some circumstances, be useful for performing a method. In some exemplary embodiments, device 600 may be, for example, a (video) camera, such as a video monitoring camera, and the additional item(s) 614 may include, for example, an image sensor and, for example, one or more lenses for focusing light from a scene onto the image sensor, so that the monitoring camera may then capture images of the scene as part of performing the envisioned method.Additional item(s) 614 may also include various other electronic components needed, e.g., to properly operate the image sensor and / or lens as desired and to gather information related to, e.g., the current orientation and / or focal length of the camera and lens(es), which may be used, e.g., to calculate how to create a motion vector based on the difference between two FOVs. Implementing the method in a monitoring camera may be useful in that processing is moved to the “edge,” i.e., closer to where the actual scene is captured, compared to performing processing and video encoding elsewhere (such as in a more centralized processing server or the like). Device 600 may, for example, be connected to a network such that the encoded video stream resulting from implementing the method may be transmitted to a user. To this end, device 600 may include a network interface 616, which may be, for example, a wireless network interface (e.g., supporting Wi-Fi, as defined in, for example, IEEE 802.11 or any successor standard) or a wired network interface (e.g., supporting Ethernet, as defined in, for example, IEEE 802.3 or any successor standard). Network interface 616 may also support any other wireless standard, such as, for example, Bluetooth or the like, capable of transferring, for example, encoded video. The various components 610, 612, 614, and 616 (if present) may be connected via one or more communication buses 620 so that these components can communicate with one another and exchange data as necessary.

[0058] Device 600 may be, for example, a monitoring camera mounted or mountable on a building, for example, in the form of a PTZ camera capable of providing a wider perspective of a scene, or, for example, a fisheye camera, or any other type of monitoring / surveillance camera. Device 600 may be, for example, a body camera, action camera, dash cam, or the like suitable for mounting on people, animals, and / or various vehicles, or the like. Device 600 may be, for example, a smartphone or tablet that a user can carry and that can capture a scene. In any such example of device 600, it is contemplated that device 600 may include all necessary components other than those already described herein (if any), so long as device 600 is still capable of performing method 200, 400, or any embodiment thereof, as contemplated herein.

[0059] 6, device 600 and processor 610 may be structured into one or more computational blocks, each responsible for performing functions according to a particular step of methods 200 and 400, for example, as shown in flowchart 300 of FIG. 3. For example, one such functional block may be configured to obtain first image 120a (i.e., to perform step S301), another functional block may be configured to encode the first image (i.e., to perform step S302), and so on until all steps of method 200, 400, or other embodiments of the methods, have been performed by device 600.

[0060] In summary of various embodiments presented herein, the present disclosure provides an improved way of encoding a video stream in situations where the FOV of a camera changes while capturing images of the video stream (e.g., during a zooming operation performed by the camera). By inserting one or more additional hidden frames that continuously transform the image content of the captured image before the FOV change begins to match the FOV of the camera at various stages of the FOV change, and by basing such transformation on motion vectors found directly from knowledge of the FOV (and without requiring a motion vector search), more efficient and less computationally intensive encoding may be implemented. Various images of the bitstream during the FOV change may then be encoded using motion vectors that reference various additional video frames, for example, by performing a motion vector search that does not result from long, changing motion vectors. The envisioned methods, as well as corresponding devices, computer programs, and computer program products, as described herein, therefore fulfill a clear need for encoding such video streams, as encoding techniques based solely on motion vector search often prove to be computationally intractable, especially in real-time or near-real-time situations.

[0061] Although the features and elements may be described above in particular combinations, each feature or element may be used alone without the other features and elements, or in various combinations with or without other features and elements. Additionally, variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

[0062] In the claims, the words "comprising" and "including" do not exclude other elements, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that certain features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage. [Explanation of symbols]

[0063] 100 video streams captured by cameras 110 objects in camera FOV 120a~f Images 1~6 200 How to encode a video stream 210 encoded video stream 220a-f Video frames of the encoded video stream 230a-c Additional hidden video frames 240a~c Reference arrow 300 Flowchart of the proposed method 310, 312 Arrows indicating method flow S301~S307 Method steps How to encode a 400 video stream (layered) 410 Encoded Video Stream (Layered) 412 Base Layer 414 Enhancement Layer 420a, e-f Video frames in the base layer 430a-c Additional hidden video frames in the base layer 440a-f Video frames in the enhancement layer 510 (video) camera 520a-b First and second pan angles of the camera 522a-b First and second tilt angles of the camera 524a-b First and second roll angles of the camera 526a-b First and second focal lengths of the camera 530a-b First motion vector 540 Overlay of the first and second images 600 devices 610 processor 612 memory 614 additional item(s) 614 Network Interface 620 communication bus(es) FOV1, FOV2 The first and second FOV of the camera.

Claims

1. 1. A method for encoding a video stream, comprising: a) obtaining a first image of a scene captured by a video camera at a first time; b) encoding the first image as part of a first video frame of an encoded video stream; c) obtaining a second image of the scene captured by the video camera at a second time; and d1) obtaining data indicating a first difference in an orientation, zoom level, and / or position of the video camera relative to the scene at the first time compared to the second time, and calculating a first motion vector based on the indicated first difference; d2) generating a first additional video frame that references the first video frame, the first additional video frame including the first motion vector for transforming image content of the first image; e) inserting the first additional video frame into the encoded video stream, including marking the first additional video frame as a non-display frame; and f) encoding the second image as part of a second video frame of the encoded video stream, the second video frame referencing the first additional video frame; and A method comprising:

2. The method of claim 1, further comprising using a layered type of coding, such as scalable video coding (SVC), wherein the first additional video frame is inserted into a base layer of the encoded video stream and the second video frame is inserted into an enhancement layer of the encoded video stream.

3. The method of claim 1 , wherein the first additional video frame is one of a predicted frame (P-frame) or a bidirectionally predicted frame (B-frame).

4. g) obtaining a third image of the scene captured by the video camera at a third time; and h1) obtaining data indicating a second difference in an orientation, zoom level, and / or position of the video camera between the second time and the third time, and calculating a second motion vector based on the second difference; h2) generating a second additional video frame that references the first additional video frame, the second additional video frame including the second motion vector for transforming image content of the second image; i) inserting the second additional video frame into the encoded video stream, including marking the second additional video frame as a non-display frame; j) encoding the third image as part of a third video frame of the encoded video stream, the third video frame referencing the second additional video frame; and The method of claim 1 further comprising:

5. The method of claim 1 , wherein the first difference is due to a zooming operation performed by the video camera.

6. The method of claim 5 , wherein the zooming operation is a zooming-in operation.

7. The method of claim 1 , wherein the first image and the second image are consecutive images of a video stream captured by the video camera.

8. 1. A device for encoding a video stream, comprising: a processor; A memory that stores instructions the instructions, when executed by the processor, cause the device to a) obtaining a first image of a scene captured by a video camera at a first time; b) encoding the first image as part of a first video frame of an encoded video stream; c) obtaining a second image of the scene captured by the video camera at a second time; and d1) obtaining data indicating a first difference in an orientation, zoom level, and / or position of the video camera relative to the scene at the first time compared to the second time, and calculating a first motion vector based on the indicated first difference; d2) generating a first additional video frame that references the first video frame, the first additional video frame including the first motion vector for transforming image content of the first image; e) inserting the first additional video frame into the encoded video stream, including marking the first additional video frame as a non-display frame; and f) encoding the second image as part of a second video frame of the encoded video stream, the second video frame referencing the first additional video frame; and A device that causes

9. 9. The device of claim 8, wherein the instructions, when executed by the processor, further cause the device to perform the method of claim 2.

10. The device of claim 8 , wherein the device is a camera for capturing the first image and the second image.

11. 1. A computer program for encoding a video stream, the computer program, when executed by a processor of a device including a computer-readable storage medium, causing the device to: a) obtaining a first image of a scene captured by a video camera at a first time; b) encoding the first image as part of a first video frame of an encoded video stream; c) obtaining a second image of the scene captured by the video camera at a second time; and d1) obtaining data indicating a first difference in an orientation, zoom level, and / or position of the video camera relative to the scene at the first time compared to the second time, and calculating a first motion vector based on the indicated first difference; d2) generating a first additional video frame that references the first video frame, the first additional video frame including the first motion vector for transforming image content of the first image; e) inserting the first additional video frame into the encoded video stream, including marking the first additional video frame as a non-display frame; and f) encoding the second image as part of a second video frame of the encoded video stream, the second video frame referencing the first additional video frame; and A computer program configured to cause a

12. 12. The computer program of claim 11, further configured to cause the device to perform the method of claim 2.

Images