Method and apparatus for encoding video stream during change of camera field of view

Abstract

The present disclosure relates to a method and apparatus for changing video stream encoding during a camera field of view. A method (400) of encoding a video stream is provided, comprising: obtaining a first image (120a) having a first FOV; encoding the first image as part of a first encoded video frame (420a); obtaining a second image (120b) having a second FOV different from the first FOV; generating a first additional video frame (430a) referencing the first video frame, comprising transforming image content of the first image into a motion vector of a FOV closer to the second FOV than the first FOV, wherein the motion vector is formed based on a difference between the first FOV and the second FOV; inserting the first additional video frame as a non-display frame into an encoded video stream (410); and encoding the second image as part of a second video frame (440b) of the encoded video stream referencing the first additional video frame. A corresponding apparatus and computer program product are also provided.

Description

Technical Field

[0001] This disclosure relates to encoding video streams recorded by a camera. In particular, this disclosure relates to encoding the video stream during the time period when the camera's field of view changes. Background Technology

[0002] In contemporary video encoding and decoding algorithms, predictive encoding and decoding are often used to take advantage of the fact that many details of a scene can remain static between one image frame and the next. Furthermore, so-called "motion vectors" can be used to interpret objects moving between image frames, instructing the decoder to search for image data belonging to that object in one or more other image frames when decoding the current image frame containing it.

[0003] However, efficiently encoding / decoding video streams using such motion vectors can be challenging due to certain types of movement and scaling of objects between image frames, caused by, for example, changes in the camera's field of view (FOV) between capturing two image frames. One such scenario is that when the camera performs a zoom operation, the motion vectors become longer, and also change over time for different parts of the image frame. As the length of the most suitable motion vector increases, the encoder will be forced to expand the search area, for example, in previous image frames, to find such a motion vector. Therefore, this can lead to an increase in the required computation time and often makes the use of motion vectors impractical and sometimes even tractable.

[0004] Therefore, an improved method is needed to encode video streams containing image frames captured during camera FOV changes. Summary of the Invention

[0005] To at least partially satisfy the needs identified above, this disclosure provides an improved method, apparatus, computer program, and computer program product for encoding video streams, as defined in and by way of the appended independent claims. Various embodiments of the method, apparatus, computer program, and computer program product are defined in and by way of the appended dependent claims.

[0006] According to a first aspect of this disclosure, a method for encoding a video stream is provided. The method includes (e.g., as step "a") obtaining a first image captured by a camera having a first field of view (FOV). The method includes (e.g., as step "b") encoding the first image as part of a first video frame of an encoded video stream. The method includes (e.g., as step "c") obtaining a second image captured by a camera having a second FOV that is different from, but at least partially overlaps with, the first FOV. The method includes (e.g., as step "d") generating a first additional video frame referencing the first video frame and including a first motion vector that transforms the image content of the first image into an FOV that is closer to the second FOV than the first FOV (closer to the second FOV), wherein the first motion vector is formed based on the difference between the second FOV and the first FOV. The method includes (e.g., as step "e") inserting the first additional video frame into the encoded video stream and includes marking the first additional video frame as a non-display frame. The method further includes (for example, as step "f") encoding the second image as part of a second video frame of the encoded video stream, wherein the second video frame references the first additional video frame.

[0007] As used herein, transforming an image's FOV to make it "closer to another FOV" also includes transforming an image's FOF to make it perfectly match another FOV. "At least partially overlapping" FOVs of two image frames should be understood as the presence of at least one object or at least a portion of an object visible in both image frames, such that when encoding the other image frame, motion vectors can be used to locate (or a portion thereof) the object within one image frame, and so on. A "non-display frame" should be understood as a frame that is marked in any way to indicate to the decoder that the frame is not presented as part of the decoded video stream, but the frame is still usable so that information can be obtained from the frame and used for decoding one or more other image frames to be displayed (i.e., not marked as non-display frames). It is conceivable that the disclosed method can be implemented using any video codec standard that supports the foregoing concepts of non-display frames, motion vectors, and the possibility of a frame referencing another (previous or subsequent) frame during encoding / decoding. Examples of such standards include (but are not limited to): High Efficiency Video Codec (HEVC) H.265, Advanced Video Codec (AVC) H.264, VP8, VP9, ​​AV1, and Universal Video Codec (VVC) H.266.

[0008] As envisioned in this paper, the first motion vector does not need to be calculated by performing a motion vector search, but can be derived / calculated based on knowledge relating to the corresponding first and second FOVs of the camera. For example, such knowledge may include data indicating, for example, orientation (as defined by, for example, translation angle / tilt angle / roll angle) and, for example, focal length of the camera lens (as changed during, for example, zoom operation), or similar data, sometimes including the time when the first FOV exists and the time when the second FOV exists.

[0009] As will be described in more detail later in this paper, the first aspect of the proposed approach improves upon currently available encoding and decoding techniques because it provides a more bitrate-efficient way to encode video streams during changes in the camera's field of view (FOV), such as during zoom operations. Specifically, since the (first) motion vector required to define the transformation from the first FOV to (or closer to) the second FOV can instead be calculated based on knowledge related to, for example, the camera's orientation and focal length (i.e., zoom level), and since no computationally cumbersome motion vector search is required, the need for searching long and changing motion vectors as part of encoding the second video frame is eliminated. Since the second video frame references the first additional video frame (where, at least after decoding, a transformed view of, for example, a particular zoomed-in object is available), finding the motion vectors for this reference is less important and can be performed using conventional search methods. In the case of, for example, zoom operations, the first additional video frame will encode a scaled and cropped version of the first image in the first video frame. Therefore, the bitrate can be reduced because motion vector searching becomes less critical, allowing for the finding of matches in the first additional video frame to encode the second video frame. Furthermore, the first additional video frame itself does not need to contain any new image data, but only motion vectors calculated based on differences in the field of view (FOV), and transformations such as scaling need to be performed efficiently. The envisioned method can also be applied to other transformations besides scaling, such as when the camera pans, tilts, and / or rolls. Panning or tilting operations may require perspective transformations that can be described in the additional video frame, and rolling operations may require rotation transformations that can also be described in the additional video frame.

[0010] In one or more embodiments of this method, encoding the video stream may include using a layered type of codec, such as, for example, "Scalable Video Codec" (SVC) or "Scalable High-Efficiency Video Codec" (SHVC). A first additional video frame may be inserted into the base layer of the encoded video stream, and a second video frame may be inserted into an enhancement layer of the encoded video stream. An example of a scalable video codec may be, for example, as defined in Appendix G extension of the H.264 / MPEG-4 AVC video compression standard. Another example of a scalable video codec may be, for example, as defined in Appendix H extension of the H.265 / MPEG-HHEVC video compression standard. As an example, the base layer may be encoded to provide a lower resolution (e.g., Full HD resolution or a similar resolution), while the enhancement layer may be encoded to provide a higher resolution (e.g., 4K resolution or a similar resolution). In a layered type of codec, one or more additional enhancement layers may of course be present, and it is conceivable that, for example, the second video frame may be encoded as part of any such enhancement layer. Using layered codecs such as SVC can, for example, help maintain the desired frames per second (FPS) because the first additional video frame and the second video frame can be inserted into different layers, thus avoiding the need for the first additional video frame to be inserted between, for example, the two video frames to be displayed.

[0011] In one or more embodiments of this method, the first additional video frame can be one of a prediction frame (“P-frame”) and a bidirectional prediction frame (“B-frame”). As used herein, these different types of video frames are defined as typically completed in contemporary codec standards. A P-frame can use / reference data from one or more previous video frames for decoding, while a B-frame can use / reference data from both one or more previous video frames and one or more future / forward video frames for decoding. Such video frames may also be referred to as “inter-frame frames” or “inter-codec frames” because they include references to one or more other video frames, in contrast to “intra-frame frames” or “intra-codec frames” (“I-frames”), which, for example, do not contain any references to one or more other video frames.

[0012] In one or more embodiments of the method, the method may include continuing to generate and insert one or more additional video frames while the camera's FOV changes (i.e., during zooming or similar operations). This may correspond to a method that further includes (e.g., as an additional step "g") obtaining a third image captured by a camera having a third FOV that is different from, but at least partially overlaps with, the second FOV. The method may further include (e.g., as an additional step "h") generating a second additional video frame including a second motion vector for transforming the image content of the second image to an FOV that is closer to the third FOV than the second FOV, wherein the second motion vector is formed based on the difference between the third FOV and the second FOV. The method may further include (e.g., as an additional step "i") inserting the second additional video frame into the encoded video stream, including marking the second additional video frame as a non-display frame. The method may further include (e.g., as an additional step "j") encoding the third image as part of a third video frame in the encoded video stream, wherein the third video frame references the second additional video frame. In other words, if more additional video frames are to be inserted into the already encoded video stream, the method may include repeating steps c through f above, but replacing “first and second” with “second and third”, and so on.

[0013] In other words, additional video frames can therefore be inserted as long as, for example, zooming is in progress. Instead of attempting to encode the actual images captured during the ongoing zooming operation (such as second and third images, and the like) by searching for motion vectors in previously captured frames, the images captured by the camera after the FOV change has begun (such as second and third video frames, and the like) are encoded by referring to the corresponding additional video frames (such as first and second additional video frames, and the like).

[0014] In one or more embodiments of the method, the difference between the first FOV and the second FOV (or, for example, the difference between the second FOV and the third FOV, etc.) may be due to zoom operations performed by the camera. The zoom operation may, for example, be performed concurrently with the execution of the method (such as when the encoded video stream represents a live stream of the captured scene). The contemplated method can also be used for transcoding, where the method is applied to already recorded video material and used, for example, to optimize (reduce) the size required to store such video material in storage (e.g., on a server). In this case, it is conceivable that, for example, data related to various FOVs of the camera during the transformation (e.g., the camera's azimuth, the focal length used to define each zoom level of the camera, the camera's mounting position relative to the scene, etc.) may be stored as metadata or similar data within the recorded video frames, for example, as part of the video frame header.

[0015] In one or more embodiments of this method, the zoom operation can be a zoom-in operation, such that the second FOV is a magnification of the first FOV (and, for example, such that the third FOV is a magnification of the second FOV, and so on). Applying this method while performing a zoom-in operation may be advantageous because, for example, a larger portion of each image can be referenced to the next image in the video stream. It should be noted that the contemplated method can also be applied, for example, during a zoom-out operation, such that, for example, the second FOV is instead a reduction / de-magnification of the first FOV, and so on, but with less advantage because only a smaller portion of each image can be referenced to the next image in the video stream.

[0016] In one or more embodiments of the method, the first and second images may be, for example, sequential images of a video stream captured by a camera, such that additional (non-display) video frames are added for each image of the video stream while performing camera transformation, and such that, for example, each encoded (to be displayed) video frame refers to a corresponding additional (non-display) video frame.

[0017] In one or more embodiments of the method, the method can be performed in a camera used to capture at least first and second images. This can allow, for example, the generation of an encoded video stream at the “edge” without the need for additional intermediate video processing equipment.

[0018] Typically, in this document, it is assumed that a video frame in the encoded video stream to be displayed (e.g., not marked as a non-display frame) references at least its corresponding additional (non-display) video frame, and each of these additional (non-display) video frames references at least a previous (non-display) video frame in the encoded video stream. Of course, it is also possible for a (display) video frame to also reference, for example, a previous (display) video frame in the encoded video stream. If layered codecs (such as SVC) are used, each video frame in the enhancement layer can therefore reference a corresponding additional video frame in the base layer, but may also reference, if desired, one or more previous (or even future / forward) video frames in the enhancement layer.

[0019] According to a second aspect of this disclosure, a corresponding apparatus for encoding a video stream is provided. The apparatus includes a processor and a memory. The memory stores instructions that, when executed by the processor, cause the apparatus to perform a method according to the first aspect. This includes causing the apparatus to: acquire a first image captured by a camera having a first field of view (FOV); encode the first image as a portion of a first video frame of an encoded video stream; acquire a second image captured by a camera having a second FOV that is different from, but at least partially overlaps with, the first FOV; generate a first additional video frame that references the first video frame and includes a first motion vector that transforms the image content of the first image into an FOV that is closer to the second FOV than the first FOV, wherein the first motion vector is formed / calculated based on the difference between the second FOV and the first FOV; insert the first additional video frame into the encoded video stream, including marking the first additional video frame as a non-display frame; and encode the second image as a portion of a second video frame of the encoded video stream, wherein the second video frame references the first additional video frame.

[0020] In one or more embodiments of the device, instructions stored in the device’s memory may further cause them, when executed by a processor, to cause the device to perform any embodiment of the method disclosed and discussed herein (the first aspect).

[0021] In one or more embodiments of the device, the device may be a camera for capturing at least first and second images.

[0022] According to a third aspect of this disclosure, a corresponding computer program is provided for encoding a video stream. The computer program is configured (i.e., includes instructions / computer code, which, when executed by a processor of a device (such as the device of the second aspect), to cause the device to perform the method of the first aspect. This includes causing the device to: acquire a first image captured by a camera having a first field of view (FOV); encode the first image as part of a first video frame of an encoded video stream; acquire a second image captured by a camera having a second FOV that is different from, but at least partially overlaps with, the first FOV; generate a first additional video frame that references the first video frame and includes a first motion vector that transforms the image content of the first image into an FOV that is closer to the second FOV than the first FOV, wherein the first motion vector is formed / calculated based on the difference between the second FOV and the first FOV; insert the first additional video frame into the encoded video stream, including marking the first additional video frame as a non-display frame; and encode the second image as part of a second video frame of the encoded video stream, wherein the second video frame references the first additional video frame.

[0023] In one or more embodiments of the computer program, the computer program (i.e., the instructions / computer code provided therewith) may be configured, when executed by a processor, to further cause the device to perform any embodiment of the method (first aspect) disclosed and discussed herein.

[0024] According to a fourth aspect of this disclosure, a corresponding computer program product is provided. The computer program product includes a computer-readable storage medium storing a computer program according to a third aspect (or any embodiment thereof described and discussed herein). As used herein, the computer-readable storage medium may be, for example, non-transitory and may be provided as, for example, a hard disk drive (HDD), a solid-state drive (SDD), a USB flash drive, an SD card, a CD / DVD, and / or any other storage medium capable of non-transitory storage of data. In other embodiments, the computer-readable storage medium may be transient and may, for example, correspond to a signal (electrical, optical, mechanical, or similar signal) present on, for example, a communication link, wiring, or similar signal transmission device.

[0025] Other objects and advantages of this disclosure will become apparent from the following detailed description, accompanying drawings, and claims. Within the scope of this disclosure, it is contemplated that all the features and advantages described with reference to, for example, the method of the first aspect are relevant to, applicable to, and can be used in combination with the apparatus of the second aspect, the computer program of the third aspect, and the computer software product of the fourth aspect, and vice versa. Attached Figure Description

[0026] The illustrated embodiments will now be described with reference to the accompanying drawings, in which:

[0027] Figure 1 The schematic map illustrates an example of a video stream, showing how the camera's field of view (FOV) changes during image capture.

[0028] Figure 2 Schematic diagrams illustrate applications according to embodiments of this disclosure. Figure 1 Examples of improved video stream coding methods;

[0029] Figure 3 A schematic diagram illustrates a flowchart of an improved video stream coding method according to embodiments of the present disclosure;

[0030] Figure 4 The schematic diagram illustrates an application of embodiments of this disclosure. Figure 1 Another example of an improved video stream coding method;

[0031] Figure 5 The schematic map illustrates an example of how an embodiment of the improved method according to this disclosure obtains motion vectors based on knowledge of how the camera's field of view (FOV) is altered, and

[0032] Figure 6 The schematic diagram illustrates an example of a device for encoding a video stream according to an embodiment of the present disclosure.

[0033] List of reference numerals

[0034] Video stream captured by 100 cameras

[0035] Objects in the FOV of a 110 camera

[0036] Images 120a-120f, first to sixth.

[0037] 200 video stream encoding methods

[0038] 210 encoded video stream

[0039] Video frames of 220a-220f encoded video streams

[0040] 230a-230c Additional Non-Display Video Frames

[0041] 240a-240c Reference Arrow

[0042] Flowchart of the 300-design method

[0043] Arrows 310 and 312 indicate the method flow.

[0044] S301-S307 Method Steps

[0045] 400 (layered) video stream coding methods

[0046] 410 (layered) encoded video stream

[0047] 412 Basic Layer

[0048] 414 reinforcement layer

[0049] Video frames in the 420a, 420e-420f base layers

[0050] Additional non-display video frames in the 430a-430c base layer

[0051] Video frames in enhancement layers 440a-440f

[0052] 510 (video) camera

[0053] The first and second translation angles of the 520a-520b cameras

[0054] The first and second tilt angles of the 522a-522b cameras

[0055] The first and second roll angles of the 524a-524b cameras

[0056] The first and second focal lengths of the 526a-526b cameras

[0057] 530a-530b First Motion Vector

[0058] 540 Overlay of the first and second images

[0059] 600 equipment

[0060] 610 processor

[0061] 612 memory

[0062] 614 Additional Items

[0063] 614 network interface

[0064] 620 communication bus

[0065] The first and second FOVs of FOV1 and FOV2 cameras

[0066] In the accompanying drawings, unless otherwise stated, the same reference numerals will be used for the same elements. Unless explicitly stated otherwise, the drawings only show those elements necessary to illustrate the exemplary embodiments, while other elements may be omitted or merely conjectured for clarity. As illustrated in the figures, for illustrative purposes, the (absolute or relative) dimensions of elements and areas may be exaggerated or underestimated relative to their true values, and are therefore provided to illustrate the general structure of the embodiments. Detailed Implementation

[0067] Figure 1 The schematic map illustrates a typical example of a video stream 100 captured by a (video) camera, where the camera's field of view (FOV) changes during capture. In this particular example, the change in FOV is caused by the camera performing a zoom operation (and more specifically a zoom-in operation), which also includes slight panning and tilting of the camera to move the object of interest to the center of the camera's FOV.

[0068] For this particular example, suppose the scene comprises a single object 110 of interest. Here, object 110 is assumed to be a flower in a flowerpot, but it could certainly be any combination of one or more other objects. Video stream 100 includes a first image 120a captured by the camera at a first field of view (FOV). Here, the first FOV is such that object 110 occupies only the lower left corner of the first image 120a. Over time, the camera then begins to zoom in on object 110, for example, by adjusting its lens and / or by using digital processing to simulate this physical zoom, and also slightly pans to the left and slightly tilts downwards, so that object 110 is centered in the middle of the FOV.

[0069] When the camera captures the next second image 120b, the camera thus has a second FOV, which provides magnification of the object 110 compared to the first FOV. When depicted in the second image 120b, the object 110 is slightly larger than when depicted in the first image 120a, and due to the translation and tilt of the camera, the object 110 also begins to move towards the center of the image.

[0070] As time progresses, the camera continues to zoom while capturing more images of the scene, causing object 110 to become, for example, larger and larger, occupying an increasingly larger portion of each image. Figure 1In the specific example shown, this includes capturing a third image 120c with a camera having a third FOV, and capturing a fourth image 120d with a camera having a fourth FOV. After the fourth image 120d has been captured, the camera stops zooming, in which object 110 now occupies almost all of the available image space and is more or less centered on that FOV. For successive frames, such as the fifth image 120e and the sixth image 120f, the camera's FOV remains unchanged and is equal to the fourth FOV used to capture the fourth image 120d.

[0071] To encode video stream 100 in order to save, for example, bit rate / bandwidth, various encoding schemes can be applied. For example, a conventional scheme of this kind could begin by encoding the first image 120a as an intra-predictive frame (I-frame) or similar frame, without referencing one or more additional video frames. The second image 120b could then be encoded as an inter-predictive frame (P-frame), providing a reference to the I-frame used to encode the first image 120a in, for example, the form of motion vectors. However, to find such motion vectors, the encoder would first have to perform a motion vector search, in which the encoder attempts, for each portion of the second image 120b, to identify which portions of the first image 120a should be retrieved for later decoding of the P-frames used to encode the second image 120b. This is typically achieved by defining a finite search region around each portion of the second image 120b and then searching that region throughout the first image 120a for a suitable match. If the changes on the camera's FOV do not include zoom operations, but only include, for example, slow offset movements of the camera (such as panning), this is likely an efficient way to encode P-frames (and thus the second image 120b), since the required search area will remain small and the motion vector will be roughly the same for all parts of the second image 120b.

[0072] However, in such Figure 1In the case of the zoom operation shown, searching for such motion vectors may become more difficult and require expanding the search area. Furthermore, the motion vectors will differ for many or even all parts of the second image 120b, because, for example, the movement of pixels belonging to the flowerpot will be less than, for example, the movement of pixels belonging to the top of the flower, and because, compared to the first image 120a, pixels belonging to, for example, the left side of object 110 will only move approximately upwards in the second image 120, while pixels belonging to, for example, the right side of object 110 will move both upwards and to the right. As zooming proceeds and the different parts of object 110 begin to move further away from each other more quickly with each newly captured image (and because each new P-frame will have to reference, for example, previously encoded P-frames), the motion vectors will become increasingly longer, and the differences (in direction) for the different parts of the second image 120b and the P-frames encoding that second image 120b will also become increasingly significant. As a result, the processing power required to perform such a motion vector search risks exceeding the capabilities of, for example, the entity responsible for encoding, which in practice could be, for example, the camera itself.

[0073] Now we will refer to, for example Figure 2 and Figure 4 This will describe in more detail how the method envisioned in this paper can improve this situation. References also include... Figure 3 , Figure 3 A schematic diagram illustrates the flowchart 300 of the steps of this method.

[0074] Figure 2 A schematic diagram illustrates the encoding of various video frames performed as part of method 200. As contemplated herein, the improved method 200 includes (as step S301) obtaining a first image (e.g., first image 120a) captured by a camera having a first FOV. Method 200 further includes (as step S302) encoding the first image 120a as part of a first video frame 220a of the encoded video stream 210. This can be done conventionally by, for example, encoding the first video frame 220a as an I-frame or similar frame, such that the first video frame 220a contains all the necessary image data required when the first image 120a is decoded again on the decoder side (not shown). This encoding can be, for example, lossless or may result in at least some image quality loss when decoding the first video frame 210a, depending on the exact method used for such encoding of the first video frame 220a.

[0075] Method 200 includes (as step S303) receiving a second image (e.g., second image 120b) captured by a camera having a second FOV. See preceding references herein for example... Figure 1As described, the first FOV differs from the second FOV, but the second FOV overlaps with the first FOV at least partially. In other words, for method 200 to function as intended, preferably, there should be at least some portions of object 110 that are visible in both the first image 120a and the second image 120b. More preferably, all portions of object 110 visible in the second image 120b should also be visible in the first image 120a.

[0076] In contrast to conventional encoding methods, the envisioned improved method 200 does not directly encode the second image 120b into a conventional P-frame by attempting to search for motion vectors in the first image 120a. Instead, in step S304, method 200 proceeds by generating a first supplementary video frame 230a, which references the first video frame 220a, and includes first motion vectors used to transform (in this case, scale and offset) the image content of the first image 120a to an FOV that is closer to the second FOV than the first FOV. These first motion vectors are not found through computationally intensive motion vector searches, but are calculated / formed based on knowledge relating to the second and first FOVs, i.e., how the camera's FOV changes between capturing the first image 120a and the second image 120b. Such information can be received, for example, from the camera itself, and / or from any other specific sensor capable of detecting, for example, the camera's orientation (such as translation / tilt / roll angles or similar angles, and / or the focal length of the camera and its lens). For example, it can be known that between capturing the first image 120a and the second image 120b, the camera's FOV has been altered, for example, by increasing the lens's focal length by a specific amount and by slightly tilting the camera downwards while slightly translating it to the left (so that the object 110 is moved toward the center of the image while zooming). This information can then be used to directly obtain the first motion vector required to perform a scaling / offset that transforms the FOV of the first image 120a to a FOV closer to or equal to the second FOV used to capture the second image 120b. As used herein, "the FOV of an image is transformed to some other FOV" should be understood as meaning that when a transformation is calculated, if that transformation is applied to the image content of the image, the resulting image will be rendered to look (more) like it was captured by a camera with that other FOV. Reference will be made here later to, for example... Figure 5 To describe in more detail how the first motion vector is formed / calculated based on knowledge of the various FOVs used to capture the two images.

[0077] After the first motion vector has been formed / calculated, method 200 continues by inserting the first additional video frame 230a into the encoded video stream 210 (in step S305), and includes marking the first additional video frame 230a as a non-display frame (as shown in the image). Figure 2 (Represented by the crossed-out screen symbol shown). Therefore, the first additional video frame 230a will not be displayed during subsequent decoding of the encoded video stream 210, but will only function as a container for information decoded / encoded for other video frames. Specifically, in this example, the first additional video frame 230a does not need to include any actual image data, but only a first motion vector, which allows (during subsequent decoding of the first additional video frame 230a) the FOV of the first image 120a to be transformed into an FOV that looks closer to the second FOV, i.e., by scaling, offsetting, and / or rotating the image content of the first image 120a as needed.

[0078] Then, in step S306, the method continues to encode the second image 120b as a portion of the second video frame 220b of the encoded video stream 210, wherein the second video frame 220b references the first additional video frame 230a. The second video frame 220b can be, for example, a P-frame or a similar frame, wherein, for example, motion vectors indicate which portions of the (decoded) first additional video frame 230a will be used to decode which portions of the second video frame 220b. Importantly, since the FOV of the (to-be-decoded) image of the first additional video frame 230a appears to be closer to (or even equal to) the FOV of the second image 120b, using motion vector search to find such motion vectors for decoding the second video frame 220b becomes less important than attempting to find such motion vectors directly based on, for example, the image content of the first image 120a. The motion vector for the second video frame 220b can be found, for example, by comparing the image content of the second image 120b with the image content of the offset first image 120a included in (or generated from) the first additional video frame 230a after decoding. Therefore, the envisioned method provides an improved way to encode video streams captured during camera FOV changes.

[0079] exist Figure 2In this document, solid arrows are used to illustrate references from one encoded video frame to another. For example, solid arrow 240a illustrates how a first additional video frame 230a references a first video frame 220a; solid arrow 240b illustrates how a second video frame 220b references a first additional video frame 230a; and solid arrow 240c illustrates how a second additional video frame 230b (if included, as discussed later below) references a first additional video frame 230a, and so on. As used herein, arrows point in the direction of information transmission, such that arrow 240a pointing from the first video frame 220a to the first additional video frame 230a indicates that the first additional video frame 230a is a video frame referencing the first video frame 220a. As contemplated herein, in some embodiments of this method, other references may, of course, be included within the various video frames. Examples of such additional references are found in… Figure 2 The reference is indicated by a dashed arrow and may include, for example, a second video frame 220b that also references the first video frame 220a, for portions where motion vectors are not required (such as constant background or similar portions, etc.). Other such optional references may include, for example, one or more frames as bidirectional prediction frames (B-frames), including references to one or more “future” video frames.

[0080] In addition to the foregoing, in some embodiments, the contemplated method may, of course, continue in a similar manner to encode one or more additional images captured as the camera's FOV changes, for example, as part of an ongoing zoom operation. Thus, the method as contemplated herein and described as in flowchart 300 may include an optional step S307 in which it checks whether the camera's FOV is still changing. If so, the method may return (as illustrated by dashed arrow 310) and repeat steps S303 through S306, each time a new image is acquired, generating a new, non-display additional video frame referencing the previously attached video frame, and encoding the new image as part of a new video frame referencing that new additional video frame. If the camera's FOV is not still changing, the method may instead, for example, stop and, for example, revert to encoding frames as normally would (as illustrated by dashed arrow 312).

[0081] like Figure 2As illustrated, continuing method 200 may include, for example, obtaining a third image (e.g., third image 120c) captured by a camera having a third FOV, wherein the third FOV differs from the second FOV but also overlaps at least partially with the second FOV. This may correspond to, for example, repeating step S303 above, but for "second and third" instead of "first and second," etc. The method may, for example, include generating a second additional video frame 230b, which references a first additional video frame 230a, and which includes a second motion vector that transforms the image content of the second image 120b into an FOV that is closer to the third FOV than the second FOV, in a manner similar to the first motion vector described above. Here, instead, the second motion vector is formed based on the difference between the third FOV and the second FOV. This may correspond to, for example, repeating step S304 above, but may also use "second and third" instead of "first and second," etc. The method can continue by inserting a second additional video frame 230b into the encoded video stream 210 and marking the second additional video frame 230b as a non-display frame (corresponding to repeating step S305 above, also using "second and third" instead of "first and second," etc.). The method can continue by encoding a third image 120c as a portion of a third video frame 220c, which may reference, for example, the second additional video frame 230b. In some embodiments, the method can continue in a similar manner until the camera's FOV change ends. In this example, this may include: obtaining a fourth image (e.g., a fourth image 120d); generating a third additional video frame 230c referencing the second additional video frame 230b; encoding the fourth image 120d in the encoded video stream 210 as a portion of the fourth video frame 230d referencing the third additional video frame 230c, etc. When it is detected, for example, that zooming has stopped, for instance, by noting that the camera's fourth FOV is the same when capturing the fourth image 104d as its fifth FOV is the same when capturing the fifth image 104e, in some embodiments, the method may, for example, stop inserting additional video frames and instead revert to encoding frames, such as the frame encoding performed when, for example, no zooming is currently in progress. For example, the fifth image 104e may be encoded as a portion of the fifth video frame 220e referencing the fourth additional video frame 230d, and the next image 230f may be encoded as video frame 220f referencing video frame 230e, and so on. In particular, the method envisioned herein is primarily applied to video frame encoding corresponding to times during which the camera changes its FOV, and during which conventional encoding and decoding using motion vectors is difficult because such motion vectors are long and change due to the camera's FOV variation.

[0082] Although the example has been given only by the change in FOV caused by zooming in (plus additional translation / tilt) performed by the camera, it should be noted that the envisioned method can also be used to encode a video stream during recording of one or more additional changes to the camera's FOV and / or for other reasons related to such changes. Such examples could include, for instance, camera rotation, or any other combination of feasible variations associated with translation, tilt, roll, and / or focal length changes or similar actions. It is conceivable that, provided knowledge of the different FOVs used when recording different images is known (or otherwise derived), the motion vectors required to generate the additional video frames can be obtained directly from that knowledge without any motion vector search by a decoder such as the camera. Therefore, as long as the FOV of the additional video frames can better or more accurately match the FOV of the images captured by the camera, motion vector searches can be performed more efficiently when encoding the captured images by searching within and referencing the additional video frames.

[0083] Now refer to Figure 4 To explain another embodiment of the conceived method, let's assume that method 400 is still considered as encoding the video stream 100 captured during the zoom-in of object 110, as... Figure 1 As shown, however, compared with the reference Figure 2 In contrast to the described method 200, method 400 instead uses a layered type of codec, such as Scalable Video Codec (SVC). The differences between method 200 and method 400 will be emphasized only in the following text.

[0084] The encoded video stream 410 has multiple layers, including at least one base layer 412 and at least one enhancement layer 414. Method 400 begins by obtaining a first image 120a and encoding that image as a portion of a first video frame 420a in the base layer 412 (having, for example, a lower resolution) and a portion of a first image frame 440a in the enhancement layer 414 (having details to obtain a higher resolution than that of the base layer 412), as is conventional in such a layered encoding / decoding scheme.

[0085] After obtaining the second image 120b, a first additional video frame 430a is generated as described earlier herein and with reference to the first video frame 420a of the base layer 412. The first additional video frame 430a is also encoded in the base layer 412 and marked as a non-display frame.

[0086] However, the second image 120b is not directly encoded as part of any video frame in the base layer 412, but is encoded only as part of the second video frame 440b in the enhancement layer 414, wherein the second video frame 440b references the first additional video frame 430a in the base layer 412.

[0087] In some embodiments, in the same manner as described above, method 400 may, of course, continue to generate and encode more video frames, as long as the camera's FOV is changing. A third image 120c can be obtained, the third FOV used to capture the third image 120c can be compared with the second FOV used to capture the second image 120b, and a second motion vector can be calculated and used to generate a second additional video frame 430b encoded in the base layer 412. As before, the third image 120c is not directly encoded in the base layer 412, but is encoded as a third video frame 440c in the enhancement layer 414, and such that the third video frame 440c references the second additional video frame 430b in the base layer 412. The same process can then, of course, be repeated to obtain, for example, a fourth image 120d, a fifth image 120e, etc., and the corresponding frames can be generated and inserted into the base layer 412 and enhancement layer 414 as described herein. In this example, this includes: generating a third additional video frame 430c in the base layer 412 (marked as non-display and referencing the second additional video frame 430b); encoding a fourth image 120d as part of a fourth video frame 440d in the enhancement layer 414 (where the fourth video frame 440d refers to the third additional video frame 430c in the base layer 412), and so on. Once the zooming in on object 110 has stopped, method 400 can stop generating and inserting additional video frames and instead return to encoding the fifth image 120e and the sixth image 120f in the base layer 412 and enhancement layer 414. For example, this could include: encoding a fifth image 120e in a fifth video frame 420e in a base layer 412 (which has a lower resolution) and a fifth video frame 440e in an enhancement layer 414 (which has details to achieve a higher resolution than the base layer 412); and encoding a sixth image 120 in a sixth video frame 420f in a base layer 412 and a sixth video frame 440f in an enhancement layer 414. Figure 4 As seen in method 400, additional video frames 230a, 230b and 230c thus replace the second, third and fourth normally encoded video frames in the base layer 412.

[0088] like Figure 4 The solid arrows provided in the image are illustrated and have been referenced. Figure 2As described, the fifth video frame 420e in the base layer 412 may, for example, refer to the third additional video frame 430c, and so on. Similarly, as described herein, other references may be included among the various video frames. For example, such as... Figure 5 As illustrated by the dashed arrows, each video frame in enhancement layer 414 may also reference one or more previous video frames in enhancement layer 414, and each “normal” video frame (i.e., non-attached video frame) in base layer 412 may reference one or more previous video frames in base layer 412. Of course, other references, such as references to future frames or similar frames (if believed), may exist.

[0089] When in reference Figure 2 Compared to the described method 200, the advantage of method 400 is that, since all the additional video frames (e.g., frames 430a-430c) are inserted into the base layer 412, they do not affect the FPS rate because the enhancement layer 414 still only includes one encoded video frame for each image captured by the camera. It is also conceivable that even if enhancement layer 414 is not decoded, for example when the encoder's processing power is insufficient for decoding, or if enhancement layer 414 is even removed from the encoded video stream before being sent via, for example, a network (to save bandwidth), this can still be somewhat useful when only the base layer 412 is retrieved and decoded, because the "FOV-transformed" image content of the various captured images generated using motion vectors for scaling, offset, and / or rotation found in the additional frames can still be used to render the image content.

[0090] Figure 5 A more detailed schematic map illustrates how the calculated result of the desired (first) motion vector is obtained. Here, as an example, camera 510 with a first FOV (“FOV1”) captures a first image 120a of object 110. At another (e.g., later) time, the same camera 510 captures a second image 120b of object 110, while having a different second FOV (“FOV2”). For illustrative purposes only, Figure 5 In the examples provided, such as Figure 1 and Figure 2 In comparison, the difference between the first FOV and the second FOV (and the resulting difference between the first image 120a and the second image 120b) is in Figure 5 The claim is exaggerated.

[0091] When capturing the first image 120a, the first FOV (FOV1) is defined, for example, by a camera 510 having a first translation angle 520a, a first tilt angle 522a, a first roll angle 524a, and a first focal length 526a. When capturing the second image 120b, the second FOV (FOV2) is defined by a camera 510 having a second translation angle 520b, a second tilt angle 522b, a second roll angle 524b, and a second focal length 526b. In this particular example, as discussed above, the roll angles 524a-524b are the same, while the second translation angle 520b is slightly smaller than the first translation angle 520a, the second tilt angle 522b is slightly larger than the first tilt angle 522a, and the second focal length 526b is slightly larger than the first focal length 526a (because the camera is closer in the second FOV than in the first FOV, and slightly translated / tilted to the left / down). The exact definitions of various angles are not considered important, as long as the definitions allow for the measurement / detection of how each angle changes between one FOV and another.

[0092] By having knowledge related to the first and second FOVs of camera 510 (e.g., by having knowledge related to angles 520a-520b, 522a-522b, 524a-524b and focal length 526a-526b), the first motion vector (e.g., 530a and 530b, etc.) can be calculated. Figure 5 As illustrated in the superposition 540 of the first image 120a and the second image 120b shown. As can be seen in superposition 540, the first motion vectors 530a and 530b can be different for different portions of the second image 120b (e.g., 550a and 550b) because the scaling and offset required to start from the first FOV and move closer to the second FOV can cause different portions of the object 110 to move in different directions and at different speeds. Of course, besides Figure 5 In addition to the first motion vector shown in the figure, other additional first motion vectors may exist, for example, for each part of the second image 120b.

[0093] Here, we also envision providing a device for encoding video streams, which will now be referred to Figure 6 Let's describe it in more detail.

[0094] Figure 6 The schematic diagram illustrates a device 600 for encoding video streams, as discussed earlier in this document, i.e., configured to perform, for example, referencing... Figure 2 The described method 200 and / or reference Figure 4The apparatus described in method 400. Apparatus 600 includes at least a processor (or “processing circuitry”) 610 and a memory 612. As used herein, a “processor” or “processing circuitry” can be, for example, any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller (µC), digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), graphics processing unit (GPU), etc., capable of executing software instructions stored in memory 612. Memory 612 can be external to processor 610 or internal to processor 610. As used herein, “memory” can be any combination of random access memory (RAM) and read-only memory (ROM), or any other type of memory capable of storing instructions. Memory 612 contains (i.e., stores) instructions that, when executed by processor 610, cause apparatus 600 to perform the methods described herein (i.e., methods 200, 400, or any embodiments thereof). Apparatus 600 may further include one or more additional items 614, which may be useful in some cases for performing the method. In some example embodiments, device 600 may be, for example, a (video) camera, such as a video surveillance camera, and additional item 614 may therefore include, for example, an image sensor and, for example, one or more lenses for focusing light from the scene onto the image sensor, such that the surveillance camera can capture an image of the scene as part of performing the contemplated method. Additional item 614 may also include, for example, various other electronic components required for capturing the scene, such as for appropriately operating the image sensor and / or lenses as needed, and for collecting information directly related to, for example, the current orientation and / or focal length of the camera and lenses, which can be used to calculate how to create a motion vector based on, for example, the difference between two FOVs. Performing this method in a surveillance camera can be useful because processing is moved to the “edge,” i.e., closer to the actual scene capture location than performing processing and video encoding in other locations, such as in a more centralized processing server or similar location. Device 600 may be connected to a network, for example, so that the encoded video stream resulting from performing this method can be transmitted to a user. For this purpose, device 600 may include network interface 616, which may be, for example, a wireless network interface (as defined in any of the standards such as IEEE 802.11 or later, supporting, for example, Wi-Fi) or a wired network interface (as defined in any of the standards such as IEEE 802.3 or later, supporting, for example, Ethernet). Network interface 616 may also support, for example, any other wireless standard capable of transmitting encoded video, such as Bluetooth or similar standards.Various components 610, 612, 614 and 616 (if present) can be connected via one or more communication buses 620, enabling these components to communicate with each other and exchange data as needed.

[0095] Device 600 may be, for example, a surveillance camera installed or mountable on a building, such as a PTZ camera or a fisheye camera capable of providing a wider field of view of the scene, or any other type of surveillance / monitoring camera. Device 600 may be, for example, a human body camera, action camera, dashcam, or the like suitable for mounting on people, animals, and / or various vehicles. Device 600 may be, for example, a user-portable smartphone or tablet that can capture scenes. In any such example of device 600, it is contemplated that device 600 may include all necessary components (if any) other than those already explained herein, provided that device 600 is still capable of performing methods 200, 400, or any embodiments thereof as contemplated herein.

[0096] Despite Figure 6 Although not explicitly shown, device 600 and processor 610 may, for example, be configured as one or more computing blocks, each computing block responsible for performing functions according to, for example, specific steps of methods 200 and 400, as in Figure 3 As shown in flowchart 300. For example, one such functional block can be configured to obtain the first image 120a (i.e., perform step S301), and another functional block can be configured to encode the first image (i.e., perform step S302), and so on, until all steps of methods 200, 400 or other embodiments thereof have been performed by the device 600.

[0097] Summarizing the various embodiments presented herein, this disclosure provides an improved method for encoding a video stream during image capture, in the event of changes in the camera's field of view (FOV), such as during zoom operations performed by the camera. More efficient and less computationally intensive encoding can be performed by inserting one or more additional, non-display frames that continuously transform the image content of images captured before the FOV change begins to match the camera's FOV at various stages of the FOV change, and by basing this transformation on motion vectors found directly from knowledge related to the FOV (and without requiring motion vector search). Then, by performing a motion vector search (which does not result in / does not originate from, for example, long and varying motion vectors), various images of the bitstream during the FOV change can be encoded using motion vectors referenced from different additional video frames. As described herein, the envisioned method, along with corresponding devices, computer programs, and computer program products, thus fulfills the obvious need for encoding such video streams, since encoding techniques based solely on motion vector search often prove computationally intractable, especially in real-time or near-real-time scenarios.

[0098] Although the features and elements can be described above in specific combinations, each feature or element can be used alone without the other features and elements, or can be used in various combinations with or without the other features and elements. Furthermore, based on a study of the drawings, this disclosure, and the appended claims, those skilled in the art will understand and implement modifications to the disclosed embodiments.

[0099] In the claims, the words "comprising" and "including" do not exclude other elements, and the terms "a" or "an" do not exclude a plurality. The fact that certain features are listed only in mutually different dependent claims does not mean that a combination of these features cannot be used to obtain benefits.

Claims

1. A method for encoding a video stream, comprising: a) Obtain the first image of the scene captured by the camera at the first moment; b) Encode the first image as a portion of a first video frame of an encoded video stream; c) Obtain a second image of the scene captured by the camera at a second time point; d1) Obtain data indicating a first difference in the orientation, zoom level, and / or position of the camera relative to the scene at the first time point compared to the second time point, and calculate a first motion vector based on the indicated first difference; d2) Generate a first additional video frame that references the first video frame, the first additional video frame including the first motion vector for transforming the image content of the first image to transform the image content of the first image into a field of view that is closer to a second field of view of the second image than a first field of view of the first image; e) Inserting the first additional video frame into the encoded video stream, including marking the first additional video frame as a non-display frame; as well as f) Encode the second image as part of a second video frame of the encoded video stream, wherein the second video frame references the first additional video frame.

2. The method according to claim 1, further comprising: Using a layered type encoding / decoding, wherein the first additional video frame is inserted into the base layer of the encoded video stream, and the second video frame is inserted into the enhancement layer of the encoded video stream.

3. The method according to claim 1, wherein, The first additional video frame is either a prediction frame (P-frame) or a bidirectional prediction frame (B-frame).

4. The method according to claim 1, further comprising: g) Obtain a third image of the scene captured by the camera at a third time point; h1) Obtain data indicating a second difference in the orientation, zoom level, and / or position of the camera at the second time and the third time, and calculate a second motion vector based on the second difference; h2) Generate a second additional video frame that references the first additional video frame, wherein the second additional video frame includes the second motion vector for transforming the image content of the second image to transform the image content of the second image into a field of view that is closer to the third field of view of the third image than the second field of view of the second image. i) Inserting the second additional video frame into the encoded video stream, including marking the second additional video frame as a non-display frame, and j) Encode the third image as part of a third video frame of the encoded video stream, wherein the third video frame references the second additional video frame.

5. The method according to claim 1, wherein, The first difference is caused by the zoom operation performed by the camera.

6. The method according to claim 5, wherein, The zoom operation is a zoom-in operation.

7. The method according to claim 1, wherein, The first image and the second image are sequential images of a video stream captured by the camera.

8. An apparatus for encoding a video stream, comprising: processor, and The memory stores instructions that, when executed by the processor, cause the device to: a) Obtain the first image of the scene captured by the camera at the first moment; b) Encode the first image as a portion of a first video frame of an encoded video stream; c) Obtain a second image of the scene captured by the camera at a second time point; d1) Obtain data indicating a first difference in the orientation, zoom level, and / or position of the camera relative to the scene at the first time point compared to the second time point, and calculate a first motion vector based on the indicated first difference; d2) Generate a first additional video frame that references the first video frame, the first additional video frame including the first motion vector for transforming the image content of the first image to transform the image content of the first image into a field of view that is closer to a second field of view of the second image than a first field of view of the first image; e) Inserting the first additional video frame into the encoded video stream, including marking the first additional video frame as a non-display frame; as well as f) Encode the second image as part of a second video frame of the encoded video stream, wherein the second video frame references the first additional video frame.

9. The device according to claim 8, wherein, The instructions, when executed by the processor, further cause the device to perform layered encoding / decoding, wherein the first additional video frame is inserted into the base layer of the encoded video stream, and the second video frame is inserted into the enhancement layer of the encoded video stream.

10. The device according to claim 8, wherein, The device is a camera used to capture the first image and the second image.

11. A computer program product for encoding a video stream, configured to, when executed by a processor of a device including a computer-readable storage medium, cause the device to: a) Obtain the first image of the scene captured by the camera at the first moment; b) Encode the first image as a portion of a first video frame of an encoded video stream; c) Obtain a second image of the scene captured by the camera at a second time point; d1) Obtain data indicating a first difference in the orientation, zoom level, and / or position of the camera relative to the scene at the first time point compared to the second time point, and calculate a first motion vector based on the indicated first difference; d2) Generate a first additional video frame that references the first video frame, the first additional video frame including the first motion vector for transforming the image content of the first image to transform the image content of the first image into a field of view that is closer to a second field of view of the second image than a first field of view of the first image; e) Inserting the first additional video frame into the encoded video stream, including marking the first additional video frame as a non-display frame; as well as f) Encode the second image as part of a second video frame of the encoded video stream, wherein the second video frame references the first additional video frame.

12. The computer program product according to claim 11, further configured as follows: The device is configured to use a layered encoding / decoding method, wherein the first additional video frame is inserted into the base layer of the encoded video stream, and the second video frame is inserted into the enhancement layer of the encoded video stream.

Images