Encoding of video stream during changing camera field-of-view
By generating additional video frames that reference previous frames using FOV-derived motion vectors, the method addresses the inefficiencies in encoding video streams during camera FOV changes, reducing bitrate and computation time.
Patent Information
- Application Number
- KR1020240033805
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2023-04-03
- Filing Date
- 2024-03-11
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2044-03-11
AI Technical Summary
Existing video coding algorithms struggle with efficiently encoding video streams when a camera's field of view (FOV) changes, particularly during zoom operations, due to the increased complexity and computation time required for motion vector searches as the ideal length of motion vectors varies, making the use of motion vectors impractical.
A method that involves generating additional video frames that reference previous frames, using motion vectors derived from the difference in camera FOVs, and encoding these frames as non-display frames, allowing for efficient encoding by transforming image content without direct motion vector searches.
This method reduces bitrate requirements and computation time by eliminating the need for extensive motion vector searches, enabling efficient encoding of video streams during camera FOV changes, such as zoom operations, while maintaining image quality.
Smart Images

Figure 112024034909960-PAT00004_ABST
Abstract
Description
Technology Field
[0001] The present disclosure relates to the encoding of a video stream recorded by a camera. In particular, the present disclosure relates to the encoding of a video stream during a time when the camera's field of view is changed. Background Technology
[0003] In modern video coding algorithms, predictive coding is often used to leverage the fact that many details of a scene may remain static between one image frame and the next. Additionally, so-called "motion vectors" can also be used to describe objects moving between image frames, instructing the decoder to locate image data belonging to an object in one or more other image frames when decoding the current image frame containing the object.
[0004] However, in cases involving specific types of movement and object scaling between image frames—for instance, caused by changing the camera's field of view (FOV) between capturing two image frames—using these motion vectors to efficiently encode / decode the video stream can be difficult. One such situation occurs when the camera performs a zoom operation, resulting in motion vectors becoming longer over time and varying for different parts of the image frame. As the ideal length of a motion vector increases, the encoder must expand the search area, for instance, from the previous image frame, to locate these vectors. Consequently, this can lead to increased computation time, often making the use of motion vectors impractical and sometimes even difficult to handle.
[0005] Therefore, an improved method is required to encode a video stream into image frames captured while the camera's FOV is changing. The problem to be solved
[0007] To satisfy at least partially the identified need above, the present disclosure provides an improved method, apparatus, computer program, and computer program product for encoding a video stream, as defined in the appended independent claim. Various embodiments of the method, apparatus, computer program, and computer program product are defined in the appended dependent claim. means of solving the problem
[0009] According to a first aspect of the present disclosure, a method for encoding a video stream is provided. The method comprises the step of acquiring a first image captured by a video camera having a first field of view (FOV) (e.g., as step "a"). The method comprises the step of encoding the first image as part of a first video frame of an encoded video stream (e.g., as step "b"). The method comprises the step of acquiring a second image captured by a video camera while having a second FOV that is different from but at least partially overlaps with the first FOV (e.g., as step "c"). The method comprises the step of generating a first additional video frame that references the first video frame (e.g., as step "d") and including a first motion vector that transforms the image content of the first image to an FOV that is closer to the second FOV than the first FOV. Herein, the first motion vector is formed based on the difference between the second FOV and the first FOV. The method includes the step of inserting a first additional video frame into an encoded video stream (e.g., as step "e") and the step of marking the first additional video frame as a non-display frame. The method further includes the step of encoding a second image as part of a second video frame of the encoded video stream (f.g., as step "f"), wherein the second video frame refers to the first additional video frame.
[0010] As used herein, the statement that the FOV of an image is transformed to be "closer to another FOV" includes situations where the FOV of an image is transformed to exactly match another FOV. The statement that the FOVs of two image frames "overlap at least partially" should be understood as meaning that there is at least one object or at least one part of an object visible in both image frames, and that the object (or part of it) can be found in one image frame when encoding another image frame, etc., using motion vectors. A "non-display frame" should be understood as a frame that is flagged in some way to instruct a decoder not to display the frame. The frame may continue to be used so that it can be used for decoding one or more other image frames (i.e., not displayed as a non-display frame) from which information can be obtained and which will be displayed, even though it is rendered as part of a decoded video stream. Any video coding standard that supports the above concepts regarding non-display frames, motion vectors, and the possibility of one frame referencing another (previous or subsequent) frame during encoding / decoding may be used to realize the disclosed method. Examples of such standards include, but are not limited to, High Efficiency Video Coding (HEVC) H.265, Advanced Video Coding (AVC) H.264, VP8, VP9, AV1, and Versatile Video Coding (VVC) H.266.
[0011] As conceived in this specification, the first motion vector does not need to be calculated by performing a motion vector search, but can be derived / calculated based on knowledge of the respective first and second FOVs of the camera. For example, such knowledge may include data indicating a direction (e.g., defined by a pan / tilt / roll angle), the focal length of the camera lens (e.g., changed during zoom-action), or similar things at a time including both the time when the first FOV exists and the time when the second FOV exists.
[0012] As will be described in more detail later in this specification, the method conceived of the first aspect improves currently available coding techniques in that it provides a more bitrate-efficient method for encoding a video stream while the camera's FOV is changing, for example, during a zoom operation. In particular, the need to search for long and varied motion vectors during the encoding of the second video frame is eliminated because the (first) motion vector required to define the transition from the first FOV to (or closer to) the second FOV can instead be calculated from knowledge of, for example, the camera orientation and focal length (e.g., zoom level) and, for example, without the need to perform a computationally cumbersome motion vector search. Since the second video frame references the first additional video frame (at least after decoding, for example, a transformed view of a specific zoomed object is available), finding the motion vector for this reference is simpler and can be performed using existing search methods. For example, in the case of zoom-motion, the first additional video frame encodes a scaled and cropped version of the first image in the first video frame, and the bitrate can be reduced because motion vector search can more simply find a match in the first additional video frame to encode the second video frame. Additionally, the first additional video frame does not need to contain the new image data itself, but only motion vectors calculated based on the FOV difference, which is necessary to efficiently perform transformations such as scaling. The method envisioned can also be applied to transformations other than scaling, such as when the camera pans, tilts, and / or rolls. Panning or tilting operations may require perspective transformations that can be described in the additional video frame, and rolling operations may require rotation transformations that can be described in the additional video frame.
[0013] In one or more embodiments of the method, encoding the video stream may involve using a layered type of coding, such as "Scalable Video Coding" (SVC) or "Scalable High Efficiency Video Coding" (SHVC). A first additional video frame may be inserted into the base layer of the encoded video stream, and a second video frame may be inserted into the enhancement layer of the encoded video stream. An example of Scalable Video Coding may be, for example, as defined in the Annex G extension of the H.264 / MPEG-4 AVC video compression standard. Another example of Scalable Video Coding may be, for example, as defined in the Annex H extension of the H.265 / MPEG-H HEVC video compression standard. For example, the base layer may be encoded to provide a lower resolution (e.g., Full HD resolution or similar), while the enhancement layer may be encoded to provide a higher resolution (e.g., 4K resolution or similar). Of course, there may be one or more additional enhancement layers in the layered type of coding. The second video frame can be encoded as part of any of these enhancement layers. For example, using layered coding such as SVC allows the first additional video frame and the second video frame to be inserted into different layers, which can help maintain the desired frames per second (FPS) rate and thus prevent a first additional video frame (non-displayable) that would otherwise need to be inserted, for example, between two video frames to be displayed.
[0014] In one or more embodiments of the method, the first additional video frame may be one of a prediction frame (“P-frame”) and a bidirectional prediction frame (“B-frame”). As used herein, these various types of video frames are defined as generally practiced in modern coding standards. A P-frame may use / reference data from one or more previous video frames for decoding, whereas a B-frame may use / reference data from one or more previous video frames and one or more future video frames for decoding. These video frames may also be referred to as “inter-frames” or “inter-coded frames” because they contain references to one or more other video frames, unlike “intra-frames” or “intra-coded frames” (“I-frames”), which do not contain references to one or more other video frames.
[0015] In one or more embodiments of the method, the method may include the step of continuously generating and inserting one or more additional video frames while the camera's FOV is changing (i.e., during an ongoing zoom-action or similar). This may correspond to a method further comprising the step of acquiring a third image captured by a video camera having a third FOV that is different from but at least partially overlaps with the second FOV (e.g., as an additional step "g"). The method may further comprise the step of generating a second additional video frame (e.g., as an additional step "h") and including a second motion vector that transforms the image content of the second image to an FOV closer to the third FOV than the second FOV, wherein the second motion vector is formed based on the difference between the third FOV and the second FOV. The method may further comprise the step of inserting the second additional video frame into an encoded video stream, including marking the second additional video frame as a non-display frame (e.g., an additional step "i"). The method may further include the step of encoding a third image as part of a third video frame of an encoded video stream (e.g., as an additional step “j”), wherein the third video frame refers to a second additional video frame. Alternatively, if more additional video frames are continuously inserted into the encoded video stream, the method may include repeating the step cf described above, but replacing “first and second” with “second and third”.
[0016] In other words, additional video frames can be inserted, for example, during a zoom operation. Instead of attempting to encode the actual images captured during the ongoing zoom operation (e.g., second and third images, etc.) by retrieving motion vectors from previously captured frames, the images captured by the camera after the FOV change are instead encoded by referencing the respective additional video frames (e.g., first and second additional video frames, etc.).
[0017] In one or more embodiments of the method, the difference between the first FOV and the second FOV (or, for example, the difference between the second FOV and the third FOV, etc.) may be caused by a zoom operation performed by the camera. The zoom operation may continue while the method is performed, such as when the encoded video stream represents a live stream of the captured scene. The envisioned method may also be used in the case of transcoding, where the method is applied to already recorded video material and is used, for example, to optimize (reduce) the size required to store said video material in a repository (e.g., a server). In such cases, data related to the various FOVs of the camera during conversion (e.g., camera orientation angle, focal length defining the camera's zoom level each time, camera mounting position relative to the scene, etc.) may be envisioned to be stored as metadata within the recorded video frame or, for similar example, as part of the video frame header.
[0018] In one or more embodiments of the method, the zoom operation may be a zoom-in operation, so that the second FOV becomes a scale of the first FOV (and, for example, the third FOV becomes a scale of the second FOV, etc.). Applying this method while performing a zoom-in operation may be advantageous, for example, in that a larger portion of each image can be referenced by the next image in the video stream. It should be noted that the conceived method may also be applied, for example, during a zoom-out operation. For example, the second FOV is instead a reduction / enlargement of the first FOV, but the advantage is minimal, as only a small portion of each image can be referenced by the next image in the video stream.
[0019] In one or more embodiments of the method, the first image and the second image may be consecutive images of a video stream captured, for example, by a camera, so that the addition of additional (non-display) video frames is performed for each image. The video stream is stored while the camera conversion is in progress and references, for example, each encoded (display) video frame is referenced.
[0020] In one or more embodiments of the method, the method may be performed on a camera used to capture at least the first and second images. This may allow, for example, to generate an encoded video stream at the "edge" without requiring additional intermediate video processing equipment.
[0021] Generally, here, a video frame of an encoded video stream to be displayed (e.g., not displayed as a non-display frame) refers to at least a corresponding additional (non-display) video frame, and each additional (non-display) video frame refers to at least a previous (non-display) video frame in the encoded video stream. Of course, the (display) video frame reference may also be, for example, a previous (display) video frame in the encoded video stream. When using layered coding such as SVC, each video frame in the enhancement layer may not only refer to a corresponding additional video frame in the base layer, but may also, if desired, refer to one or more previous (or future / forward) video frames in the enhancement layer.
[0022] According to a second aspect of the present disclosure, a corresponding device for encoding a video stream is provided. The device includes a processor and memory. The memory stores instructions that cause the device to perform a method according to a first aspect when the processor is executed. This includes causing the device to acquire a first image captured by a video camera having a first FOV; encoding the first image as part of a first video frame of an encoded video stream; acquiring a second image captured by a video camera having a second FOV that is different from but at least partially overlaps with the first FOV; generating a first additional video frame that references the first video frame and includes a first motion vector that converts the image content of the first image to an FOV closer to the second FOV than the first FOV, wherein the first motion vector is formed / calculated based on the difference between the second FOV and the first FOV; and inserting the first additional video frame into the encoded video stream, including displaying the first additional video frame as a non-display frame; And encoding a second image as part of a second video frame of an encoded video stream, wherein the second video frame refers to a first additional video frame.
[0023] In one or more embodiments of the device, an instruction stored in the memory of the device as disclosed and discussed herein may cause the device to perform any embodiment of the method (of the first aspect) when executed by a processor.
[0024] In one or more embodiments of the device, the device may be a camera for capturing at least a first and a second image.
[0025] According to a third aspect of the present disclosure, a corresponding computer program for encoding a video stream is provided. The computer program is configured to cause the device to perform the method of the first aspect (i.e., including instructions / computer code) when executed by a processor of a device (such as the device of the second aspect). This enables the device to obtain a first image captured by a video camera having a first FOV; to encode the first image as part of a first video frame of an encoded video stream; to obtain a second image captured by a video camera having a second FOV that is different from but at least partially overlaps with the first FOV; to generate a first additional video frame that references the first video frame and includes a first motion vector that converts the image content of the first image to an FOV closer to the second FOV than the first FOV, wherein the first motion vector is formed / calculated based on the difference between the second FOV and the first FOV; and to insert the first additional video frame into the encoded video stream, including displaying the first additional video frame as a non-display frame; And encoding a second image as part of a second video frame of an encoded video stream, wherein the second video frame refers to a first additional video frame.
[0026] In one or more embodiments of a computer program, the computer program (i.e., instructions / computer code provided together) may be configured to cause a device to perform any embodiment of the method (of the first aspect) when executed by a processor as disclosed and discussed herein.
[0027] According to a fourth aspect of the present disclosure, a corresponding computer program product is provided. The computer program product comprises a computer-readable storage medium that stores a computer program according to a third aspect (or according to any embodiment described and discussed herein). As used herein, a computer-readable storage medium includes, for example, a hard disk drive (HDD), a solid-state drive (SDD), a USB flash drive, an SD card, a CD / DVD, and / or other storage media capable of non-temporary storage of data. In other embodiments, the computer-readable storage medium may be temporary and may correspond, for example, to a signal (electrical, optical, mechanical, or similar) present in a communication link, wire, or similar means of signal transmission.
[0028] Other objects and benefits of the present disclosure will become apparent from the following detailed description, drawings, and claims. Within the scope of the present disclosure, all features and benefits described, for example, by reference below, are taken into account. The method of the first aspect relates to, applies to, and may be used in combination with the device of the second aspect, the computer program of the third aspect, and the computer program product of the fourth aspect, and vice versa. Brief explanation of the drawing
[0030] Exemplary embodiments will now be described below with reference to the accompanying drawings. Figure 1 schematically illustrates an example of a video stream in which the camera's FOV changes while capturing an image. FIG. 2 schematically illustrates an example of an improved method for encoding a video stream applied to the video stream of FIG. 1, according to an embodiment of the present disclosure. FIG. 3 schematically illustrates a flowchart of an improved method for encoding a video stream according to an embodiment of the present disclosure. FIG. 4 schematically illustrates another example of an improved method for encoding a video stream applied to the video stream of FIG. 1, according to an embodiment of the present disclosure. FIG. 5 schematically illustrates examples of a method for acquiring motion vectors based on knowledge of how the camera's FOV has changed, according to an embodiment of the improved method of the present disclosure. FIG. 6 schematically illustrates an example of a device for encoding a video stream according to an embodiment of the present disclosure. In the drawings, the same reference numerals are used for the same components unless otherwise noted. Unless otherwise explicitly stated, the drawings illustrate only the elements necessary to describe exemplary embodiments, while other elements may be omitted or merely suggested for clarity. As illustrated in the drawings, the (absolute or relative) sizes of elements and regions may be exaggerated or underestimated compared to their actual values for illustrative purposes and are thus provided to describe the general structure of the embodiments. Specific details for implementing the invention
[0031] FIG. 1 schematically illustrates one typical example of a video stream (100) captured by a (video) camera, wherein the camera's FOV changes during capture. In this particular example, the change in FOV occurs because the camera performs a zoom operation, specifically due to a zoom-in operation that also includes some panning and tilting of the camera to move an object of interest at the center of the camera's FOV.
[0032] In this particular example, the scene is assumed to contain a single object (110) of interest. Here, the object (110) is assumed to be a flower in a flowerpot, but of course, it could be any combination of one or more other objects. The video sequence (100) includes a first image (120a) captured by a camera in a first FOV. Here, the first FOV is a FOV where the object (110) occupies only the lower-left portion of the first image (120a). Over time, the camera begins to zoom in on the object (110). By adjusting the lens and / or using digital processing that emulates this physical zoom, it is panned slightly to the left and tilted slightly downward to center the object (110) in the center of the FOV.
[0033] Next, when the second image (120b) is captured by the camera, the camera has a second FOV that provides an enlargement of the object (110) compared to the first FOV. In the second image (120b), the size of the object (110) is somewhat larger than in the first image (120a), and it has started to move toward the center of the image due to the panning and tilting of the camera.
[0034] As time progresses, the camera continues to zoom in and out while capturing more images of the scene. It becomes larger and occupies an increasingly larger portion of each image. In the specific example illustrated in FIG. 1, this involves capturing a third image (120c) with a camera having a third FOV and capturing a fourth image (120d) with a camera having a fourth FOV. Now, after capturing the fourth image (120d), in which the object (110) occupies almost all of the available image space and is located at the center of the FOV, the camera stops the zoom operation. For consecutive frames such as the fifth image (120e) and the sixth image (120f), the camera's FOV does not change and remains the same as the fourth FOV used to capture the fourth image (120d).
[0035] For example, to encode a video stream (100) for storage, various encoding methods such as bitrate / bandwidth may be applied. Conventional such methods may, for example, start the first image (120a) with an intra-predicted frame (I-frame) or similar encoding without any reference to one or more additional video frames. Then, the second image (120b) is encoded into an inter-predicted frame (P-frame) to provide a reference to the I-frame encoding the first image (120a), for example, in the form of a single motion vector. However, to find this motion vector, the encoder must first perform a motion vector search, where the encoder attempts to identify which part of the first image (120a) image data needs to be identified for each part of the second image (120b). This can later be searched to decode the P-frame encoding the second image (120b). This is often achieved by defining a limited search area around each part of the second image (120b) and then searching this area in the first image (120a) to find a suitable match. Although the change in camera FOV does not involve zooming, for example, slow translational movement of the camera (e.g., panning), the required search area will remain small and the motion vector will be approximately the same for all parts of the second image (120b).
[0036] However, in the case of a zoom motion as illustrated in FIG. 1, searching for these motion vectors will be more difficult, and an expansion of the search area will be required. Additionally, the motion vectors may differ for, for example, many parts or even all parts of the second image (120b). Pixels belonging to the flowerpot are, for example, smaller than the pixels of the flowerpot. Pixels belonging to the top of the flower, for example, pixels belonging to the left part of the object (110), would have moved approximately only upward in the second image (120b) compared to the first image (120a). Instead, pixels belonging to the right part of the object (110) would have moved both upward and to the right. As the zoom progresses and different parts of the object (110) begin to move much faster apart from each other for each new captured image (and because each new P-frame must refer to, for example, a previously encoded P-frame), the motion vectors become increasingly longer and increasingly different (in direction) for different parts of the second image (120b) and the P-frame encoding it. Consequently, there is a risk that the processing power required to perform such motion vector retrieval will exceed the capabilities of the entity responsible for encoding, which may be itself in a real-time situation, for example.
[0037] How the method conceived in this specification can improve this situation will now be explained in more detail with reference to, for example, FIGS. 2 and FIGS. 4. Also, refer to FIG. 3, which schematically illustrates a flowchart (300) of the various steps of this method.
[0038] FIG. 2 schematically illustrates the encoding of various video frames performed as part of the method (200). As conceived herein, the improved method (200) includes the step of acquiring a first image (e.g., a first image (120a)) captured by a video camera having a first FOV (as step S301). The method (200) further includes the step of encoding the first image (120a) as part of a first video frame (220a) of an encoded video stream (210) (as step S302). This may be performed as traditionally done, for example, by encoding the first video frame (220a) as an I-frame or similarly so that the first video frame (220a) contains all the necessary image data required to decode the first image (120a) again at the decoder side (not shown). Encoding may result in no loss or at least some loss of image quality when decoding the first video frame (210a), for example, depending on the exact method used to encode the first video frame (220a).
[0039] The method (200) includes the step (step S303) of receiving a second image (e.g., a second image (120b)) captured by a video camera having a second FOV. For example, as previously described in FIG. 1, the first FOV is different from the second FOV, but the second FOV overlaps at least partially with the first FOV. In other words, for the method (200) to function as intended, at least some part must be present. The object (110) can be seen in both the first image (120a) and the second image (120b). More preferably, all parts of the object (110) visible in the second image (120b) must also be visible in the first image (120a).
[0040] In contrast to conventional encoding methods, the conceived improved method (200) does not proceed by attempting to directly encode the second image (120b) into, for example, a regular P-frame using motion vector search on the first image (120a). Instead, at step S304, the method (200) proceeds by referencing the first video frame (220a) and generating a first additional video frame (230a) containing a first motion vector that transforms (in this case, resizes and moves) the image content of the first frame. It transforms the first image (120a) into an FOV that is closer to the second FOV than the first FOV. This first motion vector is not found through a computationally intensive motion vector search, but is calculated / formed based on knowledge of the second FOV and the first FOV, that is, how the camera's FOV changed between capturing the first image (120a) and the second image (120b). This information may be received from, for example, the camera itself and / or other specific sensors capable of detecting, for example, camera orientation (e.g., pan / tilt / roll angle or similar and / or focal length of the camera and lens). For example, it may be known that the camera's FOV changed between capturing the first image and the second image (120a, 120b), for example by increasing the focal length of the lens by a certain amount, and by tilting the camera slightly downward while panning the camera slightly to the left (to move the object (110) toward the center of the image while zooming). This information may then be used to directly obtain the first motion vector required to perform scaling / transformation to convert the FOV of the first image (120a) to a FOV that is closer to or the same as the second FOV used to capture the second image (120b).As used herein, “the FOV of an image is transformed into a different FOV” should be understood as making the resulting image appear as if it were captured by a camera with a different FOV, as a transformation is calculated to render the shape of the resulting image when applied to the image content of the image. How the first motion vector can be formed / calculated based on knowledge of the various FOVs used to capture the two images will be explained in more detail later in this specification with reference, for example, to FIG. 5.
[0041] After the first motion vector is formed / calculated, the method (200) proceeds by inserting the first additional video frame (230a) into the encoded video stream (210) (at step S305), and the first additional video frame (230a) is a non-display frame (indicated by the lined screen symbol shown in FIG. 2). Thus, the first additional video frame (230a) will not be displayed during subsequent decoding of the encoded video stream (210) and will only serve as a container of information for decoding / encoding other video frames. In particular, in this example, the first additional video frame (230a) does not need to contain any actual image data, but only needs to contain the first motion vector that enables the FOV to be transformed (during subsequent decoding of the first additional video frame (230a)). That is, by scaling, transforming, and / or rotating the image content of the first image (120a) as needed, the first image (120a) is transformed into an FOV more similar to the second FOV.
[0042] Next, in step (S306), the method continues by encoding a second image (120b) as part of a second video frame (220b) of the encoded video stream (210), wherein the second video frame (220b) is configured to reference a first additional video frame (230a). The second video frame (220b) may be, for example, a P-frame or similar, and, for example, a motion vector indicates which part of the first additional video frame (230a) (decoded) is used to decode which part of the second video frame (220b). Importantly, the motion vector for decoding the second video frame (220b) is found because the FOV of the image (to be decoded) of the first additional video frame (230a) is more similar (or identical) to the FOV of the second image (120b). Using motion vector search is simpler than attempting to find these motion vectors directly based on the image content of the first image (120a), for example. Motion vectors for the second video frame (220b) can be found, for example, by comparing the image content of the second image (120b) with the converted image content of the first image (120a) contained in (or generated after decoding) the first additional video frame (230a). Thus, the conceived method provides an improved method for encoding a video stream captured while the camera's FOV is changing.
[0043] In FIG. 2, references from one encoded video frame to another frame are illustrated by solid arrows. For example, a solid arrow (240a) illustrates how a first additional video frame (230a) references the first video frame (220a); a solid arrow (240b) indicates how a second video frame (220b) references the first additional video frame (230a); and a solid arrow (240c) indicates how a second additional video frame (230b) (if included, as described later in the specification) references the first additional video frame (230a). As used in the specification, arrows indicate the direction in which directional information is transmitted. An arrow (240a) pointing from the first video frame (220a) to the first additional video frame (230a) indicates that the first additional video frame (230a) is a frame that references the first video frame (220a). As conceived in this specification, in some embodiments of the method, other references among various video frames may, of course, be included. Examples of such additional references are indicated by dashed arrows in FIG. 2 and may include a second video frame (220a) that also references the first video frame (220a) for a part that does not require motion vectors, such as a certain background or similar. Other such optional references may include one or more frames that are bidirectional prediction frames (B-frames), for example, including references to one or more "future" video frames.
[0044] In addition to what is described above, the method being conceived, as well as in some embodiments, may continue in a similar manner to encode one or more additional images captured while the camera's FOV is changing as part of an ongoing zoom operation. Accordingly, the method conceived herein and described by the flowchart (300) may include an optional step (S307) to check whether the camera's FOV is still changing. If affirmative, the method may return (as indicated by the dashed arrow (310)) and re-execute steps (S303-S306), acquiring a new image each time, creating a new non-displayed additional video frame that references the previous additional video frame, and encoding the new image as part of a new video frame that references the new additional video frame. If it is not certain that the camera FOV change is still in progress, the method may instead stop and use a method of returning to the encoding frame that was normally performed, for example, as illustrated by the dashed arrow (312).
[0045] As illustrated in FIG. 2, continuing the method (200) is a step of acquiring a third image (e.g., a third image (120c)) captured by a camera having a third FOV, for example, wherein the third FOV is different from the second FOV but also overlaps at least partially with the second FOV. This may be done by repeating the above step (S303), for example, but repeating for "second and third" instead of "first and second". Referring to the first additional video frame (230a), the image content of the second image (120b) is a FOV that is closer to the third FOV than the second FOV in a manner similar to the first motion vector described above. Here, the second motion vector is formed instead based on the difference between the third FOV and the second FOV. This may be done by repeating the above step (S304), for example, but repeating using "second and third" instead of "first and second", etc. The method may continue by inserting a second additional video frame (230b) into the encoded video stream (210) and displaying the second additional video frame. Designate the second additional video frame (230b) as a non-display frame (corresponding to repeating the above step (S305), using "second and third" instead of "first and second," etc.). The method may continue by encoding a third image (120c) as part of a third video frame (220c) that may reference, for example, the second additional video frame (230b). In some embodiments, the method may continue in a similar manner until the change in the camera's FOV ends. In the present example, this may include acquiring a fourth image (120d), generating a third additional video frame (230c) that references a second additional video frame (230b), and encoding the fourth image (120d) as part of the fourth image in a video stream (210) that is encoded as part of the fourth video frame (230d) that references the third additional video frame (230d).For example, when it is detected that zoom has stopped, for example, by recognizing that the camera's fourth FOV when capturing the fourth image (104d) is the same as the camera's fifth FOV when capturing the fifth image (104e), the method may, for example in some embodiments, stop inserting additional video frames and instead return to the encoding frames that are performed when the current zoom is not in progress. For example, the fifth image (104e) may be encoded as part of a fifth video frame (220e) that references the fourth additional video frame (230d), and the next image (230f) may be encoded as a video frame (220f) that references the video frame (230e). The method conceived herein is primarily applied to the encoding of video frames corresponding to the time when the camera changes its FOV, during which conventional coding using motion vectors is difficult due to the long and changing motion vectors caused by the camera's FOV change.
[0046] Although the example provided is solely of a change in FOV caused by zoom-in actions (+ additional panning / tilting actions) performed by the camera, it should be noted that the envisioned method can also be used for video encoding. It is a stream that contains one or more additional changes to the camera's FOV and / or other causes of change during recording. Examples include camera rotation or other combinations of possible changes related to pan, tilt, roll, and / or changes in focal length. As long as knowledge of the different FOVs used when recording different images is known (or can be derived in other ways), the motion vectors required to generate additional video frames can be obtained directly from this knowledge without requiring motion vector retrieval by a decoder such as a camera. Therefore, as long as the FOV of the additional video frames can be made to better match or exactly match the FOV of the image captured by the camera, motion vector retrieval can be performed more efficiently when encoding the captured image. Refer to additional video frames.
[0047] Now, another embodiment of the conceived method will be described with reference to FIG. 4. Here, it is assumed that the method (400) still needs to encode the video stream (100) captured during the zoom-in on the object (110) as illustrated in FIG. 1. However, in contrast to the method (200) described with reference to FIG. 2, the method (400) instead uses a layered type of coding such as Scalable Video Coding (SVC). In the following, only the differences between the method (200) and the method (400) will be highlighted.
[0048] The encoded video stream (410) has multiple layers including at least a base layer (412) and at least one enhancement layer (414). The method (400) begins by acquiring a first image (120a) and encoding this image as part of a first video frame (420a). In this layered coding method, a part of the first video frame (440a) of the enhancement layer (414) as well as the base layer (412) (e.g., low resolution) is typically performed.
[0049] After acquiring the second image (120b), the first additional video frame (430a) is generated as previously described in this specification and refers to the first video frame (420a) of the base layer (412). The first additional video frame (430a) is also encoded in the base layer (412) and displayed as a non-display frame.
[0050] However, the second image (120b) is not directly encoded as part of any video frame of the base layer (412), but instead is encoded only as part of the second video frame (440b) of the enhancement layer (414), where the second video frame (440b) refers to the first additional video frame (430a) of the base layer (412).
[0051] In the same manner as previously described, the method (400) may continue to generate and encode more video frames as long as the camera's FOV changes. A third image (120c) may be acquired, and the third FOV used to capture the third image (120c) may be compared with the second FOV used to capture the second image (120b), and a second motion vector may be calculated and used to generate a second additional video frame (430b). As before, the third image (120c) is not encoded directly in the base layer (412), but instead is encoded into a third video frame (440c) in the enhancement layer (414) to reference the second additional video frame. Of course, the same procedure may be repeated to obtain the fourth image (120d), the fifth image (120e), etc., and the corresponding frames may be generated and inserted into the base layer (412) and the enhancement layer (414) as described in the specification. In this example, this includes generating a third additional video frame (430c) in the base layer (412) (indicated as non-displayed and referencing the second additional video frame (430b)), and encoding the fourth image (120d) as part of the fourth video frame (440d) in the enhancement layer (414) (the fourth video frame (440d) refers to the third additional video frame (430c) of the base layer (412). When the zoom-in on the object (110) is stopped, the method (400) may stop generating and inserting the additional video frames. Instead, it returns to encoding the 5th and 6th images (120e, 120f) again in both the base layer (412) and the enhancement layer (414).For example, this may include encoding a fifth image (120e) in both the fifth video frame (420e) of the base layer (412) (having low resolution) and the fifth video frame (440e) of the enhancement layer (414) (containing details to obtain a higher resolution than the base layer (412)); and encoding a sixth image (120) in both the sixth video frame (420f) of the base layer (412) and the sixth video frame (440f) of the enhancement layer (414). As illustrated in FIG. 4, additional video frames (230a, 230b, and 230c) in the method (400) replace the second, third, and fourth normally encoded video frames of the base layer (412).
[0052] The fifth video frame (420e) of the base layer (412) is indicated, for example, by the solid arrow provided in FIG. 4 and refers to the third additional video frame (430c), etc., as already described with reference to FIG. 2. Additionally, as described in this specification, other references may be included between the various video frames. For example, as illustrated by the dotted arrow in FIG. 5, each video frame of the enhancement layer (414) may refer to, for example, one or more previous video frames of the enhancement layer (414), and each "normal" (i.e., unadded video frame) of the base layer (412) may refer to one or more previous video frames of the base layer (412). Of course, there may be other unillustrated references, for example, references to future frames or similar references.
[0053] Compared to the method (200) described with reference to FIG. 2, the method (400) has the advantage that all additional video frames (e.g., frames (430a-c)) are inserted into the base layer (412) so as not to affect the FPS rate. This is because the enhancement layer (414) still contains only one encoded video frame per image captured by the camera. Additionally, in situations where the enhancement layer (414) is not decoded (e.g., when the encoder's processing power is insufficient to do so, or when the enhancement layer (414) is removed from the encoded video stream (410) before, for example, transmitting the encoded video stream (410), there may still be some use for retrieving and decoding only the base layer (412) as "FOV transformed" image content of various captured images generated using motion vectors for scaling, transformation, and / or rotation in the network (to save bandwidth). The frames found in the additional frames can still be used to render image content.
[0054] FIG. 5 illustrates in more detail how the calculation of the necessary (first) motion vector can be obtained. Here, for example, a camera (510) having a first FOV ("FOV1") captures a first image (120a) of an object (110). At another time (for example, later), the same camera (510) has a different second FOV ("FOV2") while capturing a second image (120b) of the object (110). For the sake of illustrative purposes only, in the example provided in FIG. 5, the difference between the first FOV and the second FOV of FIG. 5 (and the resulting difference between the first image and the second image (120a, 120b)) is exaggerated compared to, for example, FIG. 1 and 2.
[0055] When capturing the first image (120a), the first FOV (FOV1) is defined by a camera (510) having a first pan angle (520a), a first tilt angle (522a), a first roll angle (524a), as well as a first focal length (526a). When capturing the second image (120b), the second FOV (FOV2) is defined by a camera (510) having a second pan angle (520b), a second tilt angle (522b), a second roll angle (524b), and a second focal length (526b). In this particular example, as discussed above, the roll angle (524a-b) is the same, while the second pan angle (520b) is slightly smaller than the first pan angle (520a), the second inclination angle (522b) is larger than the first tilt angle (522a), and the second focal length (526b) is larger than the first focal length (526a) (the camera is magnified more in the second FOV than in the first FOV and is panned / tilted slightly to the left / down). The precise definition of the various angles is considered not important as long as the definition can measure / detect how the angle has changed between one FOV and another FOV.
[0056] By having knowledge of the first and second FOVs of the camera (510) (e.g., by having knowledge of the angles (520a-b, 522a-b, 524a-b) and focal lengths (526a-b), the first motion vectors (e.g., 530a and 530b, etc.) can be calculated as shown in the overlay (540) of the first and second images (120a and 120b) shown in FIG. 5. As can be seen in the overlay (540), the first motion vectors (530a and 530b) may differ for different parts of the second image (120b) (e.g., 550a and 550b). This is because different parts of the object (110) may move differently in different directions and at different speeds due to the scaling and transformation required to get closer from the first FOV to the second FOV. Of course, for example, for each part of the second image (120b), there may be additional first motion vectors other than those shown in FIG. 5.
[0057] Here, it is also considered to provide a device for encoding a video stream, as described in more detail with reference to FIG. 6.
[0058] FIG. 6 schematically illustrates a device (600) for encoding a video stream as previously discussed in this specification, that is, a device configured to perform the method (200) described, for example, with reference to FIG. 2 and / or the method (400) described with reference to FIG. 4. The device (600) includes at least a processor (or "processing circuit") (610) and a memory (612). The “processor” or “processing circuit” may be, for example, a combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller (μC), digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) capable of executing software instructions stored in memory (612), and graphic processor unit (GPU). Memory (612) may be outside the processor (610) or inside the processor (610). As used herein, “memory” may be any combination of random-access memory (RAM) and read-only memory (ROM), or any other type of memory capable of storing instructions. Memory (612) contains (i.e., stores) instructions that cause the device (600) to perform the method described herein (i.e., method (200, 400) or any embodiment thereof) when executed by the processor (610). The device (600) may be useful for performing the method in some situations It may further include one or more additional items (614). In some exemplary embodiments, the device (600) may be a (video) camera, such as a video monitoring camera, for example, and the additional item(s) (614) may include, for example, an image sensor, and one or more lenses for focusing light in the scene of the image sensor so that the monitoring camera may capture an image of the scene as part of the process of carrying out the method conceived.The additional item(s) (614) may also include various other electronic components necessary for capturing the scene to collect information related to the focal length of the camera and lens, which can be used to calculate how to properly operate the image sensor and / or lens as desired, for example, and to generate motion vectors based on the current direction and / or the difference between the two FOVs. Performing the method on a monitoring camera may be useful in that the processing moves to the "edge"—that is, closer to where the actual scene is captured compared to where the processing and video encoding are performed elsewhere (e.g., a more centralized processing server or similar). The device (600) may be connected to a network so that the encoded video stream resulting from performing the method can be transmitted to a user, for example. To this end, the device (600) may include a network interface (616), such as a wireless network interface (defined in IEEE 802.11 or one of subsequent standards supporting Wi-Fi, for example) or a wired network interface (defined in IEEE 802.3 or one of subsequent standards supporting Ethernet, for example). The network interface (616) may also support Bluetooth or other similar wireless standards capable of transmitting encoded video, for example. Various components (610, 612, 614 and 616) (if present) may be connected via one or more communication buses (620) so that these components can communicate with each other and exchange data as needed.
[0059] The device (600) may be, for example, a monitoring camera mounted on or mountable on a building, for example, in the form of a PTZ camera, or a fisheye camera or other type of monitoring / surveillance camera capable of providing a wider view of the scene. The device (600) may be, for example, a body camera, action camera, dash cam, etc. suitable for mounting on people, animals, and / or various vehicles, etc. The device (600) may be, for example, a smartphone or tablet that a user can carry to capture the scene. In any such example of the device (600), the device (600) may include all necessary components (if any) other than those described in its embodiments conceived herein, as long as the device (600) can still perform the method (200, 400, or any method).
[0060] Although not explicitly illustrated in FIG. 6, the device (600) and the processor (610) may be composed of, for example, one or more computing blocks, each of which is responsible for performing a function according to a specific step of the method (200 and 400), for example, as illustrated in the flowchart (300) of FIG. 3. For example, one such function block may be configured to acquire the first image (120a) (i.e., to perform step S301), and another function block may be configured to encode the first image (120a) (i.e., to perform step S302) until all steps of the method (200, 400) or other embodiments thereof are performed by the device (600).
[0061] To summarize the various embodiments presented herein, the present disclosure provides an improved method for encoding a video stream in situations where the camera's FOV changes while capturing images of the video stream (e.g., during a zoom operation performed by the camera). By inserting one or more additional non-display frames that sequentially transform the image content of the image captured before the FOV change begins to match the camera's FOV at various stages of the FOV change, and based on these transformations, more efficient and less computationally intensive encoding can be performed. Various images of the bitstream while the FOV changes can be encoded into motion vectors that reference various additional video frames, for example, by performing motion vector search rather than the result of long and varied motion vectors. Thus, the conceived method, corresponding apparatus, computer program, and computer program product as described herein satisfy the obvious need for encoding such video streams, because encoding techniques based solely on motion vector search prove to be computationally difficult to handle, particularly in real-time or near-real-time situations.
[0062] Although features and elements may be described above in specific combinations, each feature or element may be used alone without other features and elements, or in various combinations with or without other features and elements. Additionally, variations of the disclosed embodiments may be understood and practiced by those skilled in the art of practicing the claimed invention from the study of the drawings, the disclosure, and the appended claims.
[0063] In the claims, the words "comprising" and "including" do not exclude other components, and the indefinite articles "a" or "an" do not exclude the plural. The mere fact that certain features are cited in different dependent claims does not indicate that a combination of these features cannot be used to an advantage. Explanation of the symbols
[0065] 100: Video stream captured by camera 110: Objects within the camera FOV 120a-f: Images 1 through 6 200: How to Encode a Video Stream 210: Encoded video stream 220a-f: Video frames of the encoded video stream 230a-c: Additional, non-display video frames 240a-c: Reference arrow 300: Flowchart of the conceived method 310, 312: Arrows indicating method flow S301-S307: Method Step 400: Video stream encoding method (layering) 410: Encoded video stream (layered) 412: Base Layer 414: Enhancement Layer 420a,ef: Video frames of the base layer 430a-c: Additional non-display video frames of the base layer 440a-f: Video frames of the enhancement layer 510: (Video) Camera 520a-b: Camera's first and second pan angles 522a-b: Camera's first and second tilt angles 524a-b: First and second roll angles of the camera 526a-b: First and second focal lengths of the camera 530a-b: First motion vector 540: Overlay of the first and second images 600: Device 610: Processor 612: Memory 614: Additional item(s) 614: Network Interface 620: Communication bus FOV1, FOV2: The camera's first and second FOVs
Claims
Claim 1 a) acquiring a first image of a scene captured by a video camera at a first time; b) encoding the first image as part of a first video frame of an encoded video stream; c) acquiring a second image of the scene captured by the video camera at a second time; d1) acquiring data indicating at least one first difference among the orientation, zoom level, and position of the video camera for the scene at the first time compared with the second time, and calculating a first motion vector based on the first difference; d2) generating a first additional video frame referencing the first video frame, wherein the first additional video frame includes the first motion vector for transforming the image content of the first image; e) inserting the first additional video frame into the encoded video stream, including the step of marking the first additional video frame as a non-display frame; and f) encoding the second image as part of a second video frame of the encoded video stream, wherein the second video frame refers to the first additional video frame. How to encode a video stream. Claim 2 A method according to claim 1, wherein the step of encoding the video stream includes using a layered type of coding such as Scalable Video Coding (SVC), and wherein the first additional video frame is inserted into the base layer of the encoded video stream and the second video frame is inserted into the enhancement layer of the encoded video stream. Claim 3 A method according to claim 1, wherein the first additional video frame is one of a prediction frame, a P-frame, and a bidirectional prediction frame, a B-frame. Claim 4 A method according to claim 1, further comprising: g) acquiring a third image of the scene captured by the video camera at a third time; h1) acquiring data indicating at least one second difference among the orientation, zoom level, and position of the video camera at a second time compared to the third time, and calculating a second motion vector based on the second difference; h2) generating a second additional video frame that references the first additional video frame, wherein the second additional video frame includes the second motion vector for converting the image content of the second image; i) inserting the second additional video frame into the encoded video stream, including the step of marking the second additional video frame as a non-display frame; and j) encoding the third image as part of the third video frame of the encoded video stream, wherein the third video frame references the second additional video frame. Claim 5 A method according to claim 1, wherein the first difference is due to a zoom operation performed by the camera. Claim 6 In paragraph 5, the above zoom-action is a zoom-in action, a method. Claim 7 A method according to claim 1, wherein the first image and the second image are consecutive images of a video stream captured by the camera. Claim 8 A device for encoding a video stream comprises a processor and a memory that stores instructions, when executed by the processor, cause the device to perform the following: a) acquiring a first image of a scene captured by a video camera at a first time; b) encoding the first image as part of a first video frame of an encoded video stream; c) acquiring a second image of the scene captured by the video camera at a second time; d1) acquiring data representing at least one first difference among the orientation, zoom level, and position of the video camera with respect to the scene at the first time compared with the second time, and calculating a first motion vector based on the first difference; d2) generating a first additional video frame that references the first video frame, wherein the first additional video frame includes the first motion vector for transforming the image content of the first image; e) inserting the first additional video frame into the encoded video stream, including marking the first additional video frame as a non-display frame; and f) encoding the second image as part of the second video frame of the encoded video stream, wherein the second video frame refers to the first additional video frame, comprising a device for encoding a video stream. Claim 9 In claim 8, the above instructions, when executed by the processor, cause the device to perform further layered coding such as scalable video-coding (SVC), wherein the first additional video frame is inserted into the base layer of the encoded video stream and the second video frame is inserted into the enhancement layer of the encoded video stream. Claim 10 In paragraph 8, the device is a camera for capturing the first image and the second image. Claim 11 A computer-readable storage medium for storing a computer program for encoding a video stream, wherein, when the computer program is executed by a processor of the device, the device performs the following: a) acquire a first image of a scene captured by a video camera at a first time; b) encode the first image as part of a first video frame of an encoded video stream; c) acquire a second image of the scene captured by the video camera at a second time; d1) acquire data indicating at least one first difference among the orientation, zoom-level, and position of the video camera with respect to the scene at the first time compared with the second time, and calculate a first motion vector based on the first difference; d2) generate a first additional video frame referencing the first video frame, wherein the first additional video frame includes the first motion vector for transforming the image content of the first image; e) insert the first additional video frame into the encoded video stream, including marking the first additional video frame as a non-display frame; and f) a computer-readable storage medium that encodes the second image as part of the second video frame of the encoded video stream, wherein the second video frame refers to the first additional video frame. Claim 12 A computer-readable storage medium according to claim 11, wherein the device further performs hierarchical coding such as scalable video-coding (SVC), wherein the first additional video frame is inserted into the base layer of the encoded video stream and the second video frame is inserted into the enhancement layer of the encoded video stream. Claim 13 delete Claim 14 delete
Citation Information
Patent Citations
Method and image processing unit for forming a video stream
EP3396952B1
