Encoding of video stream including overlay

Abstract

To provide a method of encoding a video stream including an overlay, a device, a computer program, and a computer program product.SOLUTION: A method includes: capturing a first image; adding an overlay to the first image at a first position, and encoding the first image in a first frame of a video stream; capturing a second image of a scene; determining a desired position of the overlay in the second image; encoding the second image in a second frame marked as a non-display frame; and generating and encoding a third frame including temporally predicted macroblocks at the desired position of the overlay referencing the first frame with motion vectors based on a difference between the desired position and the first position, and skip-macroblocks outside of the desired position of the overlay referencing the first frame.SELECTED DRAWING: Figure 2A

Description

Technical Field

[0001] This disclosure relates to the field of video encoding. More particularly, this disclosure relates to the encoding of video streams that include one or more overlays (such as overlays used in augmented reality applications).

Background Art

[0002] Advanced technologies such as augmented reality (AR) attempt to enhance the user's experience when viewing the world through a camera. This can include overlaying, for example, text or other graphics on the captured images of the video stream to provide the user with more information about exactly what the user is currently viewing. As one of many examples made available by such technologies, in a video stream depicting, for example, a street and various buildings, an overlay indicating, for example, the address of a particular building and / or, for example, the owner of that building, etc., can be added. This can help the user when, for example, trying to navigate through the street as the user can learn about various buildings by using the camera to photograph the street.

[0003] Overlays are generated and rendered on the captured images of the video stream. An overlay can be associated with, for example, a particular building, person, tourist attraction, or other object shown in the scene currently being captured by the video stream. In a real-world scenario, when the camera used to capture the scene is moved and / or, for example, the field of view (FOV) of the camera changes, the position of the overlay is updated so that the overlay is re-rendered to maintain a fixed state with respect to the scene itself when the next captured image of the video stream is processed, if the object is stationary. Another reason to move an overlay can be, for example, that the associated object itself is non-stationary (moving) within the scene.

[0004] However, if the camera is occupied simultaneously with other tasks (such as object detection and tracking, encoding the video stream, or other processing required to generate the output video stream), it may not always be possible to utilize the processing power (in terms of CPU, GPU, and / or memory) required to generate and re-render the overlay each time the camera changes and / or the object moves, and therefore, if the next image in the video stream is to be displayed / encoded, it may not be possible to complete the regeneration and re-rendering of the overlay in time. If the regeneration and re-rendering of the overlay is not completed in time, it can cause synchronization problems in the video stream, leading to stuttering, or, for example, the overlay may disappear and reappear between one frame and the next. Furthermore, since overlays often contain sharp details, such as the presence of text or other shapes, if such regeneration and re-rendering of overlays is too frequent, it can increase the overhead for the circuitry and software responsible for encoding the video stream and, for example, controlling the bitrate of the generated output video stream. If an overlay is a privacy mask, the risk of that privacy mask disappearing in one or more frames is absolutely unacceptable. Therefore, when computing budgets are limited, the (re)generation and (re)rendering of overlays in video streams need to be handled more effectively. [Overview of the Initiative]

[0005] To overcome, at least partially, the problems mentioned above, and to satisfy, at least partially, the needs mentioned above, this disclosure provides an improved method for encoding a video stream including an overlay, as defined by the attached independent claims, as well as improved devices, computer programs and computer program products. Various embodiments of the improved method, devices, computer programs and computer program products are defined in the attached dependent claims.

[0006] According to a first aspect, a method is provided for encoding a video stream including an overlay. The method includes a) capturing a first image of the scene. The method includes b) adding an overlay to a first position in the first image and encoding the first image as part of a first frame of an encoded video stream. The method includes c) capturing a second image of the scene. The method includes d) determining a desired position of the overlay in the second image, wherein the desired position of the overlay in the second image is different from the first position of the overlay in the first image. The method further includes e) encoding the second image as part of a second frame of the video stream, which includes marking the second frame as a hidden frame. The method further comprises generating and encoding (e.g., using software, without receiving a corresponding third image of the scene) a third frame of a video stream containing one or more macroblocks, f) one or more macroblocks at a desired position of the overlay (in the second image) being a temporally predicted macroblock that references a first frame, having one or more motion vectors based on the difference between a first position of the overlay in the first image and a desired position of the overlay in the second image, and one or more macroblocks in the third frame outside the desired position of the overlay (in the second image) being skip macroblocks that reference a second frame of the video stream.

[0007] As already discussed herein, an overlay may include, for example, text or other figures, which are rendered on top of the image so that the overlay can provide additional information about the objects depicted in the captured scene, for example, in a video stream, within the resulting image encoding frame.

[0008] As used herein, a “temporarily predicted macroblock” is a macroblock that forms part of a temporally predicted frame, where the information necessary to render the pixels in that macroblock is found based on the same pixels (or other pixels) in other frames of the video stream (i.e., blocks of pixels in an image, such as 8x8 pixels, 16x16 pixels, etc.). Such frames may also be called interframes, predicted frames, differential frames, or P-frames or B-frames, depending on whether the frame contains references only to already encoded frames or also to future frames. In other words, the meaning of the term “temporarily predicted macroblock” does not deviate from the meaning of established video encoding standards, and the frame containing such a macroblock serves to indicate that temporally redundant information is being used during encoding (i.e., the information necessary to render the frame does not need to be entirely contained in the encoded frame itself, but can instead be found in information found in one or more preceding frames, or even future frames, or at least approximated from this information). When used herein, “skip-macroblock” (or SKIP macroblock, skipped macroblock, etc.) also have their standardized meanings and should be understood as any macroblock that contains a reference to a macroblock found at the same location in a preceding (or future) frame, so as not to be included as part of the currently encoded frame itself, allowing image data to be copied directly from a preceding (or future) frame (without requiring any motion vectors).Finally, a “hidden frame” should be understood as a frame that is flagged in some way to instruct the decoder to make the frame available so that it will not be rendered as part of the decoded video stream, but so that information can still be obtained from the frame and used to decode one or more other frames (i.e., not marked as hidden frames) that will be displayed. The first frame may be an intra-predictive frame containing all the data necessary to decode and render the frame without relying on any references to one or more preceding or future frames. Such a frame may also be called an intra-frame (I-frame) or full frame, etc. The first, second, and third frames may form, for example, a so-called picture group (GOP), and the intra-frame (first frame) may be followed by frames containing references to one or more temporally predicted frames, or one or more other frames (such as frames containing one or more skip-macroblocks). As long as there is some equivalent to the first frame that allows the third frame to obtain image data for the area containing the overlay, other sequences are also possible, as long as there is a second frame that is not displayed but is used solely as the source of image data for the third frame for the area not containing the overlay.

[0009] It is assumed that the disclosed method can be implemented using any video coding standard that supports the above concepts of temporally predictable macroblocks, skip-macroblocks, hidden frames, and motion vectors. Examples of such standards include (but are not limited to) High Efficiency Video Coding (HEVC) H.265, Advanced Video Coding (AVC) H.264, VP8, VP9, ​​AV1, and Versatile Video Coding (VVC) H.266.

[0010] This disclosure improves upon currently available techniques by enabling the rendering of overlays with virtually no overhead cost, reducing, and even eliminating, the risk of being unable to complete rendering a frame before the next frame begins. This improvement is achieved by not performing a full re-render of the frame when the position of the overlay in the image changes, but instead reusing image data associated with the overlay from the preceding frame and providing image data associated with the portion of the scene that does not contain the overlay from the non-overlay frames generated for this purpose. Because this disclosure utilizes only frames of types that are already available and standardized, according to this disclosure, the encoder can encode as usual, and a third frame can be constructed and inserted using software. Similarly, all standards-compliant decoders can operate as usual without requiring any special considerations to successfully render the video of the scene and overlay to the user. Motion vectors can be set using the encoder, and hardware acceleration can be leveraged for this setting to further reduce the required computation time.

[0011] In some embodiments of the method, the difference between a first position of the overlay in the first image and a desired position of the overlay in the second image may be at least partially due to a change in the field of view (camera's FOV) between the capture of the first image and the capture of the second image. The camera used to capture the scene may be, for example, a so-called pan-tilt-zoom (PTZ) camera, and the FOV may be changed, for example, by zooming in or out of the camera, and even by tilting or moving (panning).

[0012] In some embodiments of the method, the difference between a first position of the overlay in the first image and a desired position of the overlay in the second image may be at least partially due to a change in the position of an object to which the overlay is associated in the scene. The overlay may be associated, for example, with a person in the scene, and if that person is moving in the scene, the overlay can be re-rendered in an effective manner using the method described herein.

[0013] In some embodiments of the method, the third frame may be a prediction frame (a P-frame containing only a reference to itself and / or a reference to one or more preceding frames in a sequence of multiple frames), or a bidirectional prediction frame (a B-frame containing, for example, a reference to itself and / or a reference to one or more preceding frames, but optionally also containing a reference to one or more future frames in a sequence). The third frame may be inserted after the second frame in the encoded video stream so that the third frame looks back at the second frame.

[0014] In some embodiments of the method, the third frame may be a B-frame and can be inserted before the second frame in the encoded video stream so that the third frame forward-references the second frame.

[0015] In some embodiments of the method, the first and second images can be captured using the same camera. In other embodiments, it may be assumed that the first image is captured using one camera and the second image is captured using a separate second camera (for example, in a camera arrangement configured to provide a panoramic image of the scene).

[0016] In some embodiments of the method, the overlay may be an augmented reality (AR) overlay.

[0017] In some embodiments of the method, the method may be carried out in a camera used to capture a first image and / or a second image. The camera may be, for example, a monitoring camera (surveillance camera, etc.). In other embodiments, the method may be carried out in a camera system including such a camera, but it does not necessarily have to be in the camera itself; instead, it may be carried out in, for example, a video processing server, etc.

[0018] In some embodiments of the method, the overlay can be fixed to the scene. For example, the overlay may be associated with an object projected onto a stationary scene, such as a building, a street, or other type of immovable, static structure.

[0019] In some embodiments of the method, the method may include predicting the computation time required to render / add and encode the overlay as part of a second image and a second frame. If it is determined that this required computation time is shorter than a threshold, the method may omit this step instead of generating a third frame, and instead add the overlay at a desired position on the second image and encode the second image as part of the second frame of the video stream. The method may omit marking the second frame as a hidden frame so that the second frame can be rendered after the first frame. This embodiment is advantageous in that, if sufficient computing power is available (i.e., the above prediction is a threshold based on available computing resources), the overlay can be directly re-rendered instead. This can provide, for example, improved quality because, when rendering the overlay, changes in viewpoint of the scene between, for example, the capture of the first image and the capture of the second image will also be taken into account.

[0020] A second aspect of this disclosure provides a device for encoding a video stream including an overlay. The device includes a processor and memory. The memory stores instructions, and when an instruction is executed by the processor, the device captures a first image of a scene, adds an overlay to a first position in the first image and encodes the first image as part of a first frame of the encoded video stream, captures a second image of a scene, determines a desired position of the overlay in the second image, the desired position of the overlay in the second image is different from a first position of the overlay in the first image, encodes the second image as part of a second frame of the video stream, the encoding of the second image includes marking the second frame as a hidden frame, and a third frame of the video stream The process involves generating and encoding frames (for example, using software), in which one or more macroblocks of the third frame at a desired position in the overlay are temporally predicted macroblocks having motion vectors based on the difference between a first position in the overlay (in the first image) and a desired position in the overlay (in the second image), referencing the first frame, and one or more macroblocks of the third frame outside the desired position in the overlay (in the second image) are skip macroblocks referencing the second frame of the (encoded) video stream.

[0021] Therefore, the device according to the second embodiment is configured to perform the corresponding steps of the method according to the first embodiment.

[0022] In some embodiments of the device, the device is further configured to carry out any embodiment of the embodiments of the methods described herein (i.e., the instructions are such that when those instructions are executed by the processor, the device will carry out any embodiment of the embodiments of the methods described herein).

[0023] In some embodiments of the device, the device is a monitoring camera. This monitoring camera can be configured to capture at least one of a first image and a second image.

[0024] A third aspect of this disclosure provides a computer program for encoding a video stream including an overlay. The computer program, when executed by the processor of a device (such as a device according to the second aspect), causes the device to capture a first image of a scene, add an overlay to a first position in the first image and encode the first image as part of a first frame of the encoded video stream, capture a second image of a scene, determine a desired position of the overlay in the second image, where the desired position of the overlay in the second image is different from a first position of the overlay in the first image, encode the second image as part of a second frame of the video stream, the encoding of the second image includes marking the second frame as a hidden frame, and video The system is configured to generate and encode (e.g., using software) a third frame of the video stream, wherein one or more macroblocks of the third frame at a desired position in the overlay are temporally predicted macroblocks having motion vectors based on the difference between a first position in the overlay (in the first image) and a desired position in the overlay (in the second image), referencing the first frame, and one or more macroblocks of the third frame outside the desired position in the overlay (in the second image) are skip macroblocks referencing the second frame of the (encoded) video stream.

[0025] Therefore, the computer program is configured such that the device carries out the method already described herein, according to the first embodiment.

[0026] In some embodiments, the computer program is further configured to cause the device to perform any of the embodiments of the methods described herein when executed by a processor of the device.

[0027] According to a fourth aspect, a computer program product is provided, the computer program product comprising a computer-readable storage medium storing a computer program according to the third aspect or any of its embodiments. The computer-readable storage medium may be, for example, non-transitory and may be provided as, for example, a hard disk drive (HDD), a solid state drive (SSD), a USB flash drive, an SD card, a CD / DVD, and / or any other storage medium capable of storing data non-transitorily.

[0028] Other objects and advantages of the present disclosure will become apparent from the following detailed description, the drawings and the claims. Within the scope of the present disclosure, all features and advantages described, for example, with reference to the method of the first aspect are also related to any features and advantages described with reference to the device of the second aspect, the computer program of the third aspect and / or the computer program product of the fourth aspect, are applicable to these features and advantages and can be used in combination with these features and advantages, and vice versa, all features and advantages described with reference to the device of the second aspect, the computer program of the third aspect and / or the computer program product of the fourth aspect are related to all features and advantages described with reference to the method of the first aspect, are applicable to these features and advantages and can be used in combination with these features and advantages.

[0029] Exemplary embodiments will be described below with reference to the accompanying drawings.

Brief Description of the Drawings

[0030] [Figure 1A]This figure schematically illustrates an example of how a video stream including an overlay is encoded using the method of the present disclosure. [Figure 1B] This figure schematically illustrates an example of how a video stream including an overlay is encoded using the method of the present disclosure. [Figure 1C] This figure schematically illustrates an example of how a video stream including an overlay is encoded using the method of the present disclosure. [Figure 1D] This figure schematically illustrates an example of how a video stream including an overlay is encoded using the method of the present disclosure. [Figure 2A] This is a flowchart of various embodiments of the method described herein. [Figure 2B] This is a flowchart of various embodiments of the method described herein. [Figure 3A] This figure schematically illustrates an additional example of how a video stream including an overlay is encoded using the method of the present disclosure. [Figure 3B] This figure schematically illustrates an additional example of how a video stream including an overlay is encoded using the method of the present disclosure. [Figure 3C] This figure schematically illustrates an additional example of how a video stream including an overlay is encoded using the method of the present disclosure. [Figure 4] This figure schematically illustrates an embodiment of the device according to the present disclosure. [Modes for carrying out the invention]

[0031] In the drawings, similar reference numerals are used for similar elements unless otherwise specified. Unless otherwise explicitly stated, the drawings show only such elements necessary to illustrate exemplary embodiments, while other elements may be omitted or merely suggested for clarity. As shown in the drawings, the (absolute or relative) sizes of elements and areas may be exaggerated or understated relative to their true values ​​for illustrative purposes and are therefore provided to illustrate the general structure of the embodiments.

[0032] Various examples of how a video stream including an overlay is encoded using the method of this disclosure are described below with reference to Figures 1A to 1D. See also Figure 2A, which schematically shows a flowchart of steps S201 to S206 of such a method 200.

[0033] Figures 1A and 1B schematically illustrate situations in which the currently disclosed method can be used. Figure 1A shows a first captured image 110. The first image 110 depicts a scene including a street scene, with several buildings / houses along the street, as well as other objects often found in such environments, such as trees, trash cans, lampposts, etc. Using augmented reality, additional information is provided to the user viewing the first image (as part of a video stream), including an overlay 120 that is added to the current additional information about a specific building 124 visible in the scene. Here, this additional information includes the address of building 124 ("123 Street"). Naturally, it is also possible to provide one or more additional overlays showing, for example, the addresses of other buildings, street names, compass directions, arrows indicating the direction the user should move to arrive at a desired location, and many other types of overlays. It is assumed that the first image 110 was captured using a camera having a specific field of view (FOV), including specific camera alignment (e.g., with respect to sway, pitch, and roll) and specific zoom level, etc. The camera may be, for example, a body camera worn by the user, a camera on a smartphone or tablet, an in-vehicle camera installed in a vehicle currently being driven by the user, or a camera installed in a building (such as a monitoring or surveillance camera) (such as a pan-tilt-zoom (PTZ) camera). Other alternatives are, of course, possible and are all intended to be used in the disclosed manner.

[0034] Figure 1B shows a second captured image 112 of the same scene, but the position of the objects in the scene is different in the second image 112 compared to the first image 110. This could be due to camera movement, tilting, panning, zooming, etc. For example, it can be assumed that between the capture of the first image 110 and the capture of the second image 112, the camera moved backward and to the left along the street, and / or the camera zoomed out on the scene. In any case, in the second image 112, a particular building 124 is shown in a different position, meaning that the desired position of the overlay 120 in the second image 112 is different from the position of the overlay in the first image 110 (shown here as overlay 120'). Therefore, it is necessary to re-render the overlay 120 in the second image 112 so that the overlay 120 appears fixed relative to the scene, that is, to follow the position of a specific building 124, which is to provide additional information about it. The desired movement of the overlay is shown by arrow 130 in Figure 1B.

[0035] As already discussed, the desired movement of the overlay may be such that the camera (or any other device responsible for encoding the video stream in which the first image 110 and the second image 112 are to form part) is occupied by one or more other tasks that occupy computational resources, and the remaining resources available for re-rendering the overlay 120 in the second image 112 may not be sufficient to complete such re-rendering in time (i.e., it must first move to a subsequent third image captured after the second image 112 before processing). This leads to synchronization problems, stuttering, and otherwise an inability to smoothly transition to the second image in the video stream, and / or the overlay may disappear from the second image by not being fully rendered in time, for example.

[0036] As will be explained next with reference to Figures 1C and 1D (and 2A), the method envisioned herein provides a solution to the problem of overcoming such issues.

[0037] Figure 1C schematically illustrates the generation of sequences of frames 150, 152, and 154 of the encoded video stream 140 using the assumed method 200.

[0038] As illustrated with reference to Figure 1A, Method 200 first begins with the step (in step S201) of capturing a first image 110 of the scene, then proceeds to the step (in step S202) of adding an overlay 120 to the first image 110 and encoding the first image 110 (and the overlay 120) as part of a first frame 150 of the encoded video stream 140. In the first frame 150, the overlay 120 is located at a first position within the first image 110, measured, for example, using image coordinates. The fact that the first image 110 is encoded as "part of" the first frame 150 naturally means that, in addition to the image information originating from the first image 110, other content provided in the first frame 150, such as additional shapes or text objects, may also be present.

[0039] Method 200 then proceeds to the step of capturing a second image 112 of the same scene (in step S203), but between the step of capturing the first image 110 and the step of capturing the second image 112, the camera has changed such that the desired position of the overlay 120 in the second image 112 is now different from the position of the overlay 120 in the first image 110. In step S204, Method 200 includes the step of determining the desired position of the overlay 120 in the second image 112, which, as described above, is different from the desired position of the overlay 120 in the first image 110. The change in position is indicated by arrow 130 in Figure 1B. The desired position in the second image can be found, for example, by knowing the movement of the camera, the change in camera zoom, etc.

[0040] Instead of attempting to re-render the overlay on the second image 112 and then encode the second image 112 and the overlay 120 together as, for example, the second frame of the encoded video stream 140, Method 200 proceeds to the step (in step S205) of encoding the second image 112 without the overlay 120 as part of the second frame 152 of the video stream 140 (as explained above, "as part of" means that there may be other content contained in the second frame 152, not just the second image 112). Furthermore, Method 200 marks the second frame 152 as a so-called hidden frame, meaning that the second frame 152 will not be rendered so that it is visible to the decoder receiving the encoded video stream 140. However, the second frame 152 can still be used to contain information that may then be used by one or more other frames in the video stream 140.

[0041] After encoding the second image 112 as part of the second frame 152, method 200 then proceeds to the step (in step S206) of generating a third frame 154 of the video stream 140. The third frame 154 does not contain any captured image, but instead contains references 170 and 172 to one or more other frames in the video stream 140, as described below.

[0042] The third frame 154 includes one or more macroblocks 160, which are temporally predicted macroblocks, at a desired position in the overlay 120 within the second image 112. This means that these macroblocks do not contain any actual image data within the third frame 154, but instead contain references 170 to macroblocks 161 in the first image 110 and first frame 150 of the video stream 140. The macroblocks 161 in the first frame 150 that the macroblocks 160 in the third frame 154 refer to are located at the position of the overlay 120 within the first image 110. To know where in the first image 110 and first frame 150 the decoder is supposed to look for these macroblocks 161, the third frame 154 also includes one or more motion vectors 132 (e.g., encoded). The motion vector 132 is based on the difference between the position of the overlay 120 in the first image 110 and the desired position of the overlay 120 in the second image 112 obtained in the preceding step S205. The motion vector 132 is in the opposite direction to the arrow 130 shown, for example in Figure 1B, and has the length of arrow 130, so that the decoder can know that the image data for macroblock 160 will be found in macroblock 161 at the position of the overlay 120 in the first image 110.

[0043] The third frame 154 also includes one or more macroblocks 162 that are not present at the desired location of the overlay 120 in the second image 112. These macroblocks 162 are so-called skip macroblocks and include one or more references 172 to macroblocks 163 present at the same location in the second image 112 and the second frame 152, so that the decoder knows that it must copy image information directly from the macroblocks 163 of the second image 112 and the second frame 152 and use this information to generate the corresponding portion of the third frame 154. Thus, the third frame 152 has enough information to generate an image (while decoding the video stream 140) that shows both the overlay 120 (the data for the overlay is obtained from the first image 110 and the first frame 150) and the area that does not contain the overlay 120 (the data is obtained from the second image 112 and the second frame 152 instead of the first image 110 and the first frame 150). When the third frame 154 is displayed after decoding, the user can see the updated scene (necessary for camera movement) including the overlay 120 in the desired, correct position.

[0044] In the example illustrated with reference to Figure 1C, the first frame 150 is, for example, an intra-frame that does not reference one or more other frames in the encoded video stream 140 at all. The first frame 150 is, for example, a so-called I-frame and is self-contained with respect to the image data necessary to view the first image 110 after decoding the first frame 150. Also in the example illustrated with reference to Figure 1C, the second (hidden) frame 152 is added after the first frame 150 but before the third frame 154. Thus the third frame 154 has a reference 172 that is temporally retrospective to the second frame 152 and is, for example, a so-called P-frame. The first frame 150 can, for example, act as the first frame in a GOP sequence of frames. Other frames can exist following the third frame 154 and / or prior to the first frame 150.

[0045] Figure 1D schematically illustrates another possible example of how method 200 may operate to encode the video stream 140 and the overlay 120. The various steps S201-S206 are here the same as the various steps described with reference to Figure 1C, except that a second frame 152 generated by the software is added after the third frame 154 instead. The third frame 154 is here a so-called B-frame, which now contains a reference to the “future frame” (second frame 152) as the second frame 152 containing a macroblock (since the macroblock 162 of the third frame 154 is a skip-macroblock that references the macroblock 163 of the second frame 152). As explained above, the third frame 154, in conjunction with the motion vector 132, also includes a reference 170 to the preceding first frame 150 in order to properly copy and process the macroblock 161 of the first frame 150, and the overlay 120 is found in the first image 110.

[0046] Next, another scenario in which method 200 as envisioned herein may operate will be described with reference to Figures 3A-3C. The following procedure is the same as the procedure already described with reference to Figures 1A-1D, but with a different scene type and camera setup. Figure 3A schematically shows a scene captured by a still camera, where the first image 310 depicts a street sidewalk. In this scene, the specific object to which the overlay 320 is provided is a non-stationary, moving object in the form of a dog 324. Here, the overlay 320 identifies the animal type as a "dog," but it can, of course, also provide other information that may be useful to a user viewing the scene through a video stream. As the dog 324 moves / walks along the sidewalk, the position of the dog 324 changes between the captured images, and in order to maintain the overlay fixed relative to the object / dog 324, the overlay must be updated in accordance with the change in the dog 324's position.

[0047] Figure 3B shows the captured second image 312, depicting the same scene, but the position of the dog 324 has changed since the first image 310 was captured, as indicated by the arrow 330. Therefore, the desired position of the overlay 320 in the second image 312 is different from the position of the overlay (indicated here by overlay 320') in the first image 310.

[0048] As already explained, Method 200 can proceed to the steps of first capturing a first image 310 (in step S201), rendering / adding an overlay 320 (in step S202), and encoding the result as part of the first frame 350 of the encoded video stream 340. Method 200 can then proceed to the step of capturing a second image 312 of the scene (in step S203), and Method 200 can then determine the difference between a desired position of the overlay 320 in the second image 312 and the position of the overlay 320 in the first image 310 (in step S204). Method 200 can encode this second image 312 as part of a second frame 352 (marked as a hidden frame) that does not contain the overlay 320 (in step S205), then continue to encode (e.g., using software) the generation of a third frame 354 having a temporally predicted macroblock 360 that references a macroblock 361 of the first frame 350 at the position of the overlay 320 in the first image 310, with the help of one or more motion vectors 332 based on the difference between a desired position of the overlay 320 in the second image 312 and a desired position of the overlay 320 in the first image 310, and one or more skip-macroblocks 362 that references one or more macroblocks 363 of the second hidden frame 352 just described above, with reference to Figures 1C and 1D (in step S206). Therefore, the method can be implemented as needed when the desired position of the overlay in the image changes due to changes / movements of the camera (e.g., changes in the camera's field of view), when the desired position of the overlay in the image changes due to the movement of an object in the scene (to which the overlay is associated), and / or when the desired position changes due to a combination of both camera changes and movement of an object in the scene.In other words, the methods disclosed and envisioned herein can function both when the overlay is fixed to the scene and when the overlay is fixed to an object moving within the scene, for example.

[0049] In this specification, it is assumed that Method 200 can be carried out, for example, by a monitoring camera used to capture images of a scene, or in any other suitable component of a (monitoring) camera system including such a monitoring camera. Method 200 can, of course, also be carried out in any other device having the ability to receive at least the captured first and second images, generate various frames, and encode a video stream as output. Examples of such devices assumed in this specification will be described in more detail below with reference to Figure 4.

[0050] Figure 4 schematically illustrates a device 400 for encoding a video stream including an overlay. Device 400 includes at least a processor (or “processing circuitry”) 410 and a memory 412. As used herein, “processor” or “processing circuitry” may be one or any combination of suitable central processing units (CPUs), multiplexers, microcontrollers (μCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), rewritable gate arrays (FPGAs), graphics processing units (GPUs), etc., which are capable of executing software instructions stored in memory 412. Memory 412 may be external to or internal to the processor 410. As used herein, “memory” may be random access memory (RAM) and read-only memory (ROM), or any combination of any other type of memory capable of storing instructions. Memory 412 contains (i.e., stores) instructions that, when executed by the processor 410, will cause device 400 to implement the methods described herein (i.e., method 200 or any embodiment thereof). Device 400 may further include one or more additional items 414 that may be necessary to carry out the method in some situations. In some exemplary embodiments, device 400 may be, for example, a monitoring camera as mentioned above, in which case additional items 414 may include, for example, an image sensor and, for example, one or more lenses for focusing light from the scene onto the image sensor, so that the monitoring camera can capture an image of the scene as part of carrying out the assumed method. Additional items 414 may also include, for example, various other electronic components necessary to properly operate, for example, the image sensor and / or lenses, as needed to capture the scene.The implementation of this method in a monitoring camera may be useful in that processing is moved to the "edge," that is, closer to where the actual scene is captured compared to when processing and video encoding are performed somewhere else (such as a more centralized processing server). Device 400 can be connected to a network, for example, so that it can transmit the encoded video stream obtained by implementing this method to a user. For this purpose, device 400 may include a network interface 416, which may be a wireless network interface (such as one defined in IEEE 802.11 or any subsequent standard that supports Wi-Fi), or a wired network interface (such as one defined in IEEE 802.3 or any subsequent standard that supports Ethernet). The network interface 416 may also support any other wireless standard, such as Bluetooth, which can transmit the encoded video. These various components 410, 412, 414, and 416 (if present) can be connected via one or more communication buses 428 so that they can communicate with each other and exchange data as needed.

[0051] Device 400 may be, for example, a monitoring camera in the form of a PTZ camera, mounted on or capable of being mounted on a building, or a fisheye camera capable of providing a wider field of view of a scene, or any other type of monitoring / surveillance camera. Device 400 may also be, for example, a body camera, action camera, or in-vehicle camera, suitable for mounting on people, animals, and / or various vehicles. Device 400 may also be, for example, a smartphone or tablet that a user can carry around and use to capture scenes of interest in order to obtain further information from one or more overlays. Device 400 may also be, or include, a virtual headset, etc., that a user can wear to observe a scene via a video stream. In any such example of Device 400, it is assumed that Device 400 may include any necessary components (if any) other than those already described herein, as long as Device 400 can still carry out Method 200 or any embodiment thereof envisioned herein.

[0052] Although no further details are provided in this specification, it is assumed that device 400 (e.g., a camera) may also have the ability to identify one or more objects in a scene, such as people, buildings, streets, vehicles, animals, tourist attractions, etc., and to generate appropriate overlays for such objects to provide the user with useful information. Object identification may be performed based on, for example, machine learning algorithms and / or location information obtained from, for example, a GPS receiver included (or possibly included) in device 400.

[0053] Next, another alternative embodiment of Method 200 will be described with reference to Figure 2B. Figure 2B schematically shows an alternative procedure following step S204 of Method 200. In step S207 (which does not necessarily have to be performed after step S204), it is determined whether the computation time required to instead directly add / render the overlay as part of the second image / second frame is below a threshold. In other words, step S207 includes the step of predicting whether there are sufficient computational resources available to re-render the overlay as usual (i.e., conventionally) after the camera and / or objects in the scene have moved or changed. If the predicted computation time is below the threshold (for example, the threshold is lower if there are few computational resources available, and higher if there are many computational resources currently available), the Method may proceed to step S205' (instead of proceeding to steps S205 and S206 described with reference to Figure 2A), in which the overlay is (directly) added / rendered at a desired position in the second image and then (directly) encoded as part of the second frame of the video stream. Here, the second frame is not marked as a hidden frame, and the generation of a third frame that references the first and second frames can be omitted. On the other hand, if it is determined that the available computing resources are not sufficient to perform such a direct re-rendering of the overlay in the second image in time, method 200 can proceed to steps S205 and S206 as already discussed.

[0054] By using an alternative example of Method 200 described with reference to Figure 2B, for example, the normal re-rendering of the overlay when the camera and / or objects in the scene change can be replaced with a pre-defined use of only a second hidden frame and a third frame generated (and inserted) by the software, only when necessary. This replacement has the advantage of, for example, improved image quality, as the normal re-rendering after the camera and / or objects move / change generally results in a better visual experience because it also takes into account any changes in the field of view of the scene that occur between the capture of the first image and the capture of the second image.

[0055] The thresholds described above can be continuously updated based on the current load of, for example, a processor or other processing circuitry responsible for outputting an encoded video stream, used in a device (such as a monitoring camera). For example, if the device is heavily occupied with other resource-intensive tasks (such as tracking multiple objects in a scene), the assumed method utilizing a second hidden frame and a third frame generated (and inserted) by the software can provide a faster way to update the overlay's position so that it remains fixed with acceptable visual quality for, for example, a scene or a specific object in the scene. Such checks of available computing resources and load can be performed, for example, multiple times per second, or at any desired interval. Thus, the assumed method can be used as a backup when normal re-rendering of the overlay is not possible.

[0056] As is generally assumed herein, the desired position of the overlay in the second image can be obtained / determined, for example, by knowing the position of the overlay in the first image, and also by knowing how the camera changed in terms of zoom level, pan, tilt, etc., between the capture of the first image and the capture of the second image. Based on such information from the camera, the difference between the desired position in the second image and the position of the overlay in the first image can be calculated. If the camera is not mounted on a stationary object such as a building, additional information may be required to determine how the orientation and position of the camera have changed. It is assumed that such additional information can be obtained, if necessary, from one or more suitable sensors mounted, for example, on the camera itself and / or on any non-stationary object to which the camera is mounted. In other embodiments, it is assumed that the desired position in the second image (and the corresponding difference to the first image, and the corresponding motion vector) can be found by identifying the correct location of the overlay in the second image using, for example, one or more image / video analysis algorithms, such as one or more image / video analysis algorithms for detecting and / or tracking an object.

[0057] To summarize the various embodiments presented herein, the disclosure provides an improved method for providing (and updating the position of) an overlay in an encoded video stream where the position of the overlay in the image of the scene needs to be updated. The proposed solution overcomes the problem of potentially having insufficient computational resources available to complete the re-rendering of the overlay between each such time interval. By marking a second frame as a hidden frame and inserting a third frame, generated by the software, that references this second frame, image information about the scene in the area where the overlay is not to be placed can be provided from the second frame to the third frame by a simple copy operation. Similarly, by also referencing the first frame from the third frame, image information about the overlay itself can be provided to the third frame by copying this information from the last frame in which the overlay was properly rendered, i.e., from the first frame (of course, using appropriate motion vectors provided based on how the camera and / or objects in the scene have changed / moved). Therefore, the encoder can generate a third frame (including an overlay at a new desired position) simply by referencing already available information in other frames, thus reducing the computation time required to prepare and encode the third frame without requiring any re-rendering of the overlay at its new position in the second image. Similarly, the corresponding decoder is not dependent on this method and, when instructed by the encoded video stream output from the assumed method, can proceed by copying information from other frames as usual. In other words, currently available encoders that support concepts such as motion vectors, temporally predicted frames, and hidden frames can be used without modification.The third frame can be generated manually (using software) and simply inserted into (or combined with) the output from such an encoder to produce the encoded video stream envisioned herein; therefore, the method can also be carried out using already available encoders without modifying them.

[0058] While the features and elements are described above, in some cases, in specific combinations, each individual feature or element can be used alone without other features and elements, or in various combinations with other features and elements, or in various combinations without other features and elements. Furthermore, those skilled in the art who practice the claimed invention will be able to understand and implement variations of the disclosed embodiments by considering the drawings, this disclosure, and the appended claims.

[0059] In the claims, the words “equip” and “include” do not exclude other elements, and the indefinite article “a” or “an” does not exclude plural expressions. The mere fact that certain features are described in mutually distinct dependent claims does not imply that combinations of these features cannot be used advantageously. [Explanation of Symbols]

[0060] Image 1 of scene 110, 310 Image 112, 312, second image of the scene 120, 320 overlay 120', 320' Position of the overlay in the first image 124, 324 Specific objects associated with the overlay that appear in the scene Movement of the 130, 330 overlay positions 132, 332 Motion vectors 140,340 video streams 150, 350 First (image) frame 152, 352 Second (image) frame 154, 354 Third (image) frame 160, 360 Temporally predicted macroblocks in the third frame 161, 361 Source macroblock in the first frame 162, 362 Skip in the third frame - Macroblock 163, 363 Source macroblock in the second frame 170, 370 References to source macroblocks in the first frame 172, 372 References to source macroblocks in the second frame 200 Methods (Flowcharts) S201~S206 Method Steps S205', S207 Alternative Method Steps 400 Devices 410 Processor / Processing Circuitry 412 memory 414 Network Interfaces 416 Additional Items 428 Communications Bus

Claims

1. A method (200) for encoding a video stream including an overlay, comprising: a) Capturing (S201) a first image (110) of a scene; b) adding an overlay (120) to a first position of the first image and encoding (S202) the first image as part of a first frame (150) of an encoded video stream (140); c) capturing (S203) a second image (112) of the scene; d) using an object detection and / or tracking algorithm, calculating a desired change in the position of the overlay between the desired position of the overlay in the second image and the position of the overlay in the first image based on at least one of: i) information regarding a known change in camera field of view between the capture of the first image and the capture of the second image; ii) information regarding a known change in camera position between the capture of the first image and the capture of the second image; and iii) a known change in position of an object with which the overlay is associated in the scene between the capture of the first image and the capture of the second image (S204); e) encoding (S205) the second image as part of the second frame (152) of the video stream, including marking the second frame as a non-display frame; f) generating and encoding (S206) a third frame (154) of the video stream, wherein one or more macroblocks (160) of the third frame at the desired position of the overlay are temporally predicted macroblocks that reference (170) the first frame, and one or more macroblocks (162) of the third frame outside the desired position of the overlay are skip-macroblocks that reference (172) the second frame of the video stream, and calculating motion vectors (132) of one or more of the temporally predicted macroblocks based on the desired change in position of the overlay; A method comprising:

2. 2. The method of claim 1, wherein the third frame is a predicted frame (P-frame) or a bidirectionally predicted frame (B-frame) inserted after the second frame in the encoded video stream.

3. 3. The method of claim 1, wherein the third frame is a bidirectionally predicted frame (B-frame) inserted before the second frame in the encoded video stream.

4. The method of claim 1 or 2, comprising capturing the first image and the second image using the same camera.

5. The method of claim 1 or 2, wherein the method is implemented in a camera used to capture the first image and / or the second image.

6. The method of claim 1 or 2, wherein the overlay is fixed relative to the scene.

7. and estimating a computation time required to render and encode the overlay as part of the second image and the second frame, and upon determining that the estimated computation time is less than a threshold, performing steps a) through d) and not performing steps e) and f), instead performing after step d): e') adding the overlay to the second image at the desired location (S205') and encoding the second image as part of the second frame (152) of the video stream; The method according to claim 1 or 2.

8. A device (400) for encoding a video stream including an overlay, comprising: a processor (410); a memory (412) for storing instructions; the instructions, when executed by the processor, cause the device to: Capturing (S201) a first image (110) of a scene; adding (S202) an overlay (120) to a first position of the first image and encoding the first image as part of a first frame (150) of an encoded video stream (140); capturing (S203) a second image (112) of the scene; using an object detection and / or tracking algorithm to calculate a desired change in the position of the overlay between the desired position of the overlay in the second image and the position of the overlay in the first image based on at least one of: i) information regarding a known change in camera field of view between the capture of the first image and the capture of the second image; ii) information regarding a known change in camera position between the capture of the first image and the capture of the second image; and iii) a known change in position of an object with which the overlay is associated in the scene between the capture of the first image and the capture of the second image (S204); encoding (S205) the second image as part of the second frame (152) of the video stream, including marking the second frame as a non-display frame; generating and encoding (S206) a third frame (154) of the video stream, wherein one or more macroblocks (160) of the third frame at the desired position of the overlay are temporally predicted macroblocks that reference (170) the first frame, and one or more macroblocks (162) of the third frame outside the desired position of the overlay are skip-macroblocks that reference (172) the second frame of the video stream, the generating and encoding (S206) comprising calculating motion vectors (132) of one or more of the temporally predicted macroblocks based on the desired change in position of the overlay; A device that causes the device to perform the above.

9. The device of claim 8 , wherein the device is a monitoring camera configured to capture at least one of the first image and the second image.

10. 1. A non-transitory computer-readable storage medium having stored thereon a computer program for encoding a video stream including an overlay, the computer program, when executed by a processor of a device, causing the device to: Capturing a first image (110) of a scene (S201); adding (S202) an overlay (120) to a first position of the first image and encoding the first image as part of a first frame (150) of an encoded video stream (140); capturing (S203) a second image (112) of the scene; using an object detection and / or tracking algorithm to calculate a desired change in position of the overlay between the desired position of the overlay in the second image and the position of the overlay in the first image based on at least one of: i) information regarding a known change in camera field of view between the capture of the first image and the capture of the second image; ii) information regarding a known change in camera position between the capture of the first image and the capture of the second image; and iii) a known change in position of an object with which the overlay is associated in the scene between the capture of the first image and the capture of the second image (S204); encoding (S205) the second image as part of the second frame (152) of the video stream, including marking the second frame as a non-display frame; generating and encoding (S206) a third frame (154) of the video stream, wherein one or more macroblocks (160) of the third frame at the desired position of the overlay are temporally predicted macroblocks that reference (170) the first frame, and one or more macroblocks (162) of the third frame outside the desired position of the overlay are skip-macroblocks that reference (172) the second frame of the video stream, the generating and encoding (S206) comprising calculating motion vectors (132) of one or more of the temporally predicted macroblocks based on the desired change in position of the overlay; 1. A non-transitory computer-readable storage medium configured to cause

Images