Encoding a video stream containing overlays

Abstract

To provide a method of encoding a video stream including an overlay, a device, a computer program, and a computer program product.SOLUTION: A method includes: capturing a first image; adding an overlay to the first image at a first position, and encoding the first image in a first frame of a video stream; capturing a second image of a scene; determining a desired position of the overlay in the second image; encoding the second image in a second frame marked as a non-display frame; and generating and encoding a third frame including temporally predicted macroblocks at the desired position of the overlay referencing the first frame with motion vectors based on a difference between the desired position and the first position, and skip-macroblocks outside of the desired position of the overlay referencing the first frame.SELECTED DRAWING: Figure 2A

Description

[Technical Field]

[0001] The present disclosure relates to the field of video coding, and more particularly to the coding of video streams that include one or more overlays (e.g., overlays such as those used in augmented reality applications). [Background technology]

[0002] Emerging technologies such as augmented reality (AR) attempt to enhance a user's experience when viewing the world through a camera. This can involve overlaying, for example, text or other graphics onto a captured image of a video stream to provide the user with more information about exactly what they are currently viewing. As one example of many possibilities enabled by such technology, in a video stream depicting, for example, a street and various buildings, an overlay can be added showing, for example, the address of a particular building and / or, for example, the owner of that building. This can aid a user, for example, as they navigate a street, because they can learn about the various buildings by photographing the street using a camera.

[0003] An overlay is generated and rendered on a captured image of the video stream. The overlay may, for example, be associated with a particular building, person, tourist attraction, or other object that appears in the scene currently captured by the video stream. In a real-world situation, if the camera used to capture the scene is moved and / or, for example, the camera's field of view (FOV) changes, the position of the overlay is updated when the next captured image of the video stream is processed so that, if the object is stationary, the overlay is re-rendered to remain fixed relative to the scene itself. Another reason for moving the overlay may, for example, be that the associated object is itself non-stationary (moving) within the scene.

[0004] However, if the camera is simultaneously occupied with performing other tasks (such as detecting and tracking objects, encoding the video stream, or other processing required to generate the output video stream), the processing power (in terms of CPU, GPU, and / or memory) required to generate and re-render the overlay, for example, every time the camera changes and / or the object moves, may not necessarily be available, and therefore the overlay may not be able to be completed re-generated and re-rendered in time for the next image in the video stream to be displayed / encoded. If the overlay re-generation and re-rendering are not completed in time, synchronization issues in the video stream may arise, causing stuttering, or, for example, the overlay may disappear and reappear between one frame and the next. Furthermore, because the overlay often contains sharp details, for example, due to the presence of text or other graphics, frequent (re)generation and (re)rendering of such overlays may increase the overhead for the circuitry and software responsible for encoding the video stream and, for example, controlling the bitrate of the generated output video stream. If the overlay is a privacy mask, then there is even no way to tolerate the risk of the privacy mask disappearing in one or more frames. Therefore, there is a need to better handle the (re)generation and (re)rendering of overlays in video streams when the computational power budget is limited. Summary of the Invention

[0005] To at least partially overcome the above-mentioned problems and to at least partially satisfy the above-mentioned needs, the present disclosure provides an improved method for encoding a video stream including an overlay, as well as an improved device, computer program and computer program product, as defined by the accompanying independent claims. Various embodiments of the improved method, device, computer program and computer program product are defined in the accompanying dependent claims.

[0006] According to a first aspect, there is provided a method for encoding a video stream including an overlay. The method includes: a) capturing a first image of a scene; b) adding an overlay to a first position of the first image and encoding the first image as part of a first frame of the encoded video stream; c) capturing a second image of the scene; d) determining a desired position of the overlay in the second image, the desired position of the overlay in the second image being different from the first position of the overlay in the first image; e) encoding the second image as part of the second frame of the video stream, including marking the second frame as a hidden frame. The method further includes f) generating and encoding (e.g., using software, without receiving a corresponding third image of the scene) a third frame of the video stream including one or more macroblocks, wherein one or more macroblocks at the desired position of the overlay (in the second image) are temporally predicted macroblocks that reference the first frame, have one or more motion vectors based on a difference between the first position of the overlay in the first image and the desired position of the overlay in the second image, and wherein one or more macroblocks of the third frame outside the desired position of the overlay (in the second image) are skip-macroblocks that reference the second frame of the video stream.

[0007] As previously discussed herein, an overlay may include, for example, text or other graphics that are rendered on top of an image such that the overlay can provide additional information about objects appearing in the scene captured in the resulting frame that encodes the image, e.g., in a video stream.

[0008] As used herein, a "temporally predicted macroblock" is a macroblock (i.e., a block of pixels in an image, such as an 8x8 pixel, 16x16 pixel, etc. block) that forms part of a temporally predicted frame, for which the information necessary to render the pixel is found based on the same pixel (or other pixels) in other frames of the video stream. Such frames may also be referred to as interframes, predicted frames, differential frames, or P-frames or B-frames, depending, for example, on whether the frame contains references only to already coded frames or also to future frames. In other words, the meaning of the term "temporally predicted macroblock" does not deviate from that of established standards for video coding, but serves to indicate that frames containing such macroblocks use temporally redundant information during encoding (i.e., the information necessary to render the frame need not be entirely contained in the coded frame itself, but instead can be found in, or at least approximated from, information found in one or more preceding or even future frames). As used herein, "skip-macroblock" (or SKIP macroblock, skipped macroblock, etc.) also have their standardized meaning and should be understood as any macroblock that contains a reference to a macroblock found at the same location in a previous (or future) frame, such that the image data from the previous (or future) frame can be directly copied (without the need for any motion vectors) and is not included as part of the currently encoded frame itself.Finally, a "hidden frame" should be understood as a frame that is flagged in some way to instruct a decoder not to render the frame as part of the decoded video stream, but still be available so that information can be derived from it and used to decode one or more other frames (i.e., not marked as hidden frames) that will be displayed. The first frame may, for example, be an intra-predicted frame that contains all the data necessary to decode and render the frame without relying on any reference to one or more preceding or future frames. Such a frame may also be referred to as an intra-frame (I-frame) or a full frame, etc. The first, second, and third frames may, for example, form part of a so-called Group of Pictures (GOP), where the intra-frame (first frame) is followed by one or more temporally predicted frames or frames that contain references to one or more other frames (e.g., frames containing one or more skipped macroblocks). Other sequences are of course possible, as long as there is some equivalent to the first frame from which the third frame can obtain its image data for the area that contains the overlay, or as long as there is a second frame that is not displayed but is used only as a source of image data for the third frame for the area that does not contain the overlay.

[0009] It is contemplated that the disclosed methods can be implemented using any video coding standard that supports the above concepts of temporally predicted macroblocks, skip-macroblocks, hidden frames, and motion vectors. Examples of such standards include (but are not limited to) High Efficiency video Coding (HEVC) H.265, Advanced video Coding (AVC) H.264, VP8, VP9, ​​AV1, and Versatile video Coding (VVC) H.266.

[0010] The present disclosure improves upon currently available technology by rendering overlays with little overhead cost and reducing or even eliminating the risk of failing to complete frame rendering before the next frame begins. This improvement is achieved by not performing a full re-rendering of a frame when the overlay's position in the image changes, but instead reusing image data associated with the overlay from a previous frame and providing image data associated with the non-overlay portion of the scene from a hidden frame created for this purpose. Because the present disclosure utilizes only frames of types that are already available and standardized, the present disclosure allows an encoder to encode normally and construct and insert a third frame using software. Similarly, all standard-compliant decoders can operate normally without any special considerations to successfully render the scene and overlay video to the user. The encoder can be used to set motion vectors, which can utilize hardware acceleration to further reduce the required computation time.

[0011] In some embodiments of the method, the difference between the first position of the overlay in the first image and the desired position of the overlay in the second image may be due, at least in part, to a change in field of view (FOV of the camera) between the capture of the first image and the capture of the second image. The camera used to capture the scene may be, for example, a so-called pan-tilt-zoom (PTZ) camera, and the FOV may be changed, for example, by zooming in or out of the camera, or even by tilting or moving (panning) it.

[0012] In some embodiments of the method, the difference between the first position of the overlay in the first image and the desired position of the overlay in the second image may be due, at least in part, to a change in the position of an object with which the overlay is associated in the scene. For example, the overlay may be associated with a person in the scene, and if the person moves in the scene, the overlay can be re-rendered in an effective manner using the methods described herein.

[0013] In some embodiments of the method, the third frame may be a predicted frame (a P frame that includes only a reference to itself and / or to one or more preceding frames in a sequence of frames) or a bidirectionally predicted frame (e.g., a B frame that includes a reference to itself and / or to one or more preceding frames, but optionally or instead also includes a reference to one or more future frames in the sequence). The third frame may be inserted after the second frame in the encoded video stream such that the third frame looks back to the second frame.

[0014] In some embodiments of the method, the third frame may be a B frame and may be inserted before the second frame in the encoded video stream such that the third frame forward references the second frame.

[0015] In some embodiments of the method, the first image and the second image may be captured using the same camera, while other embodiments may envisage using one camera to capture the first image and a second, separate camera to capture the second image (e.g., in a camera arrangement configured to provide a panoramic image of a scene).

[0016] In some embodiments of the method, the overlay may be an augmented reality (AR) overlay.

[0017] In some embodiments of the method, the method may be implemented in a camera used to capture the first image and / or the second image. The camera may be, for example, a monitoring camera (such as a surveillance camera). In other embodiments, the method may be implemented in a camera system including such a camera, but not necessarily in the camera itself, and could instead be implemented in, for example, a video processing server.

[0018] In some embodiments of the method, the overlay may be fixed relative to the scene, for example, the overlay may be associated with an object appearing in the scene that is not moving, such as a building, a street, or other type of immovable stationary structure.

[0019] In some embodiments of the method, the method further comprises determining the computation time required to (alternatively) render / add and encode the second image and the overlay as part of the second frame. EstimateIf it is determined that the required computation time is less than a threshold, the method may omit this step instead of generating a third frame and instead adding an overlay to the second image at the desired location and encoding the second image as part of the second frame of the video stream. The method may omit marking the second frame as a hidden frame so that the second frame can be rendered after the first frame. This embodiment may be implemented when sufficient computational power is available (i.e., the above-mentioned Estimate is a threshold based on available computational resources), the overlay can instead be directly re-rendered, which can provide, e.g., improved quality, since changes, e.g., viewpoint, of the scene between, e.g., capturing the first image and the second image, will also be taken into account when rendering the overlay.

[0020] According to a second aspect of the present disclosure, there is provided a device for encoding a video stream including an overlay, the device including a processor and a memory. The memory stores instructions that, when executed by the processor, cause the device to capture a first image of a scene, add an overlay to a first position in the first image, encode the first image as part of a first frame of the encoded video stream, capture a second image of the scene, determine a desired position of the overlay in the second image, the desired position of the overlay in the second image differs from the first position of the overlay in the first image, encode the second image as part of a second frame of the video stream, the encoding of the second image including marking the second frame as a hidden frame, and encode a third frame of the video stream. The method involves generating and encoding (e.g., using software) a third frame in which one or more macroblocks of the third frame at the desired position of the overlay are temporally predicted macroblocks that reference the first frame and have a motion vector based on the difference between the first position of the overlay (in the first image) and the desired position of the overlay (in the second image), and one or more macroblocks of the third frame outside the desired position of the overlay (in the second image) are skip-macroblocks that reference the second frame of the (encoded) video stream.

[0021] A device according to the second aspect is therefore configured to carry out the corresponding steps of the method of the first aspect.

[0022] In some embodiments of the device, the device is further configured to perform any of the method embodiments described herein (i.e., the instructions, when executed by a processor, cause the device to perform any of the method embodiments described herein).

[0023] In some embodiments of the device, the device is a monitoring camera, which can be configured to capture at least one of a first image and a second image.

[0024] According to a third aspect of the present disclosure, there is provided a computer program for encoding a video stream including an overlay. The computer program, when executed by a processor of a device (such as the device according to the second aspect), causes the device to capture a first image of a scene, add an overlay to a first position in the first image, encode the first image as part of a first frame of the encoded video stream, capture a second image of the scene, determine a desired position of the overlay in the second image, the desired position of the overlay in the second image differs from the first position of the overlay in the first image, and encode the second image as part of a second frame of the video stream, wherein encoding the second image includes marking the second frame as a hidden frame; The method is configured to generate and encode (e.g., using software) a third frame of the (encoded) video stream, wherein one or more macroblocks of the third frame at the desired position of the overlay are temporally predicted macroblocks that reference the first frame and have a motion vector based on the difference between the first position of the overlay (in the first image) and the desired position of the overlay (in the second image), and one or more macroblocks of the third frame outside the desired position of the overlay (in the second image) are skip-macroblocks that reference the second frame of the (encoded) video stream.

[0025] The computer program is therefore configured to cause a device to perform the method according to the first aspect as previously described herein.

[0026] In some embodiments, the computer program further configures (when executed by a processor of the device) the device to perform any embodiment of the methods described herein.

[0027] According to a fourth aspect, there is provided a computer program product, the computer program product comprising a computer readable storage medium storing a computer program according to the third aspect or any embodiment thereof. The computer readable storage medium may for example be non-transitory and may be provided, for example, as a hard disk drive (HDD), a solid state drive (SDD), a USB flash drive, an SD card, a CD / DVD, and / or any other storage medium capable of non-transitory storage of data.

[0028]

[0013] Other objects and advantages of the present disclosure will become apparent from the following detailed description, drawings, and claims. Within the scope of the present disclosure, it is envisaged that, for example, all features and advantages described with reference to the method of the first aspect can also be associated with, apply to, and be used in combination with any features and advantages described with reference to the device of the second aspect, the computer program of the third aspect, and / or the computer program product of the fourth aspect, and vice versa.

[0029] Exemplary embodiments will now be described with reference to the accompanying drawings. [Brief explanation of the drawings]

[0030] [Figure 1A]FIG. 1 is a diagram illustrating a schematic example of how a video stream including an overlay is encoded using a method according to the present disclosure. [Figure 1B] FIG. 1 is a diagram illustrating a schematic example of how a video stream including an overlay is encoded using a method according to the present disclosure. [Figure 1C] FIG. 1 is a diagram illustrating a schematic example of how a video stream including an overlay is encoded using a method according to the present disclosure. [Figure 1D] FIG. 1 is a diagram illustrating a schematic example of how a video stream including an overlay is encoded using a method according to the present disclosure. [Figure 2A] 1 is a flowchart of various embodiments of methods according to the present disclosure. [Figure 2B] 1 is a flowchart of various embodiments of methods according to the present disclosure. [Figure 3A] 10A and 10B are schematic diagrams illustrating further examples of how a video stream including an overlay is encoded using a method according to the present disclosure. [Figure 3B] 10A and 10B are schematic diagrams illustrating further examples of how a video stream including an overlay is encoded using a method according to the present disclosure. [Figure 3C] 10A and 10B are schematic diagrams illustrating further examples of how a video stream including an overlay is encoded using a method according to the present disclosure. [Figure 4] 1A and 1B are schematic diagrams illustrating embodiments of devices according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0031] In the drawings, like reference numerals are used for like elements unless otherwise noted. Unless specifically and explicitly stated otherwise, the drawings show only those elements necessary to illustrate the exemplary embodiments, while other elements may in some cases be omitted or merely suggested for clarity. As shown in the figures, the sizes (absolute or relative) of elements and regions may in some cases be exaggerated or understated relative to their true values ​​for illustrative purposes and are thus provided to illustrate the general structure of the embodiments.

[0032] In the following, various examples of how a video stream including an overlay may be encoded using a method according to the present disclosure will be described with reference to Figures 1A to 1D. Reference is also made to Figure 2A, which shows a schematic flow chart of steps S201 to S206 of such a method 200.

[0033] 1A and 1B provide a schematic illustration of a situation in which the presently disclosed method can be used. FIG. 1A shows a first captured image 110. The first image 110 depicts a scene, here including a street view, with several buildings / houses along the street, as well as other objects often found in such environments, such as trees, trash cans, lampposts, etc. Using augmented reality, additional information is provided to a user viewing the first image (as part of a video stream) that includes an overlay 120 that adds additional information to the current scene regarding a particular building 124 appearing in the scene. Here, this additional information includes the address of the building 124 ("123 Street"). Of course, one or more additional overlays could also be provided, e.g., showing addresses of other buildings, street names, compass directions, arrows indicating the direction the user should travel to reach a desired location, and many other types of overlays. It is assumed that the first image 110 was captured using a camera having a particular field of view (FOV), including a particular alignment of the camera (e.g., with respect to yaw, pitch, and roll) and a particular zoom level, etc. The camera may be, for example, a body camera worn by the user, a camera on a smartphone or tablet, an in-vehicle camera mounted in a vehicle the user is currently driving, or a camera (such as a pan-tilt-zoom (PTZ) camera) mounted on, for example, a building (such as a monitoring or surveillance camera). Other alternatives are of course possible, and all are contemplated as being usable in the disclosed methods.

[0034] 1B shows a second captured image 112 of the same scene, but the position of objects in the scene is different in the second image 112 than in the first image 110. This may be because the camera has moved, tilted, panned, zoomed, etc. For example, it may be assumed that the camera has moved back along the street to the left and / or the camera has zoomed out on the scene between the capture of the first image 110 and the capture of the second image 112. In either case, the particular building 124 is now in a different position in the second image 112, meaning that the desired position of the overlay 120 in the second image 112 is different from the position of the overlay in the first image 110 (shown here as overlay 120′). It is therefore necessary to re-render the overlay 120 in the second image 112 so that it appears fixed relative to the scene, i.e., follows the position of the particular building 124 about which the overlay 120 is to provide additional information. The desired movement of the overlay is indicated by arrow 130 in FIG. 1B.

[0035] As already discussed, the desired movement of the overlay may be such that the camera (or any other device responsible for encoding the video stream of which the first image 110 and the second image 112 are to form part) is also occupied with one or many other tasks that occupy computational resources, and the remaining resources available for re-rendering the overlay 120 in the second image 112 may not be sufficient to complete such re-rendering in time (i.e., before processing, it must move on to a subsequent third image, captured after the second image 112, etc.). This may result in synchronization problems, stuttering, or an otherwise insufficient transition to the second image in the video stream, and / or the overlay may disappear from the second image, for example by not being completely rendered in time.

[0036] As will now be explained with reference to Figures 1C and 1D (and Figure 2A), the methods contemplated herein provide a solution to how to overcome such problems.

[0037] FIG. 1C illustrates schematically the generation of a sequence of frames 150, 152 and 154 of an encoded video stream 140 according to a contemplated method 200.

[0038] As described with reference to FIG. 1A , method 200 begins by initially capturing a first image 110 of a scene (in step S201), and then proceeds to adding an overlay 120 to first image 110 (in step S202) and encoding first image 110 (and overlay 120) as part of a first frame 150 of encoded video stream 140. In first frame 150, overlay 120 is located at a first position within first image 110, e.g., measured using image coordinates. Of course, encoding first image 110 as “part of” first frame 150 means that in addition to the image information originating from first image 110, there may also be other content provided within first frame 150, such as additional graphics or text objects, etc.

[0039] The method 200 then proceeds to capture (at step S203) a second image 112 of the same scene, except that between capturing the first image 110 and the second image 112, the camera has changed such that the desired position of the overlay 120 in the second image 112 is now different from the position of the overlay 120 in the first image 110. In step S204, the method 200 includes determining the desired position of the overlay 120 in the second image 112, which, as explained above, is different from the desired position of the overlay 120 in the second image 112. The change in position is indicated by arrow 130 in FIG. 1B. The desired position in the second image can be found, for example, by knowing camera motion, a change in the camera zoom, etc.

[0040] Instead of attempting to re-render the overlay on top of the second image 112 and then jointly encode the second image 112 and overlay 120, for example, as a second frame of the encoded video stream 140, the method 200 proceeds to encode (at step S205) the second image 112 without the overlay 120 as part of the second frame 152 of the video stream 140 (as explained above, "as part of" means that there may be other content included in the second frame 152, not just the second image 112). Furthermore, the method 200 marks the second frame 152 as a so-called hidden frame, meaning that the second frame 152 will not be visibly rendered by a decoder receiving the encoded video stream 140. However, the second frame 152 may still be used to contain information that may then be used by one or more other frames in the video stream 140.

[0041] Having encoded the second image 112 as part of the second frame 152, the method 200 then proceeds (at step S206) to generate a third frame 154 of the video stream 140. The third frame 154 does not contain any captured images, but instead contains references 170 and 172 to one or more other frames in the video stream 140, as will now be described.

[0042] The third frame 154 includes one or more macroblocks 160 that are temporally predicted macroblocks at the desired location of the overlay 120 in the second image 112. This means that these macroblocks do not contain any actual image data in the third frame 154, but instead contain references 170 to macroblocks 161 in the first image 110 and first frame 150 of the video stream 140. The macroblocks 160 in the third frame 154 refer to the macroblocks 161 in the first frame 150 that are located at the location of the overlay 120 in the first image 110. The third frame 154 also includes (e.g., encodes) one or more motion vectors 132 so that the decoder knows where to look for these macroblocks 161 in the first image 110 and first frame 150. The motion vector 132 is based on the difference between the location of the overlay 120 in the first image 110 and the desired location of the overlay 120 in the second image 112 obtained in the previous step S205. The motion vector 132 is oriented in the opposite direction to and has the length of the arrow 130 shown in, for example, FIG. 1B so that the decoder knows that the image data for macroblock 160 will be found in macroblock 161 at the location of the overlay 120 in the first image 110.

[0043] The third frame 154 also contains one or more macroblocks 162 that are not present in the second image 112 at the desired location of the overlay 120. These macroblocks 162 are so-called skip-macroblocks and contain one or more references 172 to macroblocks 163 that are present in the second image 112 and the second frame 152 at the same locations, so that the decoder knows to copy image information directly from the macroblocks 163 in the second image 112 and the second frame 152 and use this information to generate the corresponding portions of the third frame 154. Thus, the third frame 152 contains enough information to generate (while decoding the video stream 140) an image that shows both the overlay 120 (data for the overlay is obtained from the first image 110 and the first frame 150) and areas that do not contain the overlay 120 (data is obtained from the second image 112 and the second frame 152 instead of the first image 110 and the first frame 150). When the third frame 154 is displayed after decoding, the user can see an updated view of the scene (necessary due to camera movement) including the overlay 120 in the desired, correct position.

[0044] In the example described with reference to FIG. 1C , the first frame 150 is, for example, an intra-frame that does not reference any other frames in the encoded video stream 140. The first frame 150 is, for example, a so-called I-frame, which is self-contained with respect to the image data necessary to view the first image 110 after decoding the first frame 150. Also in the example described with reference to FIG. 1C , the second (non-displayed) frame 152 is added after the first frame 150 but before the third frame 154. The third frame 154 therefore has a reference 172 back in time to the second frame 152, and is, for example, a so-called P-frame. The first frame 150 can, for example, serve as the first frame in a GOP sequence of frames. Other frames can follow the third frame 154 and / or precede the first frame 150.

[0045] 1D is a schematic illustration of another possible example of how method 200 may operate to encode video stream 140 and overlay 120. Various steps S201-S206 are here the same as those described with reference to FIG. 1C, except that a software-generated second frame 152 is instead added after third frame 154. Third frame 154 is now a so-called B-frame, which now contains a reference to a "future frame" (second frame 152) as second frame 152 contains the macroblock that third frame 154 is to directly copy (because macroblock 162 of third frame 154 is a skip-macroblock that references 172 macroblock 163 of second frame 152). As explained above, the third frame 154 also includes a reference 170 to the preceding first frame 150 in order to properly copy and process the macroblocks 161 of the first frame 150, along with the motion vectors 132, and overlay 120 found in the first image 110.

[0046] Another scenario in which the method 200 contemplated herein may operate will now be described with reference to FIGS. 3A-3C. The following procedure is the same as that already described with reference to FIGS. 1A-1D, but the type of scene and camera setup are different. FIG. 3A shows a schematic representation of a scene captured by a stationary camera, with a first image 310 depicting a street sidewalk. In this scene, the particular object for which an overlay 320 is provided is a non-stationary, moving object in the form of a dog 324. Here, the overlay 320 identifies the type of animal as "dog," but it could, of course, provide other information that may be useful to a user viewing the scene through a video stream. As the dog 324 moves / walks along the sidewalk, its position will change between captured images, and the overlay must be updated accordingly to maintain its fixed position relative to the object / dog 324.

[0047] 3B shows a captured second image 312 depicting the same scene, but with the dog 324 in a different position since the first image 310 was captured, as indicated by arrow 330. Thus, the desired position of the overlay 320 in the second image 312 is different from the position of the overlay in the first image 310 (now indicated by overlay 320′).

[0048] As previously described, method 200 may now proceed by first capturing a first image 310 (at step S201), rendering / adding an overlay 320 (at step S202), and encoding the result as part of a first frame 350 of an encoded video stream 340. Method 200 may then proceed to capturing a second image 312 of the scene (at step S203), and method 200 may then determine the difference between the desired position of the overlay 320 in the second image 312 and the position of the overlay 320 in the first image 310 (at step S204). The method 200 can encode this second image 312 as part of a second frame 352 (marked as a hidden frame) that does not include the overlay 320 (at step S205), and then continue to generate and encode (e.g., using software) a third frame 354 (at step S206) having a temporally predicted macroblock 360 that references 370 a macroblock 361 of the first frame 350 at the location of the overlay 320 in the first image 310 with the help of one or more motion vectors 332 based on the difference between the desired location of the overlay 320 in the second image 312 and the desired location of the overlay 320 in the first image 310, and one or more skip-macroblocks 362 that reference 372 one or more macroblocks 363 of the second hidden frame 352 just described above with reference to Figures 1C and 1D. Thus, the method can be implemented as needed both when the desired position of the overlay in the image changes due to camera changes / movement (e.g., a change in the camera's FOV), when the desired position of the overlay in the image changes due to movement of an object in the scene (with which the overlay is associated), and / or when the desired position changes due to a combination of both camera changes and movement of an object in the scene.In other words, the methods disclosed and contemplated herein can work both when the overlay is fixed relative to the scene and when, for example, the overlay is fixed relative to an object moving through the scene.

[0049] It is contemplated herein that method 200 may be implemented, for example, by a monitoring camera used to capture images of a scene, or in any other suitable component of a (monitoring) camera system, for example including such a monitoring camera. Method 200 may, of course, also be implemented in any other device capable of receiving at least the captured first and second images, generating various frames, and encoding a video stream as output. An example of such a device as contemplated herein is described in more detail below with reference to FIG. 4.

[0050] 4 is a schematic diagram of a device 400 for encoding a video stream including an overlay. The device 400 includes at least a processor (or “processing circuitry”) 410 and a memory 412. As used herein, a “processor” or “processing circuitry” may refer to any combination of one or more suitable central processing units (CPUs), multiprocessors, microcontrollers (μCs), digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), graphics processing units (GPUs), etc., capable of executing software instructions stored in the memory 412. The memory 412 may be external to the processor 410 or internal to the processor 410. As used herein, “memory” may refer to any combination of random access memory (RAM) and read-only memory (ROM), or any other type of memory capable of storing instructions. The memory 412 includes (i.e., stores) instructions that, when executed by the processor 410, cause the device 400 to perform the method described herein (i.e., method 200 or any embodiment thereof). Device 400 may further include one or more additional items 414 that may be necessary in some circumstances to implement a method. In some exemplary embodiments, device 400 may be, for example, the monitoring camera mentioned above, in which case additional item 414 may include, for example, an image sensor and one or more lenses, for example, for focusing light from the scene onto the image sensor, so that the monitoring camera can capture images of the scene as part of implementing a contemplated method. Additional item 414 may also include, for example, various other electronic components necessary to capture the scene, for example, necessary for the image sensor and / or lenses to operate properly as needed.Implementation of the method in a monitoring camera can be useful in that processing is moved to the "edge," i.e., closer to where the actual scene is captured, compared to performing processing and video encoding elsewhere (such as on a more centralized processing server). Device 400 can be connected to a network, for example, so that the encoded video stream resulting from implementing the method can be transmitted to a user. To that end, device 400 can include a network interface 416, which can be, for example, a wireless network interface (e.g., as defined in IEEE 802.11 or any subsequent standard supporting, for example, Wi-Fi), or a wired network interface (e.g., as defined in IEEE 802.3 or any subsequent standard supporting, for example, Ethernet). Network interface 416 can also support any other wireless standard, such as Bluetooth, that is capable of transferring encoded video. These various components 410, 412, 414, and 416 (if present) can be connected via one or more communication buses 428 so that these components can communicate with each other and exchange data as necessary.

[0051] Device 400 may be, for example, a monitoring camera, e.g., in the form of a PTZ camera, mounted on or capable of being mounted on a building, or a fisheye camera, e.g., capable of providing a wider view of a scene, or any other type of monitoring / surveillance camera. Device 400 may also be, for example, a body camera, an action camera, a dashcam, etc., suitable for attachment to people, animals, and / or various vehicles, etc. Device 400 may also be, for example, a smartphone or tablet that a user can carry and capture scenes of interest to obtain further information from one or more overlays. Device 400 may also be or include, for example, a virtual headset that a user can wear to observe a scene via a video stream, etc. In any of these examples of device 400, it is contemplated that device 400 may include any necessary components (if any) other than those already described herein, as long as device 400 is still capable of performing method 200 or any embodiment thereof as contemplated herein.

[0052] Without any further details being provided herein, it is assumed that device 400 (e.g., a camera, etc.) may also be capable of identifying one or more objects in a scene, such as, for example, people, buildings, streets, vehicles, animals, tourist attractions, etc., and generating appropriate overlays for such objects to provide useful information to a user. Identification of objects may be performed based on, for example, machine learning algorithms and / or, for example, location information obtained from, for example, a GPS receiver included (or may be included) in device 400.

[0053] Another alternative embodiment of method 200 will now be described with reference to Figure 2B, which illustrates an alternative procedure following step S204 of method 200. In step S207 (which does not necessarily have to be performed after step S204), it is determined whether the computation time required to instead directly add / render the overlay as part of the second image / frame is less than a threshold. In other words, step S207 determines whether there are sufficient computational resources available to allow the overlay to be re-rendered in the usual (i.e., conventional) manner after the camera and / or objects in the scene move or change. Estimate The method includes the step of: Estimate If the estimated computation time is below a threshold (e.g., a lower threshold if few computational resources are available, or a higher threshold if more computational resources are currently available), the method can proceed to step S205′ (instead of proceeding to steps S205 and S206 described with reference to FIG. 2A ), where the overlay is (directly) added / rendered to the second image at the desired location and then (directly) encoded as part of the second frame of the video stream. Here, the second frame is not marked as a hidden frame, and the generation of a third frame referencing the first and second frames can be omitted. On the other hand, if it is determined that the available computational resources are not sufficient to perform such a direct re-rendering of the overlay in the second image in time, method 200 can proceed to steps S205 and S206 as already discussed.

[0054] 2B, the normal re-rendering of the overlay, e.g., when the camera and / or objects in the scene change, can be replaced with the assumed use of only the second, hidden frame and a software-generated (and inserted) third frame only when necessary. This replacement has the advantage, e.g., of improved image quality, since the normal re-rendering after the camera and / or objects move / change also takes into account any view changes in the scene that occur, e.g., between the capture of the first and second images, generally resulting in a better viewing experience.

[0055] The thresholds described above can be continuously updated based on the current load of, for example, a processor or other processing circuitry used in a device (e.g., a monitoring camera) responsible for outputting the encoded video stream. For example, if the device is heavily occupied with other resource-intensive tasks (such as tracking multiple objects in a scene), a contemplated method utilizing a second, hidden frame and a software-generated (and inserted) third frame can provide a faster way to update the overlay's position so that it remains fixed with acceptable visual quality relative to, for example, the scene or a particular object in the scene. Such checks of available computational resources and load can be performed, for example, multiple times per second, or at any desired interval. The contemplated method can thus be used as a backup when normal re-rendering of the overlay is not possible.

[0056] As generally envisioned herein, the desired location of the overlay in the second image can be obtained / determined, for example, by knowing the location of the overlay in the first image, and also by knowing, for example, how the camera has changed with respect to zoom level, pan, tilt, etc., between the capture of the first and second images. Based on such information from the camera, the difference between the desired location in the second image and the location of the overlay in the first image can then be calculated. If the camera is not mounted on a stationary object, such as a building, additional information may be needed to determine how the camera's orientation and position have changed. It is contemplated that such additional information can be obtained, if necessary, from one or more suitable sensors mounted, for example, on the camera itself and / or on any non-stationary object to which the camera is mounted. In other embodiments, it is contemplated that the desired location in the second image (and the corresponding difference relative to the first image, and the corresponding motion vector) can be found by identifying the correct location of the overlay in the second image using, for example, one or more image / video analysis algorithms, such as, for example, one or more image / video analysis algorithms for detecting and / or tracking objects.

[0057] To summarize the various embodiments presented herein, the present disclosure provides an improved method for providing (and updating) an overlay in an encoded video stream where the overlay's position in an image of the scene needs to be updated. The envisioned solution overcomes the problem of potentially having insufficient computational resources available to complete the overlay's re-rendering during each such time. By marking a second frame as a hidden frame and inserting a software-generated third frame that references the second frame instead, image information about the scene in areas where the overlay is not to be placed can be provided from the second frame to the third frame by a simple copy operation. Similarly, by also referencing the third frame back to the first frame, image information about the overlay itself can be provided to the third frame by copying this information from the last frame in which the overlay was properly rendered, i.e., from the first frame (naturally using appropriate motion vectors provided based on how the camera and / or objects in the scene have changed / moved). The encoder can thus generate the third frame (including the overlay in its new desired location) simply by referencing information already available in other frames, thus reducing the computational time required to prepare and encode the third frame without any need to re-render the overlay at its new location in the second image. Similarly, a corresponding decoder is not affected by this approach and can proceed as usual by copying information from other frames as instructed by the encoded video stream output from the envisioned method. In other words, currently available encoders that support concepts such as motion vectors, temporally predicted frames, hidden frames, etc. can be used without modification.The method can also be implemented using already available encoders without modifying them, as the third frame can be generated manually (using software) and simply inserted into (or combined with) the output from such an encoder to generate the encoded video stream contemplated herein.

[0058] Although features and elements are described above, in some cases in particular combinations, individual features or elements can be used alone without other features and elements, or in various combinations with or without other features and elements. Moreover, variations to the disclosed embodiments can be understood and implemented by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

[0059] In the claims, the words "comprise" and "include" do not exclude other elements, and the indefinite articles "a" or "an" do not exclude plural references. The mere fact that certain features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage. [Explanation of symbols]

[0060] 110, 310 First image of the scene 112, 312 Second image of the scene 120, 320 overlay 120', 320' Position of the overlay in the first image 124, 324 The specific object in the scene with which the overlay is associated 130, 330 Overlay position movement 132, 332 motion vector 140, 340 video streams 150, 350 First (image) frame 152, 352 Second (image) frame 154, 354 Third (image) frame 160, 360 Temporally predicted macroblock in the third frame 161, 361 Source macroblocks in the first frame 162, 362 skip-macroblocks in the third frame 163, 363 Source macroblocks in the second frame 170, 370 Reference to source macroblock in first frame 172, 372 Reference to source macroblock in second frame 200 Method (Flowchart) S201~S206 Method steps S205', S207 Alternative Method Steps 400 devices 410 Processor / Processing Circuitry 412 memory 414 Network Interface 416 Additional Items 428 Communication Bus

Claims

1. A method (200) for encoding a video stream including an overlay, comprising: a) Capturing (S201) a first image (110) of a scene; b) adding an overlay (120) to a first position of the first image and encoding (S202) the first image as part of a first frame (150) of an encoded video stream (140); c) capturing (S203) a second image (112) of the scene; d) using an object detection and / or tracking algorithm, calculating a desired change in the position of the overlay between the desired position of the overlay in the second image and the position of the overlay in the first image based on at least one of: i) information regarding a known change in camera field of view between the capture of the first image and the capture of the second image; ii) information regarding a known change in camera position between the capture of the first image and the capture of the second image; and iii) a known change in position of an object with which the overlay is associated in the scene between the capture of the first image and the capture of the second image (S204); e) encoding (S205) the second image as part of the second frame (152) of the video stream, including marking the second frame as a non-display frame; f) generating and encoding (S206) a third frame (154) of the video stream, wherein one or more macroblocks (160) of the third frame at the desired position of the overlay are temporally predicted macroblocks that reference (170) the first frame, and one or more macroblocks (162) of the third frame outside the desired position of the overlay are skip-macroblocks that reference (172) the second frame of the video stream, and calculating motion vectors (132) of one or more of the temporally predicted macroblocks based on the desired change in position of the overlay; A method comprising:

2. 2. The method of claim 1, wherein the third frame is a predicted frame (P-frame) or a bidirectionally predicted frame (B-frame) inserted after the second frame in the encoded video stream.

3. 3. The method of claim 1, wherein the third frame is a bidirectionally predicted frame (B-frame) inserted before the second frame in the encoded video stream.

4. The method of claim 1 or 2, comprising capturing the first image and the second image using the same camera.

5. The method of claim 1 or 2, wherein the method is implemented in a camera used to capture the first image and / or the second image.

6. The method of claim 1 or 2, wherein the overlay is fixed relative to the scene.

7. and estimating a computation time required to render and encode the overlay as part of the second image and the second frame, and upon determining that the estimated computation time is less than a threshold, performing steps a) through d) and not performing steps e) and f), instead performing after step d): e') adding (S205') said overlay to said second image at said desired location and encoding said second image as part of said second frame (152) of said video stream; Upon determining that the estimated computation time is not less than the threshold, performing steps e) and f).

3. The method according to claim 1 or 2.

8. A device (400) for encoding a video stream including an overlay, comprising: a processor (410); a memory (412) for storing instructions; the instructions, when executed by the processor, cause the device to: Capturing (S201) a first image (110) of a scene; adding (S202) an overlay (120) to a first position of the first image and encoding the first image as part of a first frame (150) of an encoded video stream (140); capturing (S203) a second image (112) of the scene; using an object detection and / or tracking algorithm to calculate a desired change in the position of the overlay between the desired position of the overlay in the second image and the position of the overlay in the first image based on at least one of: i) information regarding a known change in camera field of view between the capture of the first image and the capture of the second image; ii) information regarding a known change in camera position between the capture of the first image and the capture of the second image; and iii) a known change in position of an object with which the overlay is associated in the scene between the capture of the first image and the capture of the second image (S204); encoding (S205) the second image as part of the second frame (152) of the video stream, including marking the second frame as a non-display frame; generating and encoding (S206) a third frame (154) of the video stream, wherein one or more macroblocks (160) of the third frame at the desired position of the overlay are temporally predicted macroblocks that reference (170) the first frame, and one or more macroblocks (162) of the third frame outside the desired position of the overlay are skip-macroblocks that reference (172) the second frame of the video stream, the generating and encoding (S206) comprising calculating motion vectors (132) of one or more of the temporally predicted macroblocks based on the desired change in position of the overlay; A device that causes the device to perform the above.

9. The device of claim 8 , wherein the device is a monitoring camera configured to capture at least one of the first image and the second image.

10. 1. A non-transitory computer-readable storage medium having stored thereon a computer program for encoding a video stream including an overlay, the computer program, when executed by a processor of a device, causing the device to: Capturing a first image (110) of a scene (S201); adding (S202) an overlay (120) to a first position of the first image and encoding the first image as part of a first frame (150) of an encoded video stream (140); capturing (S203) a second image (112) of the scene; using an object detection and / or tracking algorithm to calculate a desired change in position of the overlay between the desired position of the overlay in the second image and the position of the overlay in the first image based on at least one of: i) information regarding a known change in camera field of view between the capture of the first image and the capture of the second image; ii) information regarding a known change in camera position between the capture of the first image and the capture of the second image; and iii) a known change in position of an object with which the overlay is associated in the scene between the capture of the first image and the capture of the second image (S204); encoding (S205) the second image as part of the second frame (152) of the video stream, including marking the second frame as a non-display frame; generating and encoding (S206) a third frame (154) of the video stream, wherein one or more macroblocks (160) of the third frame at the desired position of the overlay are temporally predicted macroblocks that reference (170) the first frame, and one or more macroblocks (162) of the third frame outside the desired position of the overlay are skip-macroblocks that reference (172) the second frame of the video stream, the generating and encoding (S206) comprising calculating motion vectors (132) of one or more of the temporally predicted macroblocks based on the desired change in position of the overlay; 1. A non-transitory computer-readable storage medium configured to cause

Images