System and method for adding additional image frames to a stream of encoded image frames

Abstract

The present invention relates to a system and method for adding an additional image frame to a stream of encoded image frames. The stream of encoded image frames consists of image frames encoded by an encoder. An event related to at least one of i) a change in the operating state of a transmitter of the stream of encoded image frames and ii) a change in the connection state between the transmitter of the stream of encoded image frames and a receiver is detected. Depending on the type of the detected event, an additional image frame or a sequence of additional image frames is provided. Each additional image frame of an additional image frame or a sequence of additional image frames includes a reference to a previous image frame and encodes a predetermined visual change relative to the previous image frame. The predetermined visual change indicates the type of the detected event. An additional image frame or a sequence of additional image frames is added to the stream of encoded image frames to obtain a combined image frame stream.

Description

Technical Field

[0001] The invention relates to adding an additional image frame or a sequence of additional image frames to a coded image stream, and in particular, wherein the additional image frame or the sequence of additional image frames encodes a predetermined visual effect. Background Art

[0002] In the case where a stream of coded image frames of a video captured by a camera is received, decoded and displayed in a display (e.g., at a remote location) and the camera is intentionally turned off, the camera is turned off due to a malfunction, or the connection between the camera and the display location fails, the stream will be interrupted and image frames will no longer be received at the display location. The display of the video will then be interrupted (e.g., by not displaying image frames at all or by continuously displaying the last received image frames). Therefore, the person watching the video may be aware that the stream has been interrupted, but does not know why. Similarly, when the camera is restarted or the restoration of the connection between the camera and the display location is initiated, and before a new stream of coded image frames is generated and received at the display location for decoding and display, the person watching the display will not know that the video is about to be displayed. Summary of the invention

[0003] An object of the present invention is to enable identification of the type of a detected event in a decoded and displayed coded image frame stream, wherein the type of event is one of a change in the operating state of a transmitter of the coded image frame stream and a change in the connection state between the transmitter and a receiver of the coded image frame stream.

[0004] According to a first aspect, a method for adding an additional image frame or a sequence of additional image frames to a stream of coded image frames is provided, wherein the stream of coded image frames consists of image frames encoded by an encoder. The method includes detecting an event related to at least one of the following types: a change in the operating state of a transmitter of the stream of coded image frames and a change in the connection state between the transmitter and a receiver of the stream of coded image frames. Depending on the type of event detected, an additional image frame or a sequence of additional image frames is provided, wherein each additional image frame of an additional image frame or a sequence of additional image frames includes a reference to a previous image frame and encodes a predetermined visual change relative to the previous image frame. The predetermined visual change indicates the type of event detected. An additional image frame or a sequence of additional image frames is added to the stream of coded image frames to obtain a combined image frame stream.

[0005] Since one additional image frame or a sequence of additional image frames encodes a predetermined visual change relative to a previous image frame, and the predetermined visual change indicates the type of the detected event, when the combined stream is decoded and displayed, the combined image frame stream will include information capable of identifying the type of the detected event. Specifically, it is possible to identify whether the type of event is a change in the operating state of the transmitter of the encoded image frame stream or a change in the connection state between the transmitter of the encoded image frame stream and a receiver.

[0006] Since the predetermined visual change is encoded with the aid of an additional image frame or a sequence of additional image frames, information capable of identifying the type of detected event can be transmitted in the combined stream and decoded and displayed with the aid of a decoder and a display that are configured to decode and display the encoded image frame stream without the need for adjusting the decoder and the display.

[0007] Since the visual change is predetermined and related to the previous image frame, an additional image frame or sequence of additional image frames can be generated in advance (i.e., before the event is detected) and stored for provision and addition after the event is detected. An additional image frame or sequence of additional image frames only needs to include data encoding the visual change, without any real-time data associated with the encoded image frame stream.

[0008] The method of the first aspect may further include: generating an additional image frame or each additional image frame of the sequence of additional image frames based only on the block type (e.g., whether the block is an inter-frame block such as a P-block or a B-block, an intra-frame block or a skip block), and a predetermined motion vector and / or a predetermined frequency domain coefficient. Therefore, an additional image frame or a sequence of additional image frames can be generated using only software. In addition, an additional image frame or a sequence of additional image frames can be generated in advance (i.e., before an event is detected) and stored after the event is detected for provision and addition to the coded image frame stream.

[0009] In the method of the first aspect, an additional image frame or a sequence of additional image frames may each further include at least one of the following: one or more inter-frame blocks having predetermined motion vectors and / or predetermined quantized frequency domain coefficients and one or more intra-frame blocks having predetermined quantized frequency domain coefficients. An additional image frame or a sequence of additional image frames may each further include one or more skipped blocks.

[0010] When used in this document, the expressions "predetermined motion vector" and "predetermined quantized frequency domain coefficients" mean that the motion vector and the quantized frequency domain coefficients are predetermined and encode a predetermined visual change alone or in combination. The quantized frequency domain coefficients may be, for example, quantized discrete cosine transform (DCT) coefficients, or other alternative coefficients depending on the coding standard. In AV1, such alternative coefficients may be quantized asymmetric discrete sine transform (DST) coefficients, quantized flipped ADST (FLIPADST) coefficients, quantized identity transform coefficients, and quantized Walsh-Hadamard transform (WHT) coefficients. The latter may also be used in VP8.

[0011] In the method of the first aspect, the previous image frame can be one of the following: an encoded image frame in an encoded image frame stream, an added starting image frame for intra-frame encoding of a starting image, and an additional image frame in an additional image frame sequence. The previous image frame can further be an immediately preceding image frame in a combined image frame stream. In an alternative solution, it can be an image frame immediately before the immediately preceding image frame.

[0012] In the method of the first aspect, the sequence of additional image frames comprises at least one of: additional image frames with different references and same content, additional image frames with the same reference and different content, and additional image frames with different references and different content.

[0013] The method of the first aspect may further comprise transmitting the combined image frame stream to a decoder, decoding the combined image frame stream, and displaying the decoded combined image frame stream on a display so as to display a predetermined visual change indicative of a detected event.

[0014] In the method of the first aspect, the detected event may be associated with at least one of the following types: shutdown of the transmitter of the coded image frame stream, connection failure between the transmitter of the coded image frame stream and the receiver, failure of the transmitter of the coded image frame stream, startup of the transmitter of the coded image frame stream, reconnection between the transmitter of the coded image frame stream and the receiver, and restart of the transmitter of the coded image frame stream. In addition, depending on whether the shutdown of the transmitter is initiated at the receiver or at the transmitter, the shutdown can be divided into two independent types of events.

[0015] In the method of the first aspect, the predetermined visual change may be one of a motion, a color change, and a brightness change.

[0016] According to a second aspect, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium having instructions stored thereon for implementing the method according to the first aspect when executed in a system having at least one processor.

[0017] Where applicable, the above optional features of the method according to the first aspect also apply to the non-transitory computer readable storage medium of the second aspect. In order to avoid unnecessary repetitions, reference is made to the above.

[0018] According to a third aspect, a system for adding an additional image frame or a sequence of additional image frames to a stream of coded image frames is provided, wherein the stream of coded image frames consists of image frames encoded by an encoder. The system includes a circuit configured to perform a detection function, the detection function being configured to detect an event associated with at least one of the following types: a change in the operating state of a transmitter of the stream of coded image frames and a change in the connection state between the transmitter and a receiver of the stream of coded image frames. The circuit is further configured to perform a provision function, the provision function being configured to provide an additional image frame or a sequence of additional image frames depending on the type of event detected by the detection function, wherein each additional image frame of an additional image frame or a sequence of additional image frames includes a reference to a previous image frame and encodes a predetermined visual change relative to the previous image frame, wherein the predetermined visual change indicates the type of the detected event. The circuit is further configured to perform an addition function, the addition function being configured to add an additional image frame or a sequence of additional image frames to the stream of coded image frames to obtain a combined image frame stream.

[0019] Where applicable, the above optional features of the method according to the first aspect also apply to the system of the third aspect.In order to avoid unnecessary repetitions, reference is made to the above.

[0020] Further scope of applicability of the present invention will become apparent from the detailed description given below. However, it should be understood that the detailed description and specific examples, although indicating preferred embodiments of the present invention, are only given by way of illustration, because various changes and modifications within the scope of the present invention will become apparent to those skilled in the art from the detailed description.

[0021] Therefore, it should be understood that the present invention is not limited to the specific components of the described system or the actions of the described method, because such systems and methods can vary. It should also be understood that the terms used herein are only for the purpose of describing specific embodiments and are not intended to be limiting. It must be noted that, as used in the specification and the appended claims, the articles "a", "the" and "said" are intended to indicate the presence of one or more elements, unless the context clearly indicates otherwise. In addition, the words "comprise", "include" and similar expressions do not exclude other elements or steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other aspects of the present invention will now be described in more detail with reference to the accompanying drawings.The drawings should not be considered limiting but are for explanation and understanding.

[0023] Figure 1 A flow chart related to an embodiment of the method of the present disclosure is shown.

[0024] Figure 2 A schematic diagram related to an embodiment of the system of the present disclosure is shown.

[0025] Figure 3 A schematic diagram showing a system in which embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0026] The present invention will now be described hereinafter with reference to the accompanying drawings, in which currently preferred embodiments of the present invention are illustrated. However, the present invention may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. In the accompanying drawings, dashed boxes represent optional features.

[0027] The present invention is applicable to Figure 3 The system 300 shown in FIG. Figure 3 In the embodiment of the present invention, a stream of encoded image frames will be transmitted from a transmitter 310 to a receiver 320. The transmitter 310 may be, for example, a camera, such as a surveillance camera, or a system including a camera at a camera location. The receiver 320 may be a device including a decoder and connected to a display at a display location. Another intermediate device 315 may be located between the transmitter 310 and the receiver 320. Each of the transmitter 310, the intermediate device 315, and the receiver 320 may detect an event related to at least one type of change in an operating state of the transmitter 310 and a change in a connection state between the transmitter 310 and the receiver 320.

[0028] The present invention is applicable to the case where a coded image frame stream is transmitted from a transmitter 310 to a receiver 320 for decoding and display. The coded image frame stream may be composed of images captured by a camera and encoded by an encoder, for example. Alternatively, some or all of the coded frame streams may be images created in software. The coded frame stream may be a video, and the camera may be a camera set at a display position away from where a person watches the video. If the transmission of the coded image frame stream is interrupted so that the receiver 320 no longer receives any coded image frames, the video will be interrupted, and the person watching the video will not know why the video is interrupted. A combined stream is created by adding an additional image frame or an additional image frame sequence that encodes a predetermined visual change relative to a previous image frame. The predetermined visual change indicates the type of event detected for the coded image frame stream. For example, at least two different predetermined visual changes are selectable, and each change indicates a corresponding one of two types of events, i.e., i) a change in the operating state of the transmitter 310 of the coded image frame stream, and ii) a change in the connection state between the transmitter 310 and the receiver 320 of the coded image frame stream. When the combined stream is then decoded and displayed, the viewer can recognize from the predetermined visual change which type of event has occurred. Specifically, the viewer can recognize whether the type of event is a change in the operating state of the transmitter 310 of the encoded image frame stream or a change in the connection state between the transmitter 310 of the encoded image frame stream and the receiver 320.

[0029] By including information that can identify the type of event detected in the form of an additional image frame or additional frame sequence that encodes a predetermined visual change (the additional image frame or additional frame sequence is added to the coded image frame stream and transmitted in the combined stream), there is no need for adaptation of the interface between the transmitter 310 and the receiver 320. In addition, the decoder and display at the receiver 320 do not need to be adapted. They can decode and display the combined stream in the same way as decoding and displaying the coded image frame stream. The only thing required is that the viewer knows which type of event is indicated by which predetermined visual change. In addition, since the visual change is predetermined and related to the previous image frame, the additional image frame or additional image frame sequence can be generated in advance (i.e., before the event is detected), and they are stored after the event is detected for provision and addition. The additional image frame or additional image frame sequence only needs to include data that encodes the visual change, without any real-time data related to the coded image frame stream.

[0030] It should be noted that even though no coding is mentioned in the terms "an additional image frame" and "a sequence of additional image frames", both terms refer to one or more image frames generated in this way and having a syntax according to the relevant coding standard used, so that they can be decoded by a decoder decoding the coded image frame stream without adaptation. For example, before adding the additional image frame or the sequence of additional image frames to the coded image frame stream to obtain a combined image frame stream, entropy coding must be performed using an entropy encoder configured in hardware and / or software.

[0031] In the following, reference will be made to Figure 1 and Figure 3 An embodiment of a method 100 for adding an additional image frame or a sequence of additional image frames to a stream of coded image frames is discussed, in which embodiment the stream of coded image frames consists of image frames encoded by an encoder.

[0032] Typically, some of the image frames of the coded image frame stream are intraframes, while some are interframes. An intraframe is an encoded video frame that does not require information from other frames to be encoded. Therefore, an intraframe is encoded based on information from an image frame of video data (with which the intraframe is set to correspond). Typically, similarities within an image frame are used to encode an image frame into an intraframe. In video encoding, an intraframe is typically referred to as an I frame. An image frame between two intraframes of a video stream is encoded as an interframe. Interframes are encoded based on information from at least one other image frame to be encoded (with which the interframe is set to correspond) rather than an image frame of video data. Typically, an interframe includes less data than an intraframe. An interframe may be a P frame or a B frame. A P frame refers to a previous frame for data reference. Therefore, in order to decode a P frame, the content of the previous frame must be known. A B frame may refer to both a previous frame and a forward frame for data reference. Therefore, in order to decode a B frame, the content of both a previous frame and a forward frame must be known. When encoding an interframe, an image frame is divided into a plurality of pixel groups. A pixel group may be referred to, for example, as a block, a macroblock, or a coding tree unit. Each inter-frame may include one or more blocks encoded as inter-frame blocks, one or more blocks encoded as intra-frame blocks, and one or more blocks encoded as skip blocks. The image frame is compared to a reference frame. For example, the reference frame used to encode a P frame is a previous image frame. In order to encode a sequence of image frames in an encoder, a matching algorithm may be used to identify matching pixel groups between the image frame to be encoded and the reference frame, and if a match is found for a pixel group, the pixel group may be encoded based on a motion vector specifying how the pixel group has moved in the image frame since the reference frame.

[0033] The method 100 includes detecting S110 an event related to at least one of the following types: i) a change in the operating state of the transmitter 310 of the coded image frame stream, and ii) a change in the connection state between the transmitter 310 of the coded image frame stream and the receiver 320. The transmitter 310 may be, for example, a camera or a system including a camera that transmits the coded image frame stream. The change in the operating state of the transmitter 310 may be an operating state in which the transmitter 310 changes to an operating state in which the coded image frame stream is not transmitted or the coded image frame stream is transmitted. The change in the operating state may, for example, involve an intentional shutdown of the transmitter 310, a failure of the transmitter 310, and a restart of the transmitter 310. In addition, depending on whether the shutdown is initiated at the receiver or at the transmitter, the intentional shutdown of the transmitter may be divided into two different types of events. The change in the connection state between the transmitter 310 and the receiver 320 may be a connection state in which the coded image frame stream cannot be transmitted or the coded image frame stream can be transmitted between the transmitter 310 and the receiver 320. The change of the connection state may, for example, involve a connection failure between the transmitter 310 and the receiver 320 and a reconnection between the transmitter 310 and the receiver 320 .

[0034] Depending on the type of the detected event, an additional image frame or a sequence of additional image frames is provided S120. An additional image frame or each additional image frame in the sequence of additional image frames comprises a reference to a previous image frame and encodes a predetermined visual change relative to the previous image frame, wherein the predetermined visual change indicates the type of the detected event. An additional image frame or each additional image frame in the sequence of additional image frames comprises a reference to a previous image frame meaning that the previous frame is used as a reference frame.

[0035] The event may be detected by the sender 310 or the receiver 320 or the intermediate device 315. If the type of event is a change in the operating state of the sender (such as an intentional shutdown of the sender 310, a restart of the sender 310, or a failure of the sender 310), the event may be detected by the sender 310. If the type of event is a change in the connection state between the sender 310 and the receiver 320 (such as a connection failure between the sender 310 and the receiver 320 and a reconnection between the sender 310 and the receiver 320), the event may be detected by the receiver 320 or the intermediate device 315.

[0036] The predetermined visual change may be any kind of visual change that can be encoded and that can be recognized by a viewer and distinguished from other predetermined visual changes that are encoded to indicate other types of events. For example, the predetermined visual change may be motion, color change, or brightness change.

[0037] For example, if the predetermined visual change involves a predetermined movement of pixels (e.g., a predetermined movement of a block of pixels relative to a previous frame), such movement may be achieved by a corresponding predetermined motion vector for the block. Because the movement is predetermined, the motion vector is predetermined. Since the movement of the pixels is predetermined so that it can indicate the type of event detected, the motion vector may be determined without any input other than the predetermined movement of the pixels relative to the previous frame that should be achieved in accordance with the predetermined visual change. This is in contrast to the identification of motion vectors when encoding a sequence of image frames in an encoder, in which a matching algorithm is used to identify matching blocks between the image frame to be encoded and a reference frame, and if a match for the block is found, the block may be encoded based on a motion vector specifying how the block has moved in the image frame since the reference frame.

[0038] Similarly, if the predetermined visual change involves a predetermined change in the color or brightness of a pixel (e.g., a predetermined change in the color or brightness of a block of pixels relative to a previous frame), such a change in color or brightness may be achieved by corresponding predetermined frequency domain coefficients of the block. Since the change in color or brightness is predetermined, the frequency domain coefficients are predetermined. Since the change in color or brightness of the pixel is predetermined so that it can indicate the type of event detected, the frequency domain coefficients may be determined without any other input other than the predetermined change in color or brightness of the pixel relative to a previous frame that should be achieved according to the predetermined visual change.

[0039] An additional image frame or a sequence of additional image frames may be generated S115 after detecting S110 event or may have been generated S115 before detecting S110 event and stored for provision after detecting S110 event. Figure 1 The position of generating S115 an additional image frame or a sequence of additional image frames after detecting S110 event does not indicate its position in method 100. Generating S115 an additional image frame or a sequence of additional image frames can be performed before or after detecting S110 event. In addition, it can be performed to a certain extent before detecting S110 event and to a certain extent after detecting S110 event. For example, an additional image frame or a sequence of additional image frames can be generated until before entropy coding and stored before detecting S110 event, and then, entropy coding can be performed after detecting S110 event.

[0040] An additional image frame or each additional image frame in a sequence of additional image frames comprises a reference to a previous image frame, and a predetermined visual change relative to the previous image frame is encoded. In order to encode the predetermined visual change, an additional image frame or each additional image frame in a sequence of additional image frames is generated to include data defining the predetermined change relative to the previous image frame.

[0041] In an embodiment, an additional image frame or each additional image frame of a sequence of additional image frames may be generated S115 based only on a predetermined motion vector and / or a predetermined quantized frequency domain coefficient. An additional image frame or each additional image frame of a sequence of additional image frames may be an inter-frame frame and include one or more inter-frame blocks having a motion vector and / or a quantized frequency domain coefficient, and / or one or more intra-frame blocks having a quantized frequency domain coefficient. One or more inter-frame frames and / or one or more intra-frame blocks encode a predetermined visual change associated with a reference image frame (such as a predetermined motion associated with a reference image, a predetermined color change, and / or a predetermined brightness change). An additional image frame or each additional image frame of a sequence of additional image frames may also include one or more skip blocks.

[0042] Generating S115 an additional image frame or an additional image frame sequence based only on the block type (e.g., whether the block is an inter-frame block, an intra-frame block, or a skip block such as a P block or a B block), and a predetermined motion vector and / or a predetermined quantized frequency domain coefficient can be performed entirely in software until before entropy coding, and then the entropy coding can be performed by a software or hardware entropy encoder. This generation performed entirely in software can be referred to as procedural generation. No image data is required for generating S115. For coding standards in which slice headers and slice data can be encoded separately, slice data can be entropy encoded and stored, and slice headers need to be updated before being added. For example, CABAC in the main and advanced parts of H.264 and in H.265 belongs to this case. For coding standards in which slice headers and slice data cannot be encoded separately, for example, because entropy coding of slice data requires some bits of the slice header as input and therefore all slice headers need to be updated, the slice header needs to be updated, and then the slice data needs to be entropy encoded before being added. For example, CAVLC in H.264 belongs to this case.

[0043] An additional image frame or a sequence of additional image frames is added S130 to the stream of coded image frames to obtain a combined image frame stream.

[0044] The situation that provides and adds an additional image frame is for example when the last image frame provided to the display is displayed continuously until the next image frame is provided to the display. In this case, an additional image frame is sufficient to encode a predetermined visual change relative to the previous image frame, and the predetermined visual change can be recognized by the viewer.

[0045] In the case where a predetermined visual change that is predetermined and changes gradually over time is to be encoded, an additional frame sequence is required to encode it. For example, a plurality of additional frames of the same type may be included in the sequence, wherein each additional frame has the same predetermined motion vector and / or predetermined quantized frequency domain coefficients, but they refer to the frame immediately preceding it in the stream. Therefore, these additional frames have the same content but different references, and therefore, they encode the same visual change relative to the previous frame. When such an additional frame sequence is added after the coded frame stream and then decoded and displayed, they display the last decoded coded frame as gradually changing. For example, if each of the 10 additional frames encodes a rotation of 5 degrees to the previous frame, when added after the coded frame stream, they will cause the image of the last decoded coded frame to rotate from 0 degrees to 50 degrees in steps of 5 degrees.

[0046] An additional image frame or an additional frame sequence may be added by the transmitter 310, the receiver 320, or the intermediate device 315. If the type of event is a change in the operating state of the transmitter (such as an intentional shutdown of the transmitter 310, a restart of the transmitter 310, or a failure of the transmitter 310), an additional image frame or an additional frame sequence may be added by the transmitter 310 or the receiver 320 or the intermediate device 315. If the type of event is a change in the connection state between the transmitter 310 and the receiver 320 (such as a connection failure between the transmitter 310 and the receiver 320 and a reconnection between the transmitter 310 and the receiver 320), an additional image frame or an additional frame sequence may be added by the transmitter 310, the receiver 320, or the intermediate device 315.

[0047] An additional image frame or a sequence of additional image frames can be added, for example, after the coded image frame stream or before the coded image frame stream. An additional image frame or a sequence of additional image frames is preferably added in the position and order in the stream that should be decoded relative to other image frames of the coded image frame stream in the decoder at the receiver 320. Each additional image frame of an additional image frame or a sequence of additional frames includes a frame number and a picture sequence count to identify when it should be displayed relative to other image frames of the combined image frame stream. If an additional image frame or a sequence of additional image frames encodes a visual change, the visual change indicates that the operating state has become an operating state in which the coded image frame stream is not transmitted or the connection state between the transmitter 310 and the receiver 320 has become a connection state in which the coded image frame stream cannot be transmitted, then the predetermined visual change should be displayed at the end of the coded image frame stream, and therefore, an additional image frame or a sequence of additional image frames is preferably added after the last coded image frame of the coded image frame stream. This change in operating state can, for example, involve a deliberate shutdown of the transmitter 310, a failure of the transmitter 310, and a connection failure between the transmitter 310 and the receiver 320. If an additional image frame or a sequence of additional image frames encodes a visual change that indicates that the operating state has changed to an operating state for transmitting a stream of coded image frames or that the connection state between the transmitter 310 and the receiver 320 has changed to a connection state in which a stream of coded image frames can be transmitted, the predetermined visual change should be displayed at the beginning of the stream of coded image frames, and therefore, an additional image frame or a sequence of additional image frames is added before the first coded image frame of the stream of coded image frames. This change in operating state may, for example, involve a restart of the transmitter 310 and a reconnection between the transmitter 310 and the receiver 320. As will be described below, when an additional image frame or a sequence of additional image frames is added before the first coded image frame, a previous image frame that is a starting image frame needs to be added. In this case, the first coded image frame of the stream of coded image frames needs to be an intra-frame.

[0048] The previous image frame may be a coded image frame in a coded image frame stream. For example, if an additional image frame or a sequence of additional image frames should be added after the last coded image frame in the coded image frame stream, such as when the detected event is a deliberate shutdown of the transmitter 310, a failure of the transmitter 310, or a connection failure between the transmitter 310 and the receiver 320, then at least one image frame of an additional image frame or a sequence of additional image frames includes a reference to a coded image frame in the coded image frame stream. For example, an additional image frame may include a reference to a coded image frame in a coded image frame stream (such as the last coded image frame in the coded image frame stream). Similarly, for example, the first image frame of the sequence of additional image frames may include a reference to a coded image frame in a coded image frame stream (such as the last coded image frame in the coded image frame stream). Other image frames after the first image frame of the sequence of additional image frames may include a reference to a previous image frame in the sequence of additional image frames (such as an immediately previous image frame in the sequence of additional image frames). In this case, the previous image frame is an additional image frame in the sequence of additional image frames. Alternatively, all image frames of the additional sequence of image frames may comprise a reference to an encoded image frame in the stream of encoded image frames, such as the last encoded image frame of the stream of encoded image frames.

[0049] Alternatively, the previous image frame may be an added starting image frame for intra-frame encoding of the starting image. For example, if an additional image frame or a sequence of additional image frames should be added before the first encoded image frame of the encoded image frame stream, such as when the detected event is a restart of the transmitter 310 or a reconnection between the transmitter 310 and the receiver 320, then an additional image frame or at least one image frame of the sequence of additional image frames includes a reference to the added starting image frame. Other image frames after the first image frame of the sequence of additional image frames may then include a reference to a previous image frame in the sequence of additional image frames (such as an immediately preceding image frame in the sequence of additional image frames). In this case, the previous image frame is an additional image frame in the sequence of additional image frames. Alternatively, all image frames of the sequence of additional image frames may include a reference to the added starting image frame.

[0050] The image frames of the additional image frame sequence may have different references and picture order counts, but have the same content. For example, as described above in this document, the first image frame of the additional image frame sequence may include a reference to the last encoded image frame of the encoded image frame stream or the added starting image, and other image frames after the first image frame of the additional image frame sequence may include a reference to a previous image frame in the additional image frame sequence (such as the immediately preceding image frame in the additional image frame sequence (i.e., in the combined image frame stream)). Then, the image frames of the additional image frame sequence with the same content may represent that a predetermined same visual change relative to the previous image frame is encoded in each additional image frame of the additional image frame sequence. For example, the predetermined visual change in each consecutive additional image frame of the additional image frame sequence may be associated with a predetermined same motion, so that the additional image frame sequence encodes a predetermined uniform motion of pixels. The predetermined visual change in each consecutive additional image frame of the additional image frame sequence may alternatively or additionally be associated with a predetermined same change in color, so that the additional image frame sequence encodes a predetermined uniform change in color of pixels. The predetermined visual change in each consecutive additional image frame of the sequence of additional image frames may alternatively or additionally be associated with a predetermined identical change in brightness, such that the sequence of additional image frames encodes a predetermined uniform change in brightness.

[0051] The image frames of the additional image frame sequence may have different references and different contents. For example, as described above in this article, the first image frame of the additional image frame sequence may include a reference to the last encoded image frame of the encoded image frame stream or the added starting image, and the other image frames after the first image frame of the additional image frame sequence may include a reference to the previous image frame in the additional image frame sequence (such as the immediately previous image frame in the additional image frame sequence). The image frames of the additional image frame sequence with different contents may then represent that a predetermined different visual change relative to the previous image frame is encoded in each additional image frame of the additional image frame sequence. For example, the predetermined visual change in each consecutive additional image frame of the additional image frame sequence may be associated with a predetermined increased motion or a predetermined decreased motion, so that the additional image frame sequence encodes a predetermined accelerated motion or a predetermined decelerated motion of the pixel, respectively. The predetermined visual change in each consecutive additional image frame of the additional image frame sequence may be alternatively or additionally associated with a predetermined increased change in color or a predetermined decreased change, so that the additional image frame sequence encodes a predetermined accelerated change in color or a predetermined decelerated change in color of the pixel, respectively. The predetermined visual change in each consecutive additional image frame of the sequence of additional image frames may alternatively or additionally be associated with a predetermined increasing change or a predetermined decreasing change in brightness, such that the sequence of additional image frames encodes a predetermined accelerating change or a predetermined decelerating change in brightness of the pixels, respectively.

[0052] The image frames of the additional image frame sequence may have the same reference and different content. For example, as described above in this article, all image frames of the additional image frame sequence may include a reference to an encoded image frame in an encoded image frame stream (such as the last encoded image frame of the encoded image frame stream or an added starting image). The image frames of the additional image frame sequence with different content may then represent that different predetermined visual changes relative to the same previous image frame (such as the last encoded image frame of the encoded image frame stream or an added starting image) are encoded in each additional image frame of the additional image frame sequence. For example, the predetermined visual change in each consecutive additional image frame of the additional image frame sequence may be associated with a predetermined uniform large motion, so that the additional image frame sequence encodes the predetermined uniform motion of the pixel. The predetermined visual change in each consecutive additional image frame of the additional image frame sequence may alternatively or additionally be associated with a predetermined uniform large change in color, so that the additional image frame sequence encodes the predetermined uniform change in color of the pixel. The predetermined visual change in each consecutive additional image frame of the additional image frame sequence may alternatively or additionally be associated with a predetermined uniform large change in brightness, so that the additional image frame sequence encodes the predetermined uniform change in brightness of the pixel. The change in motion, color and / or brightness may also be made so that it may be accelerated (i.e., so that the change becomes larger and larger between successive additional image frames in the sequence of additional image frames), or decelerated (i.e., so that the change becomes smaller and smaller between successive additional image frames in the sequence of additional image frames).

[0053] For example, having the same content or different content means having the same slice data or different slice data in the H.264 and H.265 video standards, respectively.

[0054] A variety of different predetermined visual changes can be envisioned using one or a combination of, for example, motion, color change, and brightness change. For example, if the additional image frame sequence encodes a visual change relative to the encoded image frame of the encoded image frame stream, wherein the encoded visual change is that all pixels in each consecutive additional image frame of the additional image frame sequence move a given distance toward the center or only in the horizontal direction or only in the vertical direction or in both directions, then the visual change is achieved when the encoded image frames of the encoded image frame stream gradually shrink into horizontal lines, vertical lines or points, respectively. Because this will move the edge pixels toward the center in each consecutive additional image frame of the additional image frame sequence, these edge pixels need to be changed or faded to a single color so that the final result is a single color screen, except for the center pixels that can retain some other colors. The predetermined visual change can be encoded in the additional image frame sequence, which includes a predetermined motion vector and a predetermined quantized frequency domain coefficient.

[0055] In another example, if the sequence of additional image frames encodes a visual change relative to the encoded image frames of the encoded image frame stream, wherein the encoded visual change is a movement of a pixel block around a center by a given angle in each consecutive additional image frame of the sequence of additional image frames, then the visual change is achieved when the encoded image frames of the encoded image frame stream are rotated around their center. The predetermined visual change can be encoded in the sequence of additional image frames including a predetermined motion vector and works for smaller rotation angles in each consecutive additional image frame. The reason why it only works for "smaller" rotation angles is that the motion is performed by moving pixel blocks, and therefore, for any pixel block larger than one pixel, this is only an approximation of the rotational motion. Larger motions that approximate the rotational motion in each consecutive additional image frame of the sequence of additional image frames are possible, but will introduce artifacts associated with the rotational motion. The size of the angular motion in each consecutive additional image frame of the sequence of additional image frames depends on the size of the pixel block. For example, for a pixel block of size 8×8 pixels, the movement in each consecutive additional image frame of the sequence of additional image frames corresponding to 7 degrees will be at least small enough. Movement in each successive additional image frame of a sequence of additional image frames corresponding to a larger angle is also possible, but will introduce more artifacts. Movement corresponding to a desired total rotational movement can be achieved by providing a sequence of additional image frames having a sufficient number of additional image frames such that movement in each successive additional image frame of a sequence of additional image frames corresponding to a "smaller" angle will produce the desired total angle over all the sequences of additional image frames. Any desired angle of any size can then be achieved.

[0056] In another example, if the sequence of additional image frames encodes a visual change relative to the coded image frames of the stream of coded image frames, wherein the encoded visual change is a decrease or increase in brightness of all pixels in each consecutive additional image frame of the sequence of additional image frames by a given amount, then the visual change is achieved when the coded image frames of the stream of coded image frames gradually turn black or gradually turn white, respectively. The predetermined visual change may be encoded in the sequence of additional image frames comprising predetermined quantized frequency domain coefficients.

[0057] In another example, if the sequence of additional image frames encodes a visual change relative to an encoded image frame of the stream of encoded image frames, wherein the encoded visual change is a decrease or increase in brightness of a selected pixel in each consecutive additional image frame of the sequence of additional image frames by a given amount, then the visual change is achieved when the selected pixels of the encoded image frame of the stream of encoded image frames gradually become black or gradually become white, respectively. The selected pixels can, for example, be pixels selected to form a desired text message in the image frame. The predetermined visual change can be encoded using predetermined frequency domain coefficients.

[0058] In another example, if the sequence of additional image frames encodes a visual change relative to the coded image frames of the stream of coded image frames, wherein the encoded visual change is an increase of a given amount in a particular color of all pixels in each consecutive additional image frame of the sequence of additional image frames, and a decrease of a corresponding given amount in all other colors of all pixels, then a visual change is achieved where the coded image frames of the stream of coded image frames gradually become the particular color. The predetermined visual change may be encoded in the sequence of additional image frames comprising predetermined quantized frequency domain coefficients.

[0059] In another example, if the sequence of additional image frames encodes a visual change relative to the encoded image frames of the stream of encoded image frames, wherein the encoded visual change is an increase in a given amount of a specific color of a selected pixel in each consecutive additional image frame of the sequence of additional image frames, then the visual change is achieved where the selected pixels of the encoded image frames of the stream of encoded image frames gradually become a specific color. The selected pixels can, for example, be pixels selected to form a desired text message in the image frame. The predetermined visual change can be encoded in the sequence of additional image frames including predetermined quantized frequency domain coefficients.

[0060] The method 100 may further comprise transmitting S140 the combined image frame stream to a decoder, e.g. comprised in the receiver 320, decoding S150 the combined image frame stream and displaying S160 the decoded combined image frame stream, e.g. on a display connected to the receiver 320. When the combined stream is displayed, a predetermined visual change indicating the detected event will be displayed.

[0061] Figure 2 A schematic diagram related to an embodiment of a system 200 of the present disclosure is shown, and the system 200 is used to add an additional image frame or a sequence of additional image frames to a coded image frame stream, wherein the coded image frame stream is composed of image frames encoded by an encoder. The system 200 includes a circuit 210. The circuit 210 is configured to perform the functions of the system 200. The circuit 210 may include a processor 212, such as a central processing unit (CPU), a microcontroller, or a microprocessor. The processor 212 is configured to execute program code. The program code can, for example, be configured to perform the functions of the system 200.

[0062] The system 200 may further include a memory 230. The memory 230 may be one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, a random access memory (RAM), or other suitable devices. In a typical arrangement, the memory 230 may include a non-volatile memory for long-term data storage and a volatile memory used as a device memory for the circuit 210. The memory 230 may exchange data with the circuit 210 via a data bus. There may also be accompanying control lines and an address bus located between the memory 230 and the circuit 210.

[0063] The functionality of system 200 may be implemented in the form of executable logic routines (e.g., lines of code, software programs, etc.) stored on a non-transitory computer-readable medium (e.g., memory 230) of system 200 and executed by circuit 210 (e.g., using processor 212). In addition, the functionality of system 200 may be a stand-alone software application or form part of a software application that performs additional tasks associated with system 200. The described functionality may be considered as a method that a processing unit (e.g., processor 212 of circuit 210) is configured to perform. In addition, although the described functionality may be implemented in software, such functionality may also be performed via dedicated hardware or firmware, or some combination of hardware, firmware and / or software.

[0064] The circuit 210 is configured to perform a detection function 231, a providing function 233 and an adding function 234. The circuit 210 may optionally be configured to further perform a generation function 232 and a transmission function 235.

[0065] The detection function 231 is configured to detect an event related to at least one type of a change in an operation state of the transmitter 310 of the encoded image frame stream and a change in a connection state between the transmitter 310 of the encoded image frame stream and the receiver 320 .

[0066] The providing function 233 is configured to provide an additional image frame or a sequence of additional image frames depending on the type of event detected by the detection function 231, wherein the one additional image frame or each additional image frame of the sequence of additional image frames includes a reference to a previous image frame and encodes a predetermined visual change relative to the previous image frame, wherein the predetermined visual change indicates the type of detection event detected by the detection function 231.

[0067] The adding function 234 is configured to add an additional image frame or a sequence of additional image frames to the stream of encoded image frames to obtain a combined image frame stream.

[0068] The generation function 232 is configured to generate each of the sequence of additional image frames and one additional image frame based solely on a predetermined motion vector and / or predetermined frequency domain coefficients.

[0069] The transmission function 235 is configured to transmit the combined image frame stream to the decoder.

[0070] It should be noted that the system 200 and the components it includes are generally not implemented in a single location. For example, the circuit 210 and the memory 230 may be distributed between the transmitter 310 and the receiver 320 or replicated in the transmitter 310 and the receiver 320. Generally, an event of a type identified in the transmitter 310 will be processed by a portion of the circuit 210 and functions 231 to 235 in the memory 230 located in the transmitter 310 or by one of the replicated circuits 210 and functions 231 to 235 in the memory 230 located in the transmitter 310. Accordingly, an event of a type identified in the receiver 320 will generally be processed by a portion of the circuit 210 and functions 231 to 235 in the memory 230 located in the receiver 320 or by one of the replicated circuits 210 and functions 231 to 235 in the memory 230 located in the receiver 320.

[0071] The system 200 may further include a decoder 240. The decoder 240 is configured to decode the combined image frame stream. The decoder 240 is located at the receiver 320.

[0072] The system 200 may further include a display 250. The display 250 is configured to display the combined image frame stream decoded by the decoder 240. Thereby, a predetermined visual change indicating a detected event is displayed on the display 250. The display 250 is located at the location of the receiver 320.

[0073] The functions performed by circuit 210 may be described in detail with respect to Figure 1 and Figure 3 The corresponding steps of the described method embodiment are further modified in the same manner.

[0074] Those skilled in the art will recognize that the present invention is not limited to the above-described embodiments. On the contrary, many modifications and variations are possible within the scope of the appended claims. By studying the drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement these modifications and variations when practicing the claimed invention.

Claims

1. A method for adding an additional image frame or a sequence of additional image frames to a stream of coded image frames, wherein: The coded image frame stream is composed of image frames encoded by an encoder, and the method comprises: Detects events related to one of the following types: a change in the operational state of the transmitter of the coded image frame stream; and a change in the connection state between the transmitter and receiver of the coded image frame stream; providing said one additional image frame or said sequence of additional image frames depending on said type of detected event, wherein each additional image frame of said one additional image frame or said sequence of additional image frames comprises a reference to a previous image frame and encodes a predetermined visual change relative to said previous image frame, wherein said predetermined visual change indicates said type of detected event; and The one additional image frame or the sequence of additional image frames is added to the stream of encoded image frames to obtain a combined image frame stream to be decoded by a decoder and displayed on a display showing the predetermined visual change indicative of the detected event.

2. The method according to claim 1, wherein: The one additional image frame or the sequence of additional image frames each further comprises at least one of the following: one or more inter blocks with motion vectors and / or quantized frequency domain coefficients, and One or more intra blocks with quantized frequency domain coefficients.

3. The method according to claim 1, wherein: The previous image frame is at least one of: A coded image frame in the coded image frame stream; an added starting picture frame for intra-coding the starting picture; and The additional image frame in the sequence of additional image frames.

4. The method according to claim 3, wherein: The previous image frame is an immediately previous image frame in the combined image frame stream.

5. The method according to claim 1, further comprising: transmitting said combined image frame stream to a decoder located at said receiver of said stream; Decoding the combined image frame stream; as well as The decoded combined image frame stream is displayed on a display located at the location of the receiver of the stream, thereby displaying the predetermined visual change indicative of the detected event.

6. The method according to claim 1, wherein: The detected events are related to at least one of the following types: closing of said transmitter of said coded image frame stream, a connection failure between the transmitter and the receiver of the coded image frame stream, a failure of said transmitter of said coded image frame stream, starting of said transmitter of said coded image frame stream, reconnection between the transmitter and the receiver of the coded image frame stream, and Restart of the transmitter of the coded image frame stream.

7. The method according to claim 1, wherein: The predetermined visual change is one of motion, color change and brightness change.

8. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions, when executed in a device having at least one processor, for implementing the method according to claim 1.

9. A system for adding an additional image frame or a sequence of additional image frames to a stream of coded image frames, wherein: The encoded image frame stream is composed of image frames encoded by an encoder, and the system includes circuits configured to perform the following operations: A detection function configured to detect events related to one of the following types: a change in the operational state of the transmitter of the coded image frame stream; and a change in the connection state between the transmitter and receiver of the coded image frame stream; providing a function configured to provide said one additional image frame or said sequence of additional image frames, depending on said type of said event detected by said detection function, wherein each additional image frame of said one additional image frame or said sequence of additional image frames comprises a reference to a previous image frame and encodes a predetermined visual change relative to said previous image frame, wherein said predetermined visual change indicates said type of detected event; and an adding function configured to add said one additional image frame or said sequence of additional image frames to said stream of encoded image frames to obtain a combined image frame stream to be decoded by a decoder and displayed on a display, thereby showing said predetermined visual change indicative of the detected event.

10. The system according to claim 9, wherein: The circuit is further configured to perform: a transmission function configured to transmit said combined image frame stream to a decoder located at said receiver of said stream, Wherein, the system further comprises: The decoder is configured to decode the combined image frame stream; and A display located at the location of the receiver of the stream is configured to display the decoded combined image frame stream on the display so as to display the predetermined visual change indicative of the detected event.

11. The system according to claim 9, wherein: The one additional image frame or the sequence of additional image frames each further comprises at least one of the following: one or more inter blocks with motion vectors and / or quantized frequency domain coefficients, and One or more intra blocks with quantized frequency domain coefficients.

Images