Method and image-capturing device for endoding image frames of an image stream and transmitting encoded image frames on a communications network

Abstract

A method performed by an image-capture device for encoding image frames of an image stream and transmitting the encoded image frames over a communication network is: Step (501) of receiving an image frame of an image stream from an image sensor of an image-capture device; A step (502) of dividing an image frame into a plurality of slices, wherein the plurality of slices include one or more first slices defined by first slice parameters and one or more second slices defined by second slice parameters; A step (503) of prioritizing a plurality of slices such that one or more first slices are prioritized over one or more second slices; A step (504) of generating a first encoded image frame including one or more first encoded slices defined by first slice parameters and based on the encoding of one or more first slices, and one or more second encoded slices defined by second slice parameters and including one or more first skip blocks; A step (504) of generating a second encoded image frame comprising one or more additional second encoded slices defined by second slice parameters and based on the encoding of one or more second slices, and one or more additional first encoded slices defined by first slice parameters and including one or more second skip blocks; and It includes the step (505) of transmitting a first encoded image frame to a receiving device before a second encoded image frame on a communication network.

Description

Technology Field

[0001] Embodiments of the present invention relate to a method for transmitting image frames over a communication network and an image capture device. A corresponding computer program and a computer program carrier are also disclosed. Background Technology

[0002] Public surveillance using imaging, particularly video imaging, is common in many areas around the world. Areas that may require monitoring include banks, shops, schools, and other security-sensitive facilities such as government buildings. Other areas where monitoring may be necessary are processing, manufacturing, and logistics applications where video surveillance is primarily used to monitor processes.

[0003] Video is typically streamed over communication networks to access it remotely. To stream video over a communication network, video encoding is generally required. Video encoding is the process of preparing video for output, for instance, and digital video is encoded using video encoder software to meet the appropriate formats and specifications for recording and playback. In other words, video encoding is the process of converting RAW images into compressed video files to store them as fluid videos rather than individual images. Encoding can also be used as a comprehensive term to describe similar processes related to changing the size of video files. Video encoding provides a method to compress image files without compromising video quality. Both video encoding and encoding often refer to the same process.

[0004] In the field of video compression, video frames can be compressed using different algorithms that have different advantages and disadvantages, primarily based on the amount of data compressed. These different algorithms for video frames can be referred to as picture types or frame types. For example, there may be three different picture types used in different video algorithms, also referred to as prediction types: I, P, and B. They differ in the following characteristics:

[0005] I-frames have the lowest compression ratio but do not require other video frames to decode.

[0006] P-frames can be decompressed using data from previous frames and are more compressible than I-frames.

[0007] B-frames can achieve the greatest amount of data compression by using both previous and forward frames for data reference.

[0008] In the H.264 / MPEG-4 AVC standard, the granularity of prediction types is reduced to the slice level. A slice is a spatially distinct region of a frame that is encoded separately from any other region of the same frame. I-slices, P-slices, and B-slices replace I, P, and B frames.

[0009] However, since streaming video over a communication network always carries the risk of packet loss, video frames or parts of video frames may be lost during streaming. The impact of such lost packets is that parts of the video may be corrupted by errors. As objects in video images can move, errors can also be perceived as moving or interfering with the images. Examples of error effects include trails or debris left behind moving objects, backgrounds near objects that may move along with them, inaccurate or nonsensical colors, and melting images. Some players may refuse to play something that contains errors or something that relies on a previous part that contains errors.

[0010] Video coding according to standards such as H.264 can specify that slices must be encoded and transmitted in a specific order, for example, starting from the top-left corner of the image frame input to the encoder and ending at the bottom-right corner. Each slice can be placed in its own Network Abstraction Layer (NAL) unit, and each NAL unit can then be packetized into a series of Internet Protocol (IP) packets transmitted over a communication network. Statistically, the first packets are more likely to reach the receiving device over the communication network than the later packets. If the later packets consist of slices with more relevant content than the content of the slices in the earlier packets, the risk of losing high-priority content is greater than the risk of losing low-priority content. This may be the case, for example, with slices from surveillance cameras, where the most interesting part of the scene is typically directed to be depicted in the middle of the video frame, but the standard specifies that slices must be encoded and transmitted in a specific order starting from the top-left corner of the image frame input to the encoder and ending at the bottom-right corner.

[0011] Video coding according to the baseline profile of the H.264 standard includes a solution to this problem, namely flexible macroblock ordering (FMO). FMO means that it is possible to freely choose which macroblock to transmit first (and which is most likely to be successfully transmitted). Document US 2003 112 867 discloses a method for encoding and transmitting video data over a communication channel, wherein a video frame may be sliced ​​into a first tile group (slice1) covering a central portion of the video frame corresponding to a region of interest (ROI), and the area surrounding the ROI is sliced ​​into a second tile group (slice2) containing non-ROI data, and the ROI is encoded and placed in the bitstream before the non-ROI data.

[0012] US 2003 112 867 discloses a method for encoding and transmitting video frames, wherein a video frame is determined to include a region of higher relevance and slicing the video frame and grouping the slices into subgroups, wherein a first group corresponds to a region of higher relevance in the video frame and a second group includes a background, and the first group is prioritized during transmission over a wireless network.

[0013] US 2014 036 999 initiates slice group priority assignment, and higher priority slice groups may be the central part of the video frame and higher priority slice groups are transmitted first (QoS).

[0014] However, FMO-based solutions are generally not supported because they are technically complex.

[0015] Accordingly, the purpose of the embodiments of the present invention may be to eliminate some of the problems mentioned above or at least reduce their impact. In particular, the purpose may be to improve the transmission of packets through a communication network.

[0016] As mentioned above, if packet loss occurs, it may be desirable to be able to prioritize which parts of video frames or packages are lost, at least at a statistical level. In the example, frames are transmitted as bundles of packets. The first packets may have a higher probability of reaching the receiving device through the communication network than later packets. Therefore, later parts of the frame may be lost in the communication network. This is not optimal because surveillance cameras are typically directed to display the most interesting part of the scene in the middle of the video frame.

[0017] Embodiments of the present invention disclose a solution compatible with currently used standards, namely, a solution compatible with transmitting frame slices in a predefined order and reducing the probability of losing slices having higher priority content.

[0018] According to one embodiment, the objective is achieved by a method performed by an image capture device to encode image frames of an image stream and to transmit the encoded image frames over a communication network.

[0019] The method includes the step of receiving an image frame of an image stream from an image sensor of an image capture device.

[0020] The method further includes the step of dividing an image frame into a plurality of slices, wherein the plurality of slices includes one or more first slices defined by first slice parameters and one or more second slices defined by second slice parameters.

[0021] The method further includes the step of prioritizing a plurality of slices such that one or more first slices are prioritized over one or more second slices.

[0022] The method further includes the step of generating a first encoded image frame comprising one or more first encoded slices defined by first slice parameters and based on the encoding of one or more first slices, and one or more second encoded slices defined by second slice parameters and including one or more first skip blocks.

[0023] The method further includes the step of generating a second encoded image frame comprising one or more additional second encoded slices defined by second slice parameters and based on the encoding of one or more second slices, and one or more additional first encoded slices defined by first slice parameters and including one or more second skip blocks.

[0024] The method further includes the step of transmitting a first encoded image frame to a receiving device on a communication network prior to a second encoded image frame.

[0025] For each encoded image frame, the encoded slices can be transmitted in a first order, for example, as specified in a video compression standard.

[0026] According to another embodiment, the objective is achieved by an image capture device configured to perform the above method.

[0027] According to additional embodiments, the objective is achieved by a computer program and a computer program carrier corresponding to the above embodiments.

[0028] For each encoded image frame, the encoded slices may be transmitted in a first (general) order as specified by the video compression standard to satisfy the requirements for the transmission of the slices, and the priority of the original slices is performed at the frame level by integrating slices of different priority into different encoded frames associated with the same input frame received from the image sensor and transmitting the encoded frames associated with the same input frame according to a second order. The second order may be based on the priority of the content of the slices of each input frame.

[0029] Because the first encoded image frame is transmitted to the receiving device of the communication network before the second encoded image frame according to the second order, for example, the probability of loss of the prioritized slice is lower.

[0030] An additional advantage of the embodiments herein is that when the slices are encoded such that the encoded slices are independent of other encoded slices arising from the same received frame, e.g., the same raw input frame, errors caused by lost communication packets, such as IP packets, and thus the lost slices are not propagated to other slices. Brief explanation of the drawing

[0031] In the drawings, features appearing in some embodiments are indicated by dashed lines. Various aspects of the embodiments disclosed herein, including specific features and advantages, will be readily understood from the following detailed description and the accompanying drawings. FIG. 1 illustrates exemplary embodiments of image capture devices. FIG. 2a illustrates exemplary embodiments of a video network system. FIG. 2b illustrates exemplary embodiments of a video network system and a user device. FIG. 3a is a schematic block diagram illustrating an exemplary embodiment of an imaging system. FIG. 3b is a schematic block diagram illustrating a reference method in an image capture device. FIG. 4a is a schematic block diagram illustrating embodiments of a method in an image capture device. FIG. 4b is a schematic block diagram illustrating embodiments of a method in an image capture device. FIG. 4c is a schematic block diagram illustrating embodiments of a method in an image capture device. FIG. 4d is a schematic block diagram illustrating embodiments of a method in an image capture device. FIG. 4e is a schematic block diagram illustrating embodiments of a method in an image capture device. FIG. 5 is a flowchart illustrating embodiments of a method in an image capture device. FIG. 6 is a block diagram illustrating embodiments of an image capture device. Specific details for implementing the invention

[0032] Embodiments of the present invention may be implemented in one or more image capture devices, for example, a digital camera. FIG. 1 depicts various exemplary image capture devices (110). An image capture device (110) may be any of, for example, a video camera (120) such as a camcorder, a network video recorder, a camera, a surveillance camera or a monitoring camera, a digital camera, a wireless communication device (130) such as a smartphone including an image sensor, or a vehicle (140) including an image sensor.

[0033] FIG. 2a depicts an exemplary video network system (250) in which embodiments of the present invention may be implemented. The video network system (250) may include an image capture device, such as a video camera (120), capable of capturing a digital image (201), such as a digital video image, and performing image processing. The video server (260) of FIG. 2a may acquire an image from the video camera (120), for example, via a network, which is indicated by double arrows in FIG. 2a.

[0034] A video server (260) is a computer-based device dedicated to video delivery. Video servers are used in many applications and often have additional functions and capabilities to handle the requirements of specific applications. For example, video servers used in security, surveillance, and inspection applications are typically designed to capture video from one or more cameras and deliver the video over a computer network. In video generation and broadcast applications, a video server may have the ability to record video, play back recorded video, and deliver many video streams simultaneously. Today, many video server functions can be built into a video camera (120).

[0035] However, in FIG. 2a, the video server (260) is connected to an image capture device (110), exemplified by a video camera (120), via a video network system (250). The video server (260) may be further connected to a video storage (270) for storing video images and / or to a monitor (280) for displaying video images. In some embodiments, the video camera (120) is directly connected to the video storage (270) and / or the monitor (280), as indicated by the direct arrow between these devices in FIG. 2a. In some other embodiments, the video camera (120) is connected to the video storage (270) and / or the monitor (280) via the video server (260), as indicated by the arrows between the video server (260) and other devices.

[0036] FIG. 2b depicts a user device (295) connected to a video camera (120) via a video network system (250). The user device (295) may be, for example, a computer or a mobile phone. The user device (295) may, for example, control the video camera (120) and / or display video coming from the video camera (120). The user device (295) may further include both the functions of a monitor (280) and a video storage (270).

[0037] To better understand the embodiments of the present invention, an imaging system will be described first.

[0038] FIG. 3a is a schematic diagram of an imaging system (300) in the case of such a digital video camera, such as a video camera (120). The imaging system images a scene in an image sensor (301). The image sensor (301) may be provided with a Bayer filter so that different pixels receive radiation in a specific wavelength range in a known pattern. Generally, each pixel of the captured image is represented by one or more values ​​indicating the intensity of the light captured within a given wavelength band. These values ​​are generally referred to as color components, or color channels. The term "image" may refer to an image frame or video frame containing information originating from an image sensor that has captured the image.

[0039] After reading the signals of individual sensor pixels of the image sensors (301), different image processing operations can be performed by the image processing pipeline (302). The image processing pipeline (302) may include an image processing section (302a) and a video post-processing section (302b).

[0040] Generally, for video processing, images are composed of a stream of images. FIG. 3a illustrates a first video stream (310) from an image sensor (301). The first image stream (310) may include a number of captured image frames, such as a first captured image frame (311) and a second captured image frame (312).

[0041] Image processing may include demosaicing, color correction, noise filtering (to remove spatial and / or temporal noise), distortion correction (e.g., to remove the effects of barrel distortion), global and / or local tone mapping (e.g., to enable imaging of scenes containing various intensities), transformation (e.g., correction and rotation), planar field correction (e.g., to remove vignetting effects), application of overlays (e.g., privacy masks, descriptive text), etc. The image processing pipeline (302) may also be associated with an analysis engine that performs object detection, recognition, alerts, etc.

[0042] The image processing section (302a) may, for example, perform image stabilization and apply noise filtering, distortion correction, global and / or local tone mapping, transformation, and planar field correction. The video post-processing section (302b) may, for example, crop parts of the image, apply overlays, and include an analysis engine.

[0043] Following the image processing pipeline (302), the image can be forwarded to an encoder (303), where the information of the image frames is coded according to an encoding protocol such as H.264. The encoded image frames are then forwarded to a receiving client, a video server (260), a storage (270), etc., exemplified here as a monitor (280).

[0044] The video coding process generates multiple values ​​that must be encoded to form a compressed bitstream. These values ​​may include the following:

[0045] · Quantized transformation coefficients,

[0046] · Information that enables the decoder to regenerate predictions,

[0047] · Information regarding the structure of the compressed data and the compression tools used during encoding, and

[0048] · Information regarding the entire video sequence.

[0049] These values ​​and parameters (syntactic elements) are converted into binary codes using, for example, variable-length coding and / or arithmetic coding. Each of these encoding methods also produces an efficient and concise binary representation of information referred to as an encoded bitstream. Subsequently, the encoded bitstream can be stored and / or transmitted.

[0050] As mentioned above, there may be interest in improving the encoding of video images in video streams and the transmission of video streams through communication networks to reduce the impact of data packets lost through communication networks.

[0051] To better understand the embodiments of the present invention, a reference method for encoding and transmitting image frames will first be presented with reference to FIG. 3b.

[0052] As described above, the first captured image frame (311) is encoded by the encoder (303) by dividing the first captured image frame (311) into a plurality of parts referred to as slices.

[0053] In FIG. 3b, the first captured image frame (311) is divided into two slices, namely the first slice (311a) and the second slice (311b). Other image frames, such as the second captured image frame (312), are also divided into slices. Thus, the second captured image frame (312) is divided into two slices, namely the first slice (312a) and the second slice (312b).

[0054] According to standard encoding techniques, slices can be encoded and transmitted in a specific order, for example, starting from the top-left corner of an image frame input to an encoder (303) and ending at the bottom-right corner. Each slice can be placed in its own Network Abstraction Layer (NAL) unit, and then each NAL unit can be packetized into a series of Internet Protocol (IP) packets transmitted through a communication network such as a video network system (250).

[0055] However, data packets lost in the communication network can be a problem. The first parts of an encoded image frame may be more likely to reach the receiving device through the network than the later parts. Therefore, the later parts of the encoded image frame may be lost in the communication network. This is not optimal because surveillance cameras are generally directed to display the most interesting part of the scene in the middle of the video frame.

[0056] One way to solve this problem is to provide an image capture device configured to slice (or tile) an input image into multiple parts, arrange each part (or group of parts) in order of priority, and move to a separate frame in which empty areas are replaced by pre-generated empty slices. Thus, for each encoded frame, the slices may be transmitted in a first (e.g., general) order or according to a set of first priorities, such as those specified by video compression standards, to satisfy the requirements for the transmission of the slices, whereas the prioritization of the original slices is performed at the frame level by integrating slices that are differently prioritized into different encoded frames associated with the same input frame received from the image sensor, and transmitting the encoded frame associated with the same input frame in a second order. The second order may be based on the priority of the content of the slices of each input frame. The second order may differ from the first order. The second order may also be referred to as a set of second priorities. Thus, the first and second encoded frames associated with the same input frame are transmitted in a second order, for example, according to a set of second priorities. For example, based on the content of the input slices, the encoded slices are distributed into a set of encoded frames associated with the same input frame so that the encoded slice with the highest priority according to the content is transmitted first. Within each encoded frame associated with the same input frame, the slices are transmitted according to the first priority, for example, according to standard specifications.

[0057] Subsequently, individual frames can be combined into a single frame at the decoder side, just as regular video frames are combined as a result at a specific time. An empty slice functions like an empty frame and can be generated like an empty frame. An empty frame may be a frame that has no difference encoded with respect to the previous frame of the encoded image stream. In some embodiments, an empty frame may be a pre-calculated encoded frame. A pre-calculated encoded frame means that instead of generating an encoded frame with the encoder (303), the encoded frame is pre-generated by a software algorithm, for example, by encoding all empty frames in the same way. Thus, an empty frame may be a pre-calculated encoded frame that has no difference encoded with respect to the previous frame of the encoded image stream. For example, such a pre-calculated empty frame may contain one or more macroblocks indicating no change with respect to the previous frame. Such change may relate to the image content of the previous frame, such as pixel values, for example. Thus, an empty slice may be a slice that has no difference encoded with respect to the previous slice of the encoded image stream.

[0058] For example, in a slice header according to the H.264 / H.265 standard, the normal QP value per slice, or the base QP value per frame, can be set to QP 51. For empty frames, for example, frames without encoded block residuals, the QP is not used by the decoder, and if QP 51 is never used by the encoder (303), a “flag” indicating an empty frame can be used. The advantage of using QP values ​​is that no arbitrary additional processing is required because the QP values ​​are part of the video that is already ready to be used.

[0059] In a group of frames generated from the same single frame, all frames except the last one can be marked as not displayed to avoid strange visual effects, such as partial image updates. Additionally, motion vector search can be restricted to within each individual slice / tile.

[0060] Exemplary embodiments of the present invention will now be described with reference to FIGS. 4a, 4b, 4c, 4d, 4e and 5 and additionally FIGS. 1, 2a, 2b and 3a.

[0061] FIG. 4a illustrates embodiments of a method for encoding in an image capture device (110). In particular, the embodiments may be performed by an encoder (303). An image frame (410) may be divided into a plurality of slices (410a, 410b, 410c), and the plurality of slices (410a, 410b, 410c) include one or more first slices (410b) defined by first slice parameters and one or more second slices (410a, 410c) defined by second slice parameters. In FIG. 4a, one or more first slices (410b) are illustrated with a different filling pattern from one or more second slices (410a, 410c). In the example of FIG. 4a, one or more first slices (410b) are a single slice, whereas one or more second slices (410a, 410c) include two slices.

[0062] FIG. 5 illustrates a flowchart describing a method performed by an image capture device (110) to encode image frames (311, 410) of an image stream (310) and to transmit the encoded image frames (411, 412) in a communication network such as a video network system (250).

[0063] Operation 501

[0064] The image capture device (110) receives captured image frames (311, 410) of an image stream (310) from an image sensor (301) of the image capture device (110). For example, an encoder (303) can receive captured image frames (311, 410) of an image stream (310) from an image sensor (301). The image capture device (110) can receive the entire raw frame.

[0065] Operation 502

[0066] An image capture device (110) divides an image frame (311, 410) into a plurality of slices (410a, 410b, 410c). The plurality of slices (410a, 410b, 410c) include one or more first slices (410b) defined by first slice parameters and one or more second slices (410a, 410c) defined by second slice parameters. The slice parameters may include the size and position of the slices within the frame. The plurality of slices (410a, 410b, 410c) may be of the same size.

[0067] In one embodiment, the input image (410) is divided and encoded into three slices of the same size. For example, the plurality of slices (410a, 410b, 410c) may further include one or more third slices (410c) defined by third slice parameters.

[0068] Image partitioning can be dynamic to better cover the most important parts of a video that may change. For example, the number, size, and position of slices can be changed dynamically.

[0069] Operation 502 can be performed by an encoder (303).

[0070] Operation 503

[0071] The image capture device (110) is prioritized for a plurality of slices (410a, 410b, 410c) such that one or more first slices (410b) are prioritized over one or more second slices (410a, 410c).

[0072] Prioritization can be based on the contents of multiple slices (410a, 410b, 410c). For example, one or more slices can be grouped into the same priority group.

[0073] For example, one or more second slices (410a, 410c) may be grouped into a second priority group having a lower priority than one or more first slices (410b) in the first priority group.

[0074] In some embodiments of the present invention in which a raw frame is divided and encoded into three slices, the middle slice has the highest priority, while the other two slices have lower priority. Accordingly, in these embodiments, the middle slice may be included in one or more first slices (410b) which are prioritized over the other two slices included in one or more second slices (410a, 410c).

[0075] In some other embodiments of the present invention, prioritizing a plurality of slices (410a, 410b, 410c) includes prioritizing one or more second slices (410a) over one or more third slices (410c).

[0076] Operation 504

[0077] FIG. 4b illustrates first and second encoded image frames (411, 412).

[0078] The image capture device (110) generates a first encoded image frame (411) comprising one or more first encoded slices (411b) defined by first slice parameters and based on the encoding of one or more first slices (410b), and one or more second encoded slices (411a, 411c) defined by second slice parameters and including one or more first skip blocks.

[0079] Skip blocks may indicate that there is no difference in encoding between the previous encoded image frames (411, 412) and the part of the encoded image frames (421, 422) covered by the skip blocks.

[0080] The image capture device (110) further generates a second encoded image frame (412) comprising one or more additional second encoded slices (412a, 412c) defined by second slice parameters and based on the encoding of one or more second slices (410a, 410c), and one or more additional first encoded slices (412b) defined by first slice parameters and including one or more skip blocks.

[0081] Generating encoded image frames (411, 412) may include encoding multiple slices (410a, 410b, 410c) such that each encoded slice (410a, 410b, 410c) of multiple encoded slices (411a, 411b, 411c) is independent of all other encoded slices (411a, 411b, 411c) of multiple encoded slices (411a, 411b, 411c).

[0082] Accordingly, each encoded slice (411a, 411b, 411c) may be independent of other encoded slices (411a, 411b, 411c) arising from the same received input image frame (410). For example, in some embodiments of the present invention, one or more first encoded slices (411b) do not depend on one or more second encoded slices (411a, 411c).

[0083] However, a slice of an encoded image frame associated with a different input image frame, for example, an earlier or later input image frame, may depend on the earlier or later slice. For example, if an object moves across the slices, for one or more second encoded slices (411a, 411c), the encoder cannot use motion vectors pointing to one or more first encoded slices (411b). Instead, the moving object must be painted with I-blocks when entering one or more second encoded slices (411a, 411c). This is more expensive in terms of computing and transmission resources but limits dependency. As soon as the object is within one or more second encoded slices (411a, 411c), for example, motion vectors can be used, and thus one or more second encoded slices (411a, 411c) depend on the corresponding encoded slice associated with the earlier or later image frame, such as the earlier or later input frame or the earlier or later encoded frame. In embodiments of the present invention, a previous or subsequent encoded image frame refers to a set of encoded image frames associated with the same received input frame.

[0084] For example, motion vector search can be restricted to within each slice only. This means there is no prediction for slice boundaries. This will prevent errors from propagating between encoded slices.

[0085] The generation of encoded frames will be exemplified by the following two approaches.

[0086] 1. The encoder (303) performs two passes of the input image frames (311, 410), and the input image frames (311, 410) generate an encoded output frame (411, 412) having multiple, for example, three encoded slices, and is requested per slice to generate actual slice data or fill with skip blocks.

[0087] A first approach is exemplified in FIG. 4c, where the input frame (410) is encoded twice: the first is encoded into the first encoded frame (411), and the second is encoded into the second encoded frame (412).

[0088] 2. The frame is encoded into multiple, for example, two slices, in an intermediate encoded image frame (415). A second approach is illustrated in FIG. 4d. The intermediate encoded image frame (415) may include a plurality of encoded slices, such as one or more first encoded slices (415b) of the intermediate encoded image frame (415) and one or more second encoded slices (415a, 415c) of the intermediate encoded image frame (415).

[0089] Subsequently, a plurality of, for example, two output frames are constructed by selecting encoded slice data for some slices and filling the other slices with skip blocks. Thus, the intermediate encoded image frame (415) can be used to generate the first encoded image frame (411) and the second encoded image frame (412).

[0090] Accordingly, in some embodiments of the present invention, generating a first encoded image frame (411) comprises encoding a plurality of slices (410a, 410b, 410c) such that one or more first slices (410b) are encoded into one or more first encoded slices (411b) of the first encoded image frame (411) and one or more second encoded slices (411a, 411c) of the first encoded image frame (411) are generated by replacing one or more second slices (410a, 410c) with second skip blocks. Subsequently, generating a second encoded image frame (412) includes encoding a plurality of slices (410a, 410b, 410c) such that one or more second slices (410a, 410c) are encoded into one or more additional second encoded slices (412a, 412c) of the second encoded image frame (412), and one or more additional first encoded slices (411a, 411c) are generated by replacing one or more first slices (410b) with one or more first skip blocks.

[0091] In some other embodiments, generating a first encoded image frame (411) and generating a second encoded image frame (412) includes generating an intermediate encoded image frame (415) based on encoding a plurality of slices (410a, 410b, 410c) such that one or more first slices (410b) are encoded into one or more first encoded slices (415b) of an intermediate encoded image frame (415) and one or more second slices (410a, 410c) are encoded into one or more second encoded slices (415a, 415c) of an intermediate encoded image frame (415). Subsequently, generating a first encoded image frame (411) further comprises inserting one or more first encoded slices (415b) of an intermediate encoded image frame (415) into the first encoded image frame (411) and inserting skip blocks instead of one or more second encoded slices (415a, 415c) of the intermediate encoded image frame (415), wherein generating a second encoded image frame (412) further comprises inserting one or more second encoded slices (415a, 415c) of the intermediate encoded image frame (415) into the second encoded image frame (412) and inserting skip blocks instead of one or more first encoded slices (411b) of the intermediate encoded image frame (415) into the second encoded image frame (412).

[0092] FIG. 4e illustrates some embodiments in which a plurality of slices (410a, 410b, 410c) include one or more third slices (410c) defined by third slice parameters. Subsequently, the generated first encoded image frame (411) may further include one or more third encoded slices (411c) defined by third slice parameters and including one or more third skip blocks, and the generated second encoded image frame (412) may further include one or more third encoded slices (411c).

[0093] These embodiments may further include generating a third encoded image frame (413) comprising one or more additional third encoded slices (413c) defined by third slice parameters and based on the encoding of one or more third slices (410c), one or more additional first encoded slices (412b) defined by first slice parameters and including one or more second skip blocks, and one or more second encoded slices (411a) defined by second slice parameters and including one or more first skip blocks.

[0094] Operation 505

[0095] To satisfy the requirement of transmitting slices of an encoded frame in a specific order, which may not be based on the priority of the content of the slices, the image capture device (110) transmits the encoded slices of each encoded image frame (411, 412) according to a first set of priorities and transmits the encoded image frames (411, 412) generated from the same received image frame (410) according to a second set of priorities. As mentioned above, the first set of priorities may be based on standard requirements, whereas the second set of priorities may be based on the content of the slices.

[0096] For example, one or more first encoded slices (411b) are based on one or more first slices (410b) that are prioritized over one or more second slices (410a, 410c), for example, based on the content of one or more first and second slices (410a, 410b, 410c). Accordingly, one or more first encoded slices (411b) must be prioritized over one or more additional second encoded slices (412a, 412c). This is done by transmitting one or more first encoded slices (411b) as part of a separate first encoded frame (411) that is prioritized over a second encoded frame (412) containing one or more additional second encoded slices (412a, 412c).

[0097] Accordingly, the image capture device (110) transmits the first encoded image frame (411) to the receiving device (260, 270, 280, 295) before the second encoded image frame (412) on the communication network (250). Accordingly, since the image capture device (110) transmits the first encoded image frame (411) before the second encoded image frame (412), the image capture device (110) prioritizes the first encoded image frame (411) over the second encoded image frame (412).

[0098] The respective horizontal arrows in Fig. 4b and Fig. 4e illustrate the transmission time.

[0099] FIG. 4b further illustrates a method in which an image capture device (110) transmits an additional first encoded image frame (421) before an additional second encoded image frame (422) according to the same principles described above for the first encoded image frame (411) and the second encoded image frame (412). The time interval between the set of the first and second encoded image frames (411, 412) and the second set of additional first and second encoded image frames (421, 422) can be set according to a specific desired frame rate, e.g., 1 / 30 second.

[0100] FIG. 4e further illustrates a method in which an image capture device (110) transmits an additional second encoded image frame (422) prior to an additional third encoded image frame (423) according to the same principles described above.

[0101] The image capture device (110) can transmit a first indication that the first encoded image frame (411) is not displayed on the receiving device (260, 270, 280, 295).

[0102] In some embodiments of the present invention, the first indication is transmitted in the header of the first encoded image frame (411).

[0103] Generally, all transmitted encoded frames, except for the last transmitted encoded frame, may include, for example, an indication in the header of the encoded frame that the encoded frame is not displayed.

[0104] In some other embodiments of the present invention, the first encoded frame (411) includes a first indication that only the last encoded frame (412) among the encoded frames associated with the first encoded frame (411) will be displayed.

[0105] The indication can be implemented as a flag such as a 1-bit flag. By transmitting the first indication, the image capture device (110) helps the receiving device (295) understand how to display the contents of the encoded frames.

[0106] In some embodiments of the present invention, the image capture device (110) transmits a second indication to a receiving device (260, 270, 280, 295) indicating how an image frame (311, 410) is divided into a plurality of slices (410a, 410b, 410c). By transmitting the second indication image, the partitioning may be dynamic.

[0107] In addition, the second indication may be transmitted in one or more headers.

[0108] When there are three encoded image frames, the method may further include transmitting the second encoded image frame (412) to the receiving device (260, 270, 280, 295) before the third encoded image frame (413) on the communication network (250).

[0109] At the receiver side, the received encoded frames are decoded normally. Subsequently, the decoded frames can be combined into a single frame to be displayed. For example, if the decoding of the second image consists of skip blocks, blocks from the first decoded image can be reused. That is, the decoded data from the first decoded image frame can be copied to the second decoded image frame and displayed. Thus, one or more decoded frames may not be displayed. In particular, in some embodiments, only the result after the last frame containing all slices will be displayed. This can be indicated, for example, by the first indication.

[0110] If the second encoded frame is lost in the network and the receiving device (260, 270, 280, 295) knows that the first encoded frame is not displayed, for example, if it includes a first indicator, the receiving device (260, 270, 280, 295) will correctly display the next set of decoded frames anyway.

[0111] A complete embodiment will now be schematically presented:

[0112] · The input image is encoded into three slices of the same size.

[0113] · The slices are separated into three individual frames, and the empty areas are replaced with pre-generated empty slices (e.g., indicated by QP-value = 51).

[0114] · Frames are arranged so that the frame with the highest priority slice is placed first.

[0115] · The two first frames are indicated as not displayed.

[0116] · Each frame is transmitted as one or more network packets. For example, each slice can result in a packet. Losses are most likely to affect Frame #2 or #3.

[0117] · If the frames are decoded normally, only the result after the last frame containing all data slices will be displayed.

[0118] Accordingly, according to embodiments of the present invention, a plurality of encoded image frames may be generated from a single input frame. The number of encoded image frames may correspond to or be equal to the number of priority groups for the slices constituting the frames. Each encoded image frame corresponding to a specific priority group may include skip blocks instead of slices generated by an encoder by encoding a corresponding input slice from the corresponding priority group and encoded slices of other priority groups. At the receiver side, the plurality of encoded frames are decoded, and the determined slices are combined and displayed normally.

[0119] Accordingly, some embodiments of the present invention disclose a method in a video camera. The method may include the following:

[0120] - A step of receiving frame data from the image sensor of a video camera;

[0121] - A step of dividing the received frame into multiple slices, such as 2 slices;

[0122] - Step to set priorities for the slices;

[0123] - A step of constructing a first frame from video data of a prioritized slice and skip blocks of non-priority slices, wherein, for example, the prioritized slice is image-coded, i.e., updated with respect to the previous video frame;

[0124] - A step of constructing a second frame from video data of a non-priority slice and skip blocks of a priority slice;

[0125] - A step of transmitting the first and second frames to a receiving device in a communication network.

[0126] Referring to FIG. 6, a schematic block diagram of embodiments of an image capture device (110) is shown. As mentioned above, the image capture device (110) is configured to encode image frames (311, 410) of an image stream (310) and to transmit the encoded image frames (411, 412) over a communication network such as a video network system (250).

[0127] As mentioned above, the image capture device (110) may include any of a camera, a video camera (120), a surveillance camera, a monitoring camera or camcorder, a network video recorder, and a wireless communication device (130).

[0128] In particular, the image-processing device (110) may be a video camera (120), such as a surveillance camera.

[0129] The image-processing device (110) may include an encoder (303) configured to perform one or more of the operations of the method according to FIG. 5. For example, the encoder (303) may be configured to encode an input image frame into at least two encoded image frames based on the method described above in relation to FIG. 5.

[0130] The image capture device (110) may further include a processing module (601), such as means for performing the methods described herein. The means may be implemented in the form of one or more hardware modules and / or one or more software modules.

[0131] The image capture device (110) may further include a memory (602). The memory may include, for example, or store instructions in the form of a computer program (603) that may include computer-readable code units, and when the instructions are executed in the image capture device (110), the image capture device (110) performs a method of encoding image frames (311, 410) of an image stream (310) and transmitting the encoded image frames (411, 412) over a communication network (250).

[0132] According to some embodiments of the present invention, the image capture device (110) and / or processing module (601) may include a processing circuit (604) as an exemplary hardware module that may include one or more processors. Accordingly, the processing module (601) may be implemented in the form of the processing circuit (604) or 'realized thereby'. Instructions may be executable by the processing circuit (604), thereby causing the image capture device (110) to operate to perform the methods of FIG. 5 as described above. As another example, instructions may cause the image capture device (110) to perform the methods according to FIG. 5 when executed by the image capture device (110) and / or processing circuit (604).

[0133] In the above view, in one example, an image capture device (110) is provided for encoding image frames (311, 410) of an image stream (310) and transmitting the encoded image frames (411, 412) over a communication network (250).

[0134] Again, the memory (602) includes commands executable by the processing circuit (604), and thereby the image capture device (110) operates to perform the method according to FIG. 5.

[0135] FIG. 6 further illustrates a carrier (605) or program carrier containing the computer program (603) described immediately above. The carrier (605) may be one of an electronic signal, an optical signal, a wireless signal, and a computer-readable medium.

[0136] In some embodiments, the image capture device (110) and / or processing module (601) may include one or more of a receiving module (610), a sharing module (620), a prioritizing module (630), an encoded frame generation module (640), and a transmission module (650) as examples of hardware modules. In other embodiments, one or more of the aforementioned exemplary hardware modules may be implemented as one or more software modules.

[0137] In particular, the encoder (303) may include one or more or even all of the receiving module (610), the splitting module (620), the prioritization module (630), the encoded frame generation module (640), and the transmission module (650).

[0138] Accordingly, the encoder (303) can perform one or more or even all of the functions of the receiving module (610), the dividing module (620), the prioritization module (630), the encoded frame generation module (640), and the transmission module (650).

[0139] In addition, the processing module (601) may include an input / output unit (606). According to an embodiment, the input / output unit (606) may include an image sensor configured to capture raw image frames, such as original image frames included in the first video stream (310) from the image sensor (301), as described above.

[0140] According to the various embodiments described above, the image capture device (110) and / or processing module (601) and / or receiving module (610) and / or encoder (303) are configured to receive captured image frames (311, 410) of an image stream (310) from an image sensor (301) of the image capture device (110).

[0141] An image capture device (110) and / or a processing module (601) and / or a splitting module (520) and / or an encoder (303) are configured to divide an image frame (311, 410) into a plurality of slices (410a, 410b, 410c). The plurality of slices (410a, 410b, 410c) include one or more first slices (410b) defined by first slice parameters and one or more second slices (410a, 410c) defined by second slice parameters.

[0142] The image capture device (110) and / or processing module (601) and / or prioritization module (630) are configured to prioritize a plurality of slices (410a, 410b, 410c) such that one or more first slices (410b) are prioritized over one or more second slices (410a, 410c).

[0143] The image capture device (110) and / or processing module (601) and / or encoded frame generation module (640) and / or encoder (303) are configured to generate the following:

[0144] A first encoded image frame (411) comprising one or more first encoded slices (411b) defined by first slice parameters and based on the encoding of one or more first slices (410b), and one or more second encoded slices (411a, 411c) defined by second slice parameters and including one or more first skip blocks; and

[0145] A second encoded image frame (412) comprising one or more additional second encoded slices (411a, 411c) defined by second slice parameters and based on the encoding of one or more second slices (410a, 410c), and one or more additional first encoded slices (412b) defined by first slice parameters and including one or more second skip blocks.

[0146] In some embodiments of the present invention, the image capture device (110) and / or processing module (601) and / or encoded frame generation module (640) and / or encoder (303) are configured to perform the following:

[0147] A first encoded image frame (411) is generated by configuring to encode a plurality of slices (410a, 410b, 410c) such that one or more first slices (410b) are encoded into one or more first encoded slices (411b) of a first encoded image frame (411), and one or more second encoded slices (411a, 411c) are generated by replacing one or more second slices (410a, 410c) with second skip blocks;

[0148] A second encoded image frame (412) is generated by configuring to encode a plurality of slices (410a, 410b, 410c) such that one or more second slices (410a, 410c) are encoded into one or more additional second encoded slices (411a, 411c) of the second encoded image frame (411), and one or more additional first encoded slices (411a, 411c) are generated by replacing one or more first slices (410b) with one or more first skip blocks.

[0149] In some embodiments of the present invention, the image capture device (110) and / or processing module (601) and / or encoded frame generation module (640) and / or encoder (303) are further configured to generate a third encoded image frame (413) comprising one or more additional third encoded slices (413c) defined by third slice parameters and based on encoding one or more third slices (410c), one or more additional first encoded slices (412b) defined by first slice parameters and including one or more second skip blocks, and one or more second encoded slices (411a) defined by second slice parameters and including one or more first skip blocks.

[0150] The image capture device (110) and / or processing module (601) and / or transmission module (650) and / or encoder (303) are configured to transmit a first encoded image frame (411) to a receiving device (260, 270, 280, 295) before a second encoded image frame (412) on a communication network (250).

[0151] The image capture device (110) and / or processing module (601) and / or transmission module (650) and / or encoder (303) may be configured to transmit a first indication that the first encoded image frame (411) is not displayed on the receiving device (260, 270, 280, 295).

[0152] In some embodiments of the present invention, the image capture device (110) and / or the processing module (601) and / or the transmission module (650) and / or the encoder (303) are configured to transmit a first indication in the header of a first encoded image frame (411).

[0153] The image capture device (110) and / or processing module (601) and / or transmission module (650) and / or encoder (303) may be configured to transmit a second indication to a receiving device (260, 270, 280, 295) indicating how an image frame (311, 410) is divided into a plurality of slices (410a, 410b, 410c).

[0154] As used herein, the term “module” may refer to one or more functional modules, each of which may be implemented as one or more hardware modules and / or one or more software modules and / or a combined software / hardware module. In some examples, a module may represent a functional unit implemented in software and / or hardware.

[0155] In this document, the terms “computer program carrier,” “program carrier,” or “carrier” may refer to any of an electronic signal, an optical signal, a radio signal, and a computer-readable medium. In some examples, the computer program carrier may exclude transient radio signals such as electronic, optical, and / or radio signals. Accordingly, in these examples, the computer program carrier may be a non-transient carrier, such as a non-transient computer-readable medium.

[0156] As used herein, the term “processing module” may include one or more hardware modules, one or more software modules, or a combination thereof. Any such module, whether hardware, software, or a combined hardware-software module, may be a connected means, a providing means, a configured means, a responding means, a deactivating means, etc., disclosed herein. For example, the expression “means” may be a module corresponding to the modules listed above together with the drawings.

[0157] As used herein, the term "software module" may refer to a software application, a DLL (Dynamic Link Library), a software component, a software object, an object according to the COM (Component Object Model), a software component, a software function, a software engine, an executable binary software file, etc.

[0158] The terms "processing module" or "processing circuit" may include, for example, a processing unit comprising one or more processors, an ASIC (Application Specific Integrated Circuit), a FPGA (Field-Programmable Gate Array), etc. The processing circuit, etc. may include one or more processor kernels.

[0159] As used herein, the expression “configured to” may mean that the processing circuit is adapted to perform or configured to perform one or more of the operations described herein by software configuration and / or hardware configuration.

[0160] As used herein, the term “action” may refer to an action, step, operation, reaction, response, activity, etc. It should be noted that an action of the present invention may be divided into two or more sub-actions where applicable. It should also be noted that, where applicable, two or more actions described herein may be combined into a single action.

[0161] As used herein, the term "memory" may refer to a hard disk, magnetic storage medium, portable computer diskette or disk, flash memory, RAM (Random Access Memory), etc. Additionally, the term "memory" may refer to internal register memory of a processor, etc.

[0162] As used herein, the term "computer-readable medium" may include Universal Serial Bus (USB) memory, DVD discs, Blu-ray discs, software modules received as data streams, flash memory, hard drives, memory cards such as MemorySticks, Multimedia Cards (MMCs), Secure Digital (SD) cards, etc. One or more of the aforementioned examples of computer-readable media may be provided as one or more computer program products.

[0163] As used herein, the term “computer-readable code units” may be the text of a computer program, part or all of a binary file representing a computer program in a compiled format, or anything in between.

[0164] As used herein, the terms “number” and / or “value” may be any kind of number, such as binary, real, imaginary, or rational numbers. Additionally, “number” and / or “value” may be one or more characters, such as a character or a string of characters. “Number” and / or “value” may also be represented by a string of bits, i.e., zeros and / or ones.

[0165] As used herein, the expression “in some embodiments” is used to indicate that the features of the described embodiments may be combined with any other embodiments disclosed herein.

[0166] Although embodiments of various aspects have been described, many different changes, modifications, etc. will be apparent to those skilled in the art. Therefore, the described embodiments are not intended to limit the scope of this disclosure.

Claims

Claim 1 A method performed by an image capture device to encode image frames of an image stream and transmit the encoded image frames over a communication network, comprising: receiving an image frame of the image stream from an image sensor of the image capture device; dividing the image frame into a plurality of slices, wherein the plurality of slices include one or more first slices defined by first slice parameters and one or more second slices defined by second slice parameters; prioritizing the plurality of slices such that the one or more first slices are prioritized over the one or more second slices; and generating a first encoded image frame, wherein one or more first encoded slices are defined by the first slice parameters and are based on the encoding of the one or more first slices. The method comprises: generating a first encoded image frame, wherein the first slice parameters include at least one second encoded slice defined by the second slice parameters and including one or more first skip blocks, the first slice parameters include at least one of a first slice size or a first slice position within the first encoded image frame, the second slice parameters include at least one of a second slice size or a second slice position within the first encoded image frame, and at least one of the second slice size or a second slice position is different from the corresponding value of at least one of the first slice size or a first slice position;A method comprising: generating a second encoded image frame comprising one or more additional second encoded slices defined by the second slice parameters and based on the encoding of the one or more second slices, and one or more additional first encoded slices defined by the first slice parameters and including one or more second skip blocks; and transmitting the first encoded image frame to a receiving device on the communication network prior to the second encoded image frame. Claim 2 A method according to claim 1, further comprising the step of transmitting a first indication indicating that the first encoded image frame is not displayed on the receiving device. Claim 3 A method according to claim 2, wherein the step of transmitting the first mark includes the step of transmitting the first mark in the header of the first encoded image frame. Claim 4 The method of claim 1, wherein the step of generating the encoded image frames comprises: encoding the plurality of slices such that each of the encoded slices is independent of all other encoded slices of the plurality of encoded slices. Claim 5 A method according to claim 1, further comprising the step of transmitting to the receiving device a second indication indicating how the image frame is divided into the plurality of slices. Claim 6 The method according to claim 1, wherein the step of generating the first encoded image frame comprises the step of encoding the plurality of slices such that the one or more first slices are encoded into the one or more first encoded slices of the first encoded image frame and the one or more second encoded slices are generated by replacing the one or more second slices with the second skip blocks; and the step of generating the second encoded image frame comprises the step of encoding the plurality of slices such that the one or more second slices are encoded into the one or more additional second encoded slices of the second encoded image frame and the one or more additional first encoded slices are generated by replacing the one or more first slices with the one or more first skip blocks. Claim 7 A method according to claim 1, wherein the step of generating the first encoded image frame and the step of generating the second encoded image frame include the step of generating the intermediate encoded image frame by encoding the plurality of slices such that the one or more first slices are encoded into one or more first encoded slices of the intermediate encoded image frame and the one or more second slices are encoded into one or more second encoded slices of the intermediate encoded image frame, and the step of generating the first encoded image frame further includes the step of inserting the one or more first encoded slices of the intermediate encoded image frame into the first encoded image frame and the step of inserting skip blocks instead of the one or more second encoded slices of the intermediate encoded image frame, and the step of generating the second encoded image frame further includes the step of inserting the one or more second encoded slices of the intermediate encoded image frame into the second encoded image frame and the step of inserting skip blocks instead of the one or more first encoded slices of the intermediate encoded image frame. Claim 8 In claim 1, the plurality of slices includes one or more third slices defined by third slice parameters, and the step of prioritizing the plurality of slices includes the step of prioritizing one or more second slices with respect to the one or more third slices, and the generated first encoded image frame further includes one or more third encoded slices defined by the third slice parameters and including one or more third skip blocks, and the generated second encoded image frame further includes the one or more third encoded slices, and the method comprises: generating a third encoded image frame including one or more additional third encoded slices defined by the third slice parameters and based on the encoding of one or more third slices, one or more additional first encoded slices defined by the first slice parameters and including one or more second skip blocks, and one or more second encoded slices defined by the second slice parameters and including one or more first skip blocks; A method further comprising the step of transmitting the second encoded image frame to the receiving device on the communication network prior to the third encoded image frame. Claim 9 A method according to claim 1, wherein the skip blocks indicate that there is no difference in encoding between the previously encoded image frame for the part of the encoded image frame covered by the skip blocks and the encoded image frame including the skip blocks. Claim 10 A method according to claim 1, wherein the plurality of slices have the same size. Claim 11 A non-transient computer program storage device comprising computer-readable code units that, when executed on an image-processing device, cause the image-processing device to perform the method according to claim 1. Claim 12 An image-processing device comprises a processor configured to perform a method for encoding image frames of an image stream and transmitting the encoded image frames over a communication network, wherein the processor: receives an image frame of the image stream from an image sensor of an image capture device; divides the image frame into a plurality of slices, wherein the plurality of slices include one or more first slices defined by first slice parameters and one or more second slices defined by second slice parameters; prioritizes the plurality of slices such that the one or more first slices are prioritized over the one or more second slices; generates a first encoded image frame, wherein the first encoded image frame comprises: one or more first encoded slices defined by the first slice parameters and based on the encoding of the one or more first slices; and one or more second encoded slices defined by the second slice parameters and including one or more first skip blocks, wherein the first slice parameters include at least one of a first slice size or a first slice position within the first encoded image frame, and the second slice parameters within the first encoded image frame It includes at least one of a second slice size or a second slice position, wherein at least one of the second slice size or the second slice position is different from the corresponding one of the first slice size or the first slice position;An image-processing device configured to generate a second encoded image frame comprising one or more additional second encoded slices defined by the second slice parameters and based on the encoding of the one or more second slices, and one or more additional first encoded slices defined by the first slice parameters and including one or more second skip blocks; and to transmit the first encoded image frame to a receiving device on the communication network prior to the second encoded image frame. Claim 13 In claim 12, the image-processing device is further configured such that the processor transmits a first indication indicating that the first encoded image frame is not displayed on the receiving device. Claim 14 In claim 12, the image-processing device is an image-processing device that is a video camera. Claim 15 delete

Images