Method and image-capturing device for encoding image frames of image stream and transmitting encoded image frames on communications network

Abstract

To provide a device for encoding image frames of an image stream and transmitting encoded image frames on a communications network.SOLUTION: A method comprises: dividing an image frame into multiple slices wherein the multiple slices comprise first slices and second slices and prioritizing (503) the multiple slices such that the one or more first slices are prioritized over the one or more second slices; generating (504) a first encoded image frame comprising one or more second encoded slices (411a, 411c) comprising one or more first skip blocks; and transmitting (505) the first encoded image frame before the second encoded image frame on the communications network to a receiving device.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The embodiments herein relate to a method and an image capture device for transmitting image frames over a communications network. A corresponding computer program and a computer program carrier are also disclosed. [Background technology]

[0002] Public surveillance using imaging, and particularly video imaging, is common in many areas around the world. Areas that may require surveillance are, for example, banks, stores, and other areas requiring security, such as schools and government facilities. Other areas requiring surveillance are processing, manufacturing, and logistics applications, with video surveillance being primarily used to monitor processing.

[0003] Video is typically streamed over a communication network to access the video remotely. To stream video over a communication network, video encoding is typically required. Video encoding is the process of preparing video for output, for example, digital video is encoded to meet the appropriate format and specifications for recording and playback through the use of video encoder software. In other words, video encoding is the process of converting raw images into compressed video files so that they are saved as a flowing video, not as individual images. Encoding may also be used as an umbrella term to describe similar processes related to resizing video files. Video encoding provides a way to compress image files without compromising video quality. Both video encoding and encoding often refer to the same process.

[0004] In the field of video compression, video frames can be compressed using different algorithms with different advantages and disadvantages, mainly centered on the amount of data compression. These different algorithms for video frames can be called picture types or frame types. For example, there can be three different picture types, also called prediction types, namely I, P, and B, that are used in different video algorithms. They differ in the following characteristics:

[0005] I-frames are the least compressible, but do not require other video frames in order to be decoded.

[0006] P-frames can be decompressed using data from the previous frame and are more compressible than I-frames.

[0007] B-frames can achieve the highest amount of data compression by using both previous and subsequent frames for data reference.

[0008] In the H.264 / MPEG-4 AVC standard, the granularity of prediction types is down to the slice level. A slice is a spatially distinct region of a frame that is coded separately from any other region in the same frame. I-slices, P-slices, and B-slices take the place of I, P, and B frames.

[0009] However, streaming video over a communication network always involves the risk of dropping packets and therefore video frames or even parts of video frames may be dropped when streamed over the communication network. The effect of such missing packets is that parts of the video are corrupted by errors. As objects in the video image may be moving, the errors may also move around in the video image, which may be perceived as disturbances. Examples of error effects are traces / dirt left behind a moving object, background near an object may move with the object, inaccurate or unreasonable colors and melting of the image. Some players may refuse to play anything that has errors or that relies on previous parts that had errors.

[0010] Video coding according to standards such as the H.264 standard may specify that slices must be coded and transmitted in a particular order, for example starting from the top left corner and ending at the bottom right corner of an image frame input to an encoder. Each slice may be placed in its own Network Abstraction Layer (NAL) unit, and each NAL unit may then be packetized into a series of Internet Protocol (IP) packets sent over a communication network. Statistically, the first packet has a higher probability of reaching the receiving device through the communication network than a later packet. If a later packet contains a slice with content that has a higher relevance than the content of the slice in the previous packet, there is a higher risk of losing the high priority content than the risk of losing the low priority content. This is the case, for example, for slices from a surveillance camera that is typically oriented so that the most interesting part of the scene is shown in the center of the video frame, whereas the standard specifies that slices must be coded and transmitted in a particular order, starting from the top left corner and ending at the bottom right corner of an image frame input to an encoder.

[0011] Video coding according to the Baseline Profile of the H.264 standard includes a solution to this problem: Flexible Macroblock Ordering (FMO). FMO means that one can freely choose which macroblock to transmit first (and which one is most likely to be transmitted successfully). US Patent Application Publication No. 2003 / 0112867 discloses a method for encoding and transmitting video data over a communication channel, where a video frame is sliced ​​into a first tile group (slice 1) covering a central portion of the video frame corresponding to a region of interest (ROI), the area around the ROI is sliced ​​into a second tile group (slice 2) containing non-ROI data, and the ROI is coded and placed in the bitstream before the non-ROI data.

[0012] U.S. Patent Application Publication No. 2003 / 0112867 discloses a method for encoding and transmitting video frames, where the video frame is determined to contain areas of higher relevance, slicing the video frame and grouping the slices into subgroups, a first group corresponding to areas of higher relevance within the video frame and a second group containing background, and where the first group is prioritized during transmission over a wireless network.

[0013] US Patent Application Publication No. 2014 / 0036999 discloses slice group priority assignment, where higher priority slice groups may be in the central portion of a video frame, with higher priority slice groups being transmitted first (QoS).

[0014] However, FMO-based solutions are not generally supported due to their technical complexity. Summary of the Invention [Problem to be solved by the invention]

[0015] Accordingly, an objective of embodiments herein may be to obviate or at least reduce the effects of some of the problems mentioned above, and in particular to improve the transport of packets over communication networks.

[0016] As mentioned above, when packet dropping occurs, it may be preferable to be able to prioritize, at least at a statistical level, which parts of a video frame or package are dropped. In one example, a frame is transmitted in a bunch of packets. The first packets may have a higher probability of reaching the receiving device through the communication network than later packets. Thus, later parts of the frame may be dropped on the communication network. This is not optimal, since surveillance cameras are usually oriented so that the most interesting parts of the scene are shown in the center of the video frame. [Means for solving the problem]

[0017] The embodiments herein disclose a solution that is compatible with currently used standards, i.e., compatible with transmitting slices of a frame in a predefined order, and reduces the probability of dropping slices with higher priority content.

[0018] According to one aspect, this object is achieved by a method performed by an image capture device for encoding image frames of an image stream and transmitting the encoded image frames over a communications network.

[0019] The method includes receiving image frames of an image stream from an image sensor of an image capture device.

[0020] The method further comprises dividing the image frame into a plurality of slices, the plurality of slices comprising one or more first slices defined by first slice parameters and one or more second slices defined by second slice parameters.

[0021] The method further comprises prioritizing the plurality of slices such that one or more first slices are prioritized over one or more second slices.

[0022] The method further comprises generating a first encoded image frame comprising one or more first encoded slices defined by the first slice parameters and comprising one or more second encoded slices defined by the second slice parameters and comprising one or more first skip blocks based on encoding the one or more first slices.

[0023] The method further comprises generating a second encoded image frame comprising one or more further first encoded slices defined by the first slice parameters and comprising one or more second skip blocks based on encoding the one or more second slices.

[0024] The method further comprises transmitting the first encoded image frame before the second encoded image frame over the communications network to a receiving device.

[0025] For each coded image frame, the coded slices may be transmitted, for example, according to a first order as specified in the video compression standard.

[0026] According to another aspect, the above object is achieved by an image capture device configured to perform the above method.

[0027] According to a further aspect, this object is achieved by a computer program and a computer program carrier corresponding to the above aspects.

[0028] For each encoded image frame, the encoded slices may be transmitted in a first (normal) order as specified by the video compression standard to meet the requirements for slice transmission, while the original slice prioritization is performed at the frame level by incorporating slices of different priority in different encoded frames associated with the same input frame received from the image sensor and transmitting the encoded frames associated with the same input frame according to a second order. The second order may be based on a prioritization of the contents of the slices of the respective input frames.

[0029] Since the first encoded image frame is transmitted to a receiving device on the communication network before the second encoded image frame, e.g. according to the second order, the probability of losing the priority slice is lower.

[0030] A further advantage of embodiments herein is that when slices are coded such that a coded slice is independent of other coded slices that result from the same received frame, e.g., the same raw input frame, errors due to lost communication packets, such as IP packets, and therefore lost slices, do not spread to other slices. [Brief description of the drawings]

[0031] In the figures, features which appear in some embodiments are indicated by dashed lines.

[0032] The various aspects, including particular features and advantages of the embodiments disclosed herein, will be readily understood from the following detailed description and the accompanying drawings.

[0033] [Figure 1] FIG. 1 is a diagram illustrating an exemplary embodiment of an image capture device. [Figure 2a] FIG. 2a illustrates an exemplary embodiment of a video network system. [Figure 2b] FIG. 2b illustrates an exemplary embodiment of a video network system and user equipment. [Figure 3a] FIG. 3a is a schematic block diagram illustrating an exemplary embodiment of an imaging system. [Figure 3b] FIG. 3b is a schematic block diagram illustrating the referencing method in an image capture device. [Figure 4a] FIG. 4a is a schematic block diagram illustrating an embodiment of a method in an image capture device. [Figure 4b] FIG. 4b is a schematic block diagram illustrating an embodiment of a method in an image capture device. [Figure 4c] FIG. 4c is a schematic block diagram illustrating an embodiment of a method in an image capture device. [Figure 4d] FIG. 4d is a schematic block diagram illustrating an embodiment of a method in an image capture device. [Figure 4e] FIG. 4e is a schematic block diagram illustrating an embodiment of a method in an image capture device. [Diagram 5] FIG. 5 is a flow chart illustrating an embodiment of a method in an image capture device. [Figure 6] FIG. 6 is a block diagram illustrating an embodiment of an image capture device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Embodiments herein may be implemented in one or more image capture devices, such as digital cameras. Figure 1 illustrates various exemplary image capture devices 110. Image capture device 110 may be or may include, for example, a video camera 120, such as a camcorder, a network video recorder, a camera, a security camera or a monitoring camera, a digital camera, a wireless communication device 130, such as a smartphone that includes an image sensor, or an automobile 140 that includes an image sensor.

[0035] 2a illustrates an exemplary video network system 250 in which embodiments herein may be implemented. The video network system 250 may include an image capture device, such as a video camera 120, that can capture digital images 201, such as digital video images, and perform image processing thereon. A video server 260 in FIG. 2a may obtain images from the video camera 120, for example, over a network or the like, which is indicated in FIG. 2a by a bidirectional arrow.

[0036] Video server 260 is a dedicated computer-based device for distributing video. Video servers are used in many applications and often have additional features and capabilities that address the needs of a particular application. For example, video servers used in security, surveillance, and inspection applications are typically designed to capture video from one or more cameras and distribute the video over a computer network connection. In video production and broadcast applications, the video server may have the ability to record and play back recorded video and distribute many video streams simultaneously. Today, many video server functions are built into video cameras 120.

[0037] However, in FIG. 2a, video server 260 is connected to image capture device 110, exemplified by video camera 120, via video network system 250. Video server 260 may be further connected to video storage 270 for storage of video images and / or to monitor 280 for display of video images. In some embodiments, video camera 120 is directly connected to video storage 270 and / or monitor 280, as indicated by the direct arrows between these devices in FIG. 2a. In some other embodiments, video camera 120 is connected to video storage 270 and / or monitor 280 via video server 260, as indicated by the arrows between video server 260 and the other devices.

[0038] 2b shows a user device 295 connected to the video camera 120 via the video network system 250. The user device 295 may be, for example, a computer or a mobile phone. The user device 295 may, for example, control the video camera 120 and / or display video emanating from the video camera 120. The user device 295 may further include the functionality of both the monitor 280 and the video storage 270.

[0039] To better understand the embodiments herein, an imaging system is first described.

[0040] FIG. 3a is a schematic diagram of an imaging system 300, in this case a digital video camera such as video camera 120. The imaging system images a scene onto an image sensor 301. The image sensor 301 may be equipped with a Bayer filter so that different pixels receive radiation in specific wavelength regions in a known pattern. Typically, each pixel of a captured image is represented by one or more values ​​that represent the intensity of the captured light in a certain wavelength band. These values ​​are usually called color components or color channels. The term "image" can refer to an image frame or a video frame that contains information resulting from the image sensor that captured the image.

[0041] After reading the signals of the individual sensor pixels of the image sensor 301, different image processing actions can be performed by the image processing pipeline 302. The image processing pipeline 302 may comprise an image processor 302a and a video post-processor 302b.

[0042] Typically, for video processing, images are included in a stream of images. Figure 3a shows a first video stream 310 from an image sensor 301. The first image stream 310 may comprise multiple captured image frames, such as a first captured image frame 311 and a second captured image frame 312.

[0043] Image processing may include demosaicing, color correction, noise filtering (to remove spatial and / or temporal noise), distortion correction (e.g., to remove the effects of barrel distortion), global and / or local tone mapping (e.g., to enable imaging of scenes containing a wide range of intensities), transformations (e.g., rectification and rotation), flat-field correction (e.g., to remove the effects of vignetting), application of overlays (e.g., privacy masks, legends), etc. The image processing pipeline 302 may also be associated with an analytics engine that performs object detection, recognition, alarms, etc.

[0044] The image processor 302a may, for example, perform image stabilization, apply noise filtering, distortion correction, global and / or local tone mapping, transformations, and flat-field correction. The video post-processor 302b may, for example, crop portions of an image, apply overlays, and include an analysis engine.

[0045] Following the image processing pipeline 302, the images may be forwarded to an encoder 303 where the information in the image frames is encoded according to an encoding protocol such as H.264. The encoded image frames are then forwarded to, for example, a receiving client, here exemplified by a monitor 280, a video server 260, storage 270, etc.

[0046] The video encoding process produces a number of values ​​that can be encoded to form a compressed bitstream. These values ​​may include the following: Quantized transform coefficients information to enable the decoder to recreate the prediction; Information about the structure of the compressed data and the compression tool used during encoding; Information about the complete video sequence.

[0047] These values ​​and parameters (syntax elements) are converted into binary code, for example using variable length coding and / or arithmetic coding. Each of these coding methods produces an efficient and compact binary representation of the information, also called a coded bitstream. The coded bitstream can then be stored and / or transmitted.

[0048] As mentioned above, it is important to improve the encoding of video images of a video stream and the transmission of the video stream over a communication network in order to reduce the effect of dropped data packets on the communication network.

[0049] To better understand the embodiments herein, a reference method for encoding and transmitting an image frame is first presented with reference to FIG. 3b.

[0050] As mentioned above, the first captured image frame 311 is encoded by the encoder 303 by dividing the first captured image frame 311 into a number of parts called slices. In Fig. 3b, the first captured image frame 311 is divided into two slices, namely a first slice 311a and a second slice 311b. Other image frames, such as the second captured image frame 312, are also divided into slices. Thus, the second captured image frame 312 is divided into two slices, namely a first slice 312a and a second slice 312b.

[0051] According to standard encoding techniques, the slices may be encoded and transmitted in a particular order, for example starting from the upper left corner and ending at the lower right corner of the image frame input to encoder 303. Each slice may be placed into its own Network Abstraction Layer (NAL) unit, and each NAL unit may then be packetized into a series of Internet Protocol (IP) packets that are sent over a communications network, such as video network system 250.

[0052] However, dropped data packets on the communication network can be a problem. The first part of the encoded image frame may have a higher probability of reaching the receiving device through the communication network than the later parts. Thus, the later parts of the encoded image frame may be dropped on the communication network. This is not optimal because surveillance cameras are usually oriented so that the most interesting part of the scene is shown in the center of the video frame.

[0053] One way to solve this problem is by providing an image capture device configured to slice (or tile) an input image into several parts, move each part (or group of parts) into a separate frame arranged in a priority order, and replace empty areas with pre-generated empty slices. Thus, for each encoded frame, the slices may be sent in a first (e.g., normal) order or according to a first set of priorities as specified by a video compression standard to meet the requirements for slice transmission, while the prioritization of the original slices is performed at the frame level by incorporating slices of different priorities in different encoded frames associated with the same input frame received from the image sensor and transmitting the encoded frames associated with the same input frame according to a second order. The second order may be based on a prioritization of the contents of the slices of the respective input frames. The second order may be different from the first order. The second order may also be referred to as a second priority set. Thus, the first and second encoded frames associated with the same input frame are transmitted in a second order, for example according to a second set of priorities. For example, based on the content of the input slices, the coded slices are distributed to a set of coded frames associated with the same input frame such that the coded slice with the highest priority according to the content is transmitted first. Within each coded frame associated with the same input frame, slices are transmitted according to the first priority, for example according to a standard specification.

[0054] The separate frames may then be combined into one frame at the decoder side, similar to how regular video frames are combined into a result at a particular time. An empty slice functions like an empty frame and may be generated like an empty frame. An empty frame may be a frame that has no coded differences with respect to a previous frame of the encoded image stream. In some embodiments, an empty frame may be a pre-computed encoded frame. By pre-computed encoded frame, it is meant that instead of generating an encoded frame with the encoder 303, the encoded frame is generated in advance by a software algorithm, for example, by encoding all empty frames in the same way. Thus, an empty frame may be a pre-computed encoded frame that has no coded differences with respect to a previous frame of the encoded image stream. For example, such a pre-computed empty frame may include one or more macroblocks that indicate that there was no change with respect to the previous frame. Such changes may relate to the image content of the previous frame, such as, for example, pixel values. Thus, an empty slice may be a slice that has no coded differences with respect to a previous slice of the encoded image stream.

[0055] For example, the normal QP value per slice in the slice header according to the H.264 / H.265 standard, or the base QP per frame, may be set to QP 51. For empty frames, e.g., frames that have no coded block residual, the QP is not used by the decoder, and if QP 51 is never used by the encoder 303, it may be used as a "flag" to indicate an empty slice. The advantage of using a QP value is that there is no need for extra processing of it since it is already part of the video ready to be used.

[0056] All but the last frame of a group of frames created from the same single frame may be marked as invisible to avoid strange visual effects such as partial image updates. Additionally, motion vector search may be restricted only within each slice / tile.

[0057] 4a, 4b, 4c, 4d, 4e and 5, and further with reference to FIGS. 1, 2a, 2b and 3a, exemplary embodiments herein will be described.

[0058] FIG. 4a illustrates an embodiment of an encoding method in the image capture device 110. In particular, the embodiment may be performed by the encoder 303. An image frame 410 may be divided into a plurality of slices 410a, 410b, 410c, the plurality of slices 410a, 410b, 410c comprising one or more first slices 410b defined by a first slice parameter and one or more second slices 410a, 410c defined by a second slice parameter. In FIG. 4a, the one or more first slices 410b are shown with a different fill pattern than the one or more second slices 410a, 410c. In the example of FIG. 4a, the one or more first slices 410b are single slices and the one or more second slices 410a, 410c include two slices.

[0059] FIG. 5 shows a flowchart illustrating a method performed by the image capture device 110 for encoding image frames 311, 410 of an image stream 310 and transmitting the encoded image frames 411, 412 over a communications network, such as the video network system 250.

[0060] Action 501 The image capture device 110 receives captured image frames 311, 410 of the image stream 310 from the image sensor 301 of the image capture device 110. For example, the encoder 303 may receive captured image frames 311, 410 of the image stream 310 from the image sensor 301. The image capture device 110 may receive complete raw frames.

[0061] Action 502 The image capture device 110 divides the image frame 311, 410 into a number of slices 410a, 410b, 410c. The number of slices 410a, 410b, 410c includes one or more first slices 410b defined by first slice parameters and one or more second slices 410a, 410c defined by second slice parameters. The slice parameters may comprise a size and a position within the frame of the slice.

[0062] The slices 410a, 410b, 410c may be of equal size. In one embodiment, the input image 410 is divided into three slices of equal size and encoded. For example, the slices 410a, 410b, 410c may further include one or more third slices 410c defined by a third slice parameter.

[0063] Image partitioning can be dynamic to better cover the most important parts of the video, which can vary: for example, the number, size, and position of slices can be dynamically changed.

[0064] Action 502 may be performed by the encoder 303 .

[0065] Action 503 The image capture device 110 prioritizes the multiple slices 410a, 410b, 410c such that one or more first slices 410b are given priority over one or more second slices 410a, 410c.

[0066] The prioritization may be based on the content of the slices 410a, 410b, 410c. For example, one or more slices may be grouped into the same priority group. For example, one or more second slices 410a, 410c may be grouped into a second priority group having a lower priority than one or more first slices 410b in the first priority group.

[0067] In some embodiments herein, a raw frame is divided into three slices and encoded, with the middle slice getting the highest priority and the other two slices getting lower priority. Thus, in these embodiments, the middle slice may be included in one or more first slices 410b that take priority over the other two slices that are included in one or more second slices 410a, 410c.

[0068] In some other embodiments herein, prioritizing the plurality of slices 410a, 410b, 410c includes prioritizing one or more second slices 410a over one or more third slices 410c.

[0069] Action 504 FIG. 4b shows a first and a second encoded image frame 411,412.

[0070] The image capture device 110 generates a first encoded image frame 411 including one or more first encoded slices 411b defined by first slice parameters and based on encoding of one or more first slices 410b, and one or more second encoded slices 411a, 411c defined by second slice parameters and including one or more first skip blocks.

[0071] A skip block may indicate that for the portion of the coded image frame 421, 422 that the skip block covers, there are no coded differences between the coded image frame 421, 422 containing the skip block and the previous coded image frame 411, 412.

[0072] The image capture device 110 further generates a second encoded image frame 412 including one or more further second encoded slices 412a, 412c defined by second slice parameters and based on the encoding of the one or more second slices 410a, 410c, and including one or more further first encoded slices 412b defined by the first slice parameters and including one or more second skip blocks.

[0073] Generating the encoded image frames 411, 412 may include encoding the multiple slices 411a, 411b, 411c such that each encoded slice 410a, 410b, 410c of the multiple encoded slices 410a, 410b, 410c is independent of every other encoded slice 411a, 411b, 411c of the multiple encoded slices 411a, 411b, 411c.

[0074] Thus, each coded slice 411a, 411b, 411c may be independent of other coded slices 411a, 411b, 411c that result from the same received input image frame 410. For example, in some embodiments herein, one or more first coded slices 411b are independent of one or more second coded slices 411a, 411c.

[0075] However, slices in an encoded image frame associated with a different input image frame, e.g., associated with a previous or subsequent input image frame, may depend on a previous or subsequent slice. For example, if an object moves across a slice, for one or more second encoded slices 411a, 411c, the encoder cannot use a motion vector pointing to one or more first encoded slices 411b. Instead, the moving object must be filled with an I-block when it enters one or more second encoded slices 411a, 411c. This is more costly in terms of computation and transmission resources, but limits the dependency. As soon as an object is in one or more second encoded slices 411a, 411c, a motion vector may be used, e.g., such that one or more second encoded slices 411a, 411c depend on a corresponding encoded slice associated with a previous or subsequent image frame, such as a previous or subsequent input frame or a previous or subsequent encoded frame. In the embodiments herein, a previous or subsequent encoded image frame refers to a set of encoded image frames associated with the same received input frame.

[0076] For example, motion vector search may be restricted to only within each slice, meaning there is no prediction across slice boundaries, which prevents errors from propagating between coded slices.

[0077] The generation of encoded frames is exemplified by the following two approaches.

[0078] 1. The encoder 303 performs two passes of the input image frame 311, 410, each of which produces an encoded output frame 411, 412 having multiple, e.g. three, coded slices, and for each slice it is required to either generate actual slice data or fill with skip blocks.

[0079] In the first approach, shown in Fig. 4c, an input frame 410 is encoded twice: once into a first encoded frame 411 and once into a second encoded frame 412.

[0080] 2. A frame is encoded in multiple, e.g., two slices, into an intermediate coded image frame 415. A second approach is shown in Fig. 4d. The intermediate coded image frame 415 may include multiple coded slices, such as one or more first coded slices 415b of the intermediate coded image frame 415 and one or more second coded slices 415a, 415c of the intermediate coded image frame 415.

[0081] Then, multiple, e.g., two, output frames are constructed by selecting the coded slice data for some slices and filling in the skip blocks for other slices. Thus, the intermediate coded image frame 415 may be used to generate the first coded image frame 411 and the second coded image frame 412.

[0082] Thus, in some embodiments herein, generating the first encoded image frame 411 includes encoding multiple slices 410a, 410b, 410c such that one or more first slices 410b are encoded into one or more first encoded slices 411b of the first encoded image frame 411, and one or more second encoded slices 411a, 411c of the first encoded image frame 411 are generated by replacing one or more second slices 410a, 410c with a second skip block. Then, generating the second encoded image frame 412 includes encoding the multiple slices 410a, 410b, 410c such that one or more second slices 410a, 410c are encoded into one or more further second encoded slices 412a, 412c of the second encoded image frame 412, and one or more further first encoded slices 411a, 411c are generated by replacing one or more first slices 410b with one or more first skip blocks.

[0083] In some other embodiments, generating the first encoded image frame 411 and generating the second encoded image frame 412 include generating the intermediate encoded image frame 415 based on encoding the image frames 311, 410 by encoding the multiple slices 410a, 410b, 410c, such that one or more first slices 410b are encoded into one or more first encoded slices 415b of the intermediate encoded image frame 415 and one or more second slices 410a, 410c are encoded into one or more second encoded slices 415a, 415c of the intermediate encoded image frame 415. Then, generating the first encoded image frame 411 further includes inserting one or more first encoded slices 415b of the intermediate encoded image frame 415 into the first encoded image frame 411 and inserting a skip block in place of one or more second encoded slices 415a, 415c of the intermediate encoded image frame 415, and generating the second encoded image frame 412 further includes inserting one or more second encoded slices 415a, 415c of the intermediate encoded image frame 415 into the second encoded image frame 412 and inserting a skip block in place of one or more first encoded slices 411b of the intermediate encoded image frame 415.

[0084] 4e shows some embodiments when the plurality of slices 410a, 410b, 410c includes one or more third slices 410c defined by third slice parameters. The generated first encoded image frame 411 may then further comprise one or more third encoded slices 411c defined by the third slice parameters and comprising one or more third skip blocks, and the generated second encoded image frame 412 may further comprise one or more third encoded slices 411c.

[0085] These embodiments may further comprise generating a third coded image frame 413 comprising one or more further third coded slices 413c defined by third slice parameters and based on encoding the one or more third slices 410c, one or more further first coded slices 412b defined by the first slice parameters and comprising one or more second skip blocks, and one or more second coded slices 411a defined by the second slice parameters and comprising one or more first skip blocks.

[0086] Action 505 To meet the requirement of transmitting slices of an encoded frame in a particular order that is not based on the priority of the slices' content, the image capture device 110 transmits the encoded slices of each encoded image frame 411, 412 according to a first set of priorities and transmits the encoded image frames 411, 412 generated from the same received image frame 410 according to a second set of priorities. As discussed above, the first set of priorities may be based on standard requirements and the second set of priorities may be based on the slices' content.

[0087] For example, one or more first coded slices 411b may be based on one or more first slices 410b that are prioritized over one or more second slices 410a, 410c, e.g., based on the content of one or more of the first and second slices 410a, 410b, 410c. Thus, one or more first coded slices 411b should be prioritized over one or more further second coded slices 412a, 412c. This is done by transmitting one or more first coded slices 411b as part of a separate first coded frame 411 that is prioritized over a second coded frame 412 that includes one or more further second coded slices 412a, 412c.

[0088] Thus, image capture device 110 transmits the first encoded image frame 411 to receiving device 260, 270, 280, 295 over communications network 250 before the second encoded image frame 412. Thus, because image capture device 110 transmits the first encoded image frame 411 before the second encoded image frame 412, image capture device 110 prioritizes the first encoded image frame 411 over the second encoded image frame 412.

[0089] The horizontal arrows in each of Figures 4b and 4e indicate the transmission time.

[0090] 4b further illustrates how the image capture device 110 transmits a further first encoded image frame 421 before a further second encoded image frame 422, following the same principles as described above for the first encoded image frame 411 and the second encoded image frame 412. The time interval between the set of first and second encoded image frames 411, 412 and the second set of further first and second encoded image frames 421, 422 may be set according to a particular desired frame rate, for example 1 / 30 seconds.

[0091] FIG. 4e further shows how the image capture device 110 transmits a further second encoded image frame 422 before a further third encoded image frame 423, following the same principles as described above.

[0092] The image capture device 110 may send a first indication to the receiving device 260, 270, 280, 295 indicating that the first encoded image frame 411 should not be displayed.

[0093] In some embodiments herein, the first instruction is transmitted in a header of the first encoded image frame 411 .

[0094] In general, all transmitted encoded frames except the last transmitted encoded frame may include an indication, for example in the encoded frame's header, that the encoded frame should not be displayed.

[0095] In some other embodiments herein, the first encoded frame 411 comprises a first indication indicating that only the last encoded frame 412 of the encoded frames related to the first encoded frame 411 should be displayed.

[0096] This indication may be implemented as a flag, such as a one-bit flag. By sending the first instruction, the image capture device 110 helps the receiving device 295 understand how to display the contents of the encoded frame.

[0097] In some embodiments herein, the image capture device 110 transmits a second instruction to the receiving device 260, 270, 280, 295 indicating how the image frame 311, 410 is to be divided into multiple slices 410a, 410b, 410c. By transmitting the second instruction, the image partitioning may be dynamic.

[0098] The second indication may also be sent in one or more headers.

[0099] If there are three encoded image frames, the method may further comprise transmitting the second encoded image frame 412 before the third encoded image frame 413 over the communications network 250 to the receiving devices 260 , 270 , 280 , 295 .

[0100] At the receiver side, the received encoded frames are decoded as usual. The decoded frames are combined into a single frame to be displayed. For example, the decoding of the second image may reuse blocks from the first decoded image when they are skip blocks. In other words, the decoded data from the first decoded image frame may be copied to the second decoded image frame, and then the second decoded image frame is displayed. Thus, one or more decoded frames may not be displayed. In particular, in some embodiments, only the results after the last frame are displayed, including all slices. This may be indicated, for example, by a first indication.

[0101] If the second encoded frame is dropped on the network and the receiving device 260, 270, 280, 295 knows that the first encoded frame should not be displayed, for example if the first encoded frame includes the first indicator, the receiving device 260, 270, 280, 295 will correctly display the next set of decoded frames anyway.

[0102] A complete embodiment is now shown diagrammatically. The input image is encoded into three slices of equal size. The slices are separated into 3 separate frames and the empty regions are replaced with pre-generated empty slices, e.g. indicated by QP value=51. · Frames are ordered such that the frame with the highest priority slice is placed first. The first two frames are marked as hidden. Each frame is sent as one or more network packets. For example, each slice may result in a packet. A drop is most likely to hit frame #2 or #3. Decode the frames normally, and only the results after the last frame are displayed, including every slice of data.

[0103] Thus, according to the embodiments herein, multiple coded image frames may be generated from a single input frame. The number of coded image frames may correspond to or be equal to the number of priority groups of slices that make up the frame. Each coded image frame corresponding to a particular priority group comprises slices generated by the encoder by encoding corresponding input slices from the corresponding priority group, and may skip blocks instead of coded slices from other priority groups. At the receiver side, the multiple coded frames are decoded and the determined slices are combined and displayed as usual.

[0104] Accordingly, some embodiments herein disclose a method in a video camera, the method comprising: - receiving frame data from the image sensor of the video camera; - dividing the received frame into a number of slices, such as two slices; - prioritizing the slices; and - constructing a first frame from video data in prioritized slices and skipping blocks in non-prioritized slices, whereby for example prioritized slices are image coded, i.e. updated, relative to a preceding video frame; - constructing a second frame from video data in the non-prioritized slices and skip blocks in the prioritized slices; - transmitting the first and second frames over a communications network to a receiving device. and

[0105] 6, there is shown a schematic block diagram of an embodiment of an image capture device 110. As described above, the image capture device 110 is configured to encode image frames 311, 410 of an image stream 310 and transmit the encoded image frames 411, 412 over a communications network, such as the video network system 250.

[0106] As mentioned above, the image capture device 110 may include any of the following: a camera, a video camera 120, a security camera, a surveillance camera or camcorder, a network video recorder, and a wireless communication device 130.

[0107] In particular, the image processing device 110 may be a video camera 120, such as a surveillance camera.

[0108] The image processing device 110 may include an encoder 303 configured to perform one or more of the actions of the method according to Fig. 5. For example, the encoder 303 may be configured to encode an input image frame into at least two encoded image frames based on the method described above in relation to Fig. 5.

[0109] The image capture device 110 may further comprise a processing module 601, such as means for performing the methods described herein, which may be embodied in the form of one or more hardware modules and / or one or more software modules.

[0110] The image capture device 110 may further comprise a memory 602. The memory may include, e.g., contain or store, instructions, e.g., in the form of a computer program 603, which may include computer readable code units that, when executed on the image capture device 110, cause the image capture device 110 to perform a method of encoding image frames 311, 410 of an image stream 310 and transmitting the encoded image frames 411, 412 over the communications network 250.

[0111] According to some embodiments herein, image capture device 110 and / or processing module 601 comprise processing circuitry 604 as an exemplary hardware module that may comprise one or more processors. Thus, processing module 601 may be embodied in the form of, or "implemented by," processing circuitry 604. Instructions are executable by processing circuitry 604 such that image capture device 110 may operate to perform the method of FIG. 5 as described above. As another example, the instructions, when executed by image capture device 110 and / or processing circuitry 604, may cause image capture device 110 to perform the method according to FIG. 5.

[0112] In view of the above, in one example, an image capture device 110 is provided for encoding image frames 311, 410 of an image stream 310 and transmitting the encoded image frames 411, 412 over a communications network 250.

[0113] Again, memory 602 includes instructions executable by the processing circuitry 604 such that image capture device 110 operates to perform the method according to FIG.

[0114] 6 further shows a carrier 605, i.e. a program carrier, which contains the immediately above described computer program 603. The carrier 605 may be one of an electronic, optical, or radio signal and a computer readable medium.

[0115] In some embodiments, the image capture device 110 and / or the processing module 601 may include one or more of the following exemplary hardware modules: a receiving module 610, a segmenting module 620, a prioritization module 630, an encoded frame generation module 640, and a transmitting module 650. In other examples, one or more of the above-mentioned exemplary hardware modules may be implemented as one or more software modules.

[0116] In particular, the encoder 303 may comprise one or more, or even all, of a receiving module 610 , a segmenting module 620 , a prioritization module 630 , an encoded frame generation module 640 , and a transmitting module 650 .

[0117] Thus, the encoder 303 may perform the functions of one or more, or all, of the receiving module 610 , the segmenting module 620 , the prioritization module 630 , the encoded frame generation module 640 , and the transmitting module 650 .

[0118] Furthermore, the processing module 601 may comprise an input / output unit 606. According to one embodiment, the input / output unit 606 may comprise an image sensor configured to capture raw image frames as mentioned above, such as the raw image frames included in the first video stream 310 from the image sensor 301.

[0119] According to various embodiments described above, the image capture device 110 and / or the processing module 601 and / or the receiving module 610 and / or the encoder 303 are configured to receive captured image frames 311, 410 of the image stream 310 from the image sensor 301 of the image capture device 110.

[0120] The image capture device 110 and / or the processing module 601 and / or the segmentation module 520 and / or the encoder 303 are configured to segment the image frame 311, 410 into a plurality of slices 410a, 410b, 410c. The plurality of slices 410a, 410b, 410c includes one or more first slices 410b defined by first slice parameters and one or more second slices 410a, 410c defined by second slice parameters.

[0121] The image capture device 110 and / or the processing module 601 and / or the prioritization module 630 are configured to prioritize the multiple slices 410a, 410b, 410c such that one or more first slices 410b are given priority over one or more second slices 410a, 410c.

[0122] The image capture device 110 and / or the processing module 601 and / or the encoded frame generation module 640 and / or the encoder 303 may: the first coded image frame 411 comprising one or more first coded slices 410b defined by the first slice parameters and based on coding the one or more first slices 411b, and one or more second coded slices 411a, 411c defined by the second slice parameters and comprising one or more first skip blocks; The encoding unit 410 is configured to generate the second encoded image frame 412 comprising one or more further second encoded slices 411a, 411c defined by the second slice parameters and based on encoding the one or more second slices 410a, 410c, and one or more further first encoded slices 412b defined by the first slice parameters and comprising one or more second skip blocks.

[0123] In some embodiments herein, the image capture device 110 and / or the processing module 601 and / or the encoded frame generation module 640 and / or the encoder 303 may: generating the first coded image frame 411 by being configured to encode the plurality of slices 410a, 410b, 410c such that the one or more first slices 410b are encoded into one or more first coded slices 411b of the first coded image frame 411 and the one or more second coded slices 411a, 411c are generated by replacing the one or more second slices 410a, 410c with the second skip blocks; configured to generate the second encoded image frame 411 by being configured to encode the plurality of slices 411 a, 411 b, 411 c such that the one or more second slices 410 a, 410 c are encoded into the one or more further second encoded slices 410 a, 410 c of the second encoded image frame 412, and the one or more further first encoded slices 410 a, 411 c are generated by replacing the one or more first slices 410 b with the one or more first skip blocks.

[0124] In some embodiments herein, the image capture device 110 and / or the processing module 601 and / or the encoded frame generation module 640 and / or the encoder 303 are further configured to generate a third encoded image frame 413 including one or more further first encoded slices 410b defined by the first slice parameters, including one or more second skip blocks, including one or more second encoded slices 412a defined by the second slice parameters, and including one or more first skip blocks based on encoding the one or more third slices 413c, including one or more further third encoded slices 411c defined by the third slice parameters.

[0125] The image capture device 110 and / or the processing module 601 and / or the transmission module 650 and / or the encoder 303 are configured to transmit the first encoded image frame 411 to the receiving device 260, 270, 280, 295 before the second encoded image frame 412 over the communications network 250.

[0126] The image capture device 110 and / or the processing module 601 and / or the transmission module 650 and / or the encoder 303 may be configured to send a first indication to the receiving device 260, 270, 280, 295 that the first encoded image frame 411 should not be displayed.

[0127] In some embodiments herein, the image capture device 110 and / or the processing module 601 and / or the transmission module 650 and / or the encoder 303 are configured to transmit the first instruction within a header of the first encoded image frame 411.

[0128] The image capture device 110 and / or the processing module 601 and / or the transmission module 650 and / or the encoder 303 may be configured to send a second instruction to the receiving device 260, 270, 280, 295 indicating how the image frame 311, 410 is divided into multiple slices 410a, 410b, 410c.

[0129] As used herein, the term "module" may refer to one or more functional modules, each of which may be implemented as one or more hardware modules and / or one or more software modules and / or a combined software / hardware module. In some examples, a module may represent a functional unit that is implemented as software and / or hardware.

[0130] As used herein, the terms "computer program carrier," "program carrier," or "carrier" may refer to one of an electronic signal, an optical signal, a radio signal, and a computer-readable medium. In some examples, the computer program carrier may exclude transitory propagating signals, such as electronic signals, optical signals, and / or radio signals. Thus, in these examples, the computer program carrier may be a non-transitory carrier, such as a non-transitory computer-readable medium.

[0131] As used herein, the term "processing module" may include one or more hardware modules, one or more software modules, or a combination thereof. Any such module, be it a hardware, software, or combination of hardware and software module, may be a connecting means, a providing means, a configuring means, a responding means, a disabling means, etc., as disclosed herein. As an example, the term "means" may be a module corresponding to the modules listed above in conjunction with the figures.

[0132] As used herein, the term "software module" may refer to a software application, a dynamic link library (DLL), a software component, a software object, an object under the Component Object Model (COM), a software component, a software function, a software engine, an executable binary software file, etc.

[0133] The terms "processing module" or "processing circuitry" as used herein may encompass processing units including, for example, one or more processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. A processing circuitry or the like may comprise one or more processor kernels.

[0134] As used herein, the phrase "configured to / for" may mean that a processing circuit is configured, e.g., adapted, or operable, by a software and / or hardware configuration, to perform one or more of the actions described herein.

[0135] As used herein, the term "action" may refer to an operation, step, operation, response, reaction, activity, etc. It should be noted that an action herein may be divided into two or more sub-actions, where applicable. Additionally, it should be noted that two or more actions described herein may be merged into a single action, where applicable.

[0136] As used herein, the term "memory" can refer to a hard disk, a magnetic storage medium, a portable computer diskette or disk, a flash memory, a random access memory (RAM), etc. Memory may also refer to a processor's internal register memory, etc.

[0137] As used herein, the term "computer readable medium" may be a Universal Serial Bus (USB) memory, a DVD-disc, a Blu-ray disc, a software module received as a stream of data, a flash memory, a hard drive, a memory stick, a multimedia card (MMC), a memory card such as a secure digital (SD) card, etc. One or more of the foregoing examples of computer readable medium may be provided as one or more computer program products.

[0138] As used herein, the term "computer readable code unit" can be the text of a computer program, part or all of a binary file representing a computer program in compiled format, or anything in between.

[0139] As used herein, the terms "number" and / or "value" may be any type of number, such as a binary number, a real number, an imaginary number, or a rational number. Furthermore, a "number" and / or "value" may be one or more characters, such as a letter or a string of characters. A "number" and / or "value" may be represented by a string of bits, i.e., 0s and / or 1s.

[0140] As used herein, the phrase "in some embodiments" is used to indicate that features of the described embodiments can be combined with any other embodiment disclosed herein.

[0141] While embodiments of various aspects have been described, many different changes, modifications, etc. thereof will become apparent to those skilled in the art. Accordingly, the described embodiments are not intended to limit the scope of the present disclosure.

Claims

1. A method for encoding image frames of an image stream and transmitting the encoded image frames over a communication network, performed by an image capture device, The image sensor of the image capture device receives an image frame from the image stream, The image frame is divided into a plurality of slices, each of which includes one or more first slices defined by a first slice parameter and one or more second slices defined by a second slice parameter. Prioritize the plurality of slices such that the one or more first slices have priority over the one or more second slices. A first encoded image frame, One or more first coded slices based on the coding of one or more first slices defined by the first slice parameter, and Includes one or more second coding slices that include one or more first skip blocks defined by the second slice parameter, The first slice parameter includes at least one of a first slice size or a first slice position in the first encoded image frame, and further The second slice parameter includes at least one of a second slice size or a second slice position in the first encoded image frame, wherein at least one of the second slice size or second slice position generates a first encoded image frame that is different from the corresponding first slice size or first slice position. A second encoded image frame is generated, which includes one or more further second encoded slices based on the encoding of one or more second slices defined by the second slice parameter, and one or more further first encoded slices including one or more second skip blocks defined by the first slice parameter. A method for transmitting a first encoded image frame to a receiving device on a communication network prior to a second encoded image frame.

2. The method according to claim 1, further comprising transmitting a first instruction to the receiving device indicating that the first encoded image frame is not to be displayed.

3. The method according to claim 2, wherein the transmission of the first instruction is further comprising transmitting the first instruction in the header of the first encoded image frame.

4. The method according to claim 1, wherein the generation of the encoded image frame comprises encoding the plurality of slices such that each of the plurality of encoded slices is independent of all the other encoded slices of the plurality of encoded slices.

5. The method according to claim 1, further comprising transmitting a second instruction to the receiving device indicating how the image frame is divided into the plurality of slices.

6. The generation of the first encoded image frame comprises encoding a plurality of slices, wherein one or more first slices are encoded into one or more first encoded slices of the first encoded image frame, and one or more second slices are generated by replacing the second skip blocks with the first or more second encoded slices. The method according to claim 1, wherein the generation of the second encoded image frame comprises encoding a plurality of slices, wherein one or more second slices are encoded into one or more further second encoded slices of the second encoded image frame, and one or more further first encoded slices are generated by replacing one or more first slices with one or more first skip blocks.

7. The generation of the first encoded image frame and the generation of the second encoded image frame are performed by encoding an image frame by encoding a plurality of slices to generate an intermediate encoded image frame, wherein one or more first slices are encoded into one or more first encoded slices of the intermediate encoded image frame, and one or more second slices are encoded into one or more second encoded slices of the intermediate encoded image frame. The generation of the first encoded image frame further involves inserting one or more first encoded slices of intermediate encoded image frames into the first encoded image frame, and inserting one or more first skip blocks in place of one or more second encoded slices of intermediate encoded image frames. The method according to claim 1, wherein the generation of a second encoded image frame further includes inserting one or more second encoded slices of intermediate encoded image frames into the second encoded image frame, and inserting one or more second skip blocks into the second encoded image frame in place of one or more first encoded slices of intermediate encoded image frames.

8. The plurality of slices includes one or more third slices defined by a third slice parameter, Prioritizing the plurality of slices includes prioritizing one or more second slices over one or more third slices, The generated first encoded image frame further includes one or more third encoded slices defined by a third slice parameter and containing one or more third skip blocks, The generated second encoded image frame further includes one or more third encoded slices, and the method further A third encoded image frame is generated, comprising one or more further third encoded slices defined by a third slice parameter and generated based on the encoding of one or more third slices, and one or more further first encoded slices defined by a first slice parameter and containing one or more second skip blocks, and one or more second encoded slices defined by a second slice parameter and containing one or more first skip blocks. The method according to claim 1, wherein a second encoded image frame is transmitted to a receiving device on a communication network prior to a third encoded image frame.

9. The method according to claim 1, wherein each skip block of one or more first skip blocks and one or more second skip blocks indicates that there is no encoded difference between the encoded image frame containing the skip block and the previous encoded image frame with respect to a portion of the encoded image frame covered by the skip block.

10. The method according to claim 1, wherein the plurality of slices are of equal size.

11. A non-temporary computer program storage device including a computer-readable code unit that, when executed on an image processing device, causes the image processing device to perform the method described in Claim 1.

12. An image processing apparatus, A processor configured to perform a method for encoding image frames of an image stream and transmitting the encoded image frames over a communication network, wherein the processor The image sensor of the image capture device receives an image frame from the image stream. The aforementioned image frame is divided into multiple slices, The plurality of slices includes one or more first slices defined by a first slice parameter and one or more second slices defined by a second slice parameter. Prioritize the aforementioned multiple slices such that one or more first slices have priority over one or more second slices. A first encoded image frame, One or more first encoded slices defined by a first slice parameter and based on the encoding of one or more first slices, It includes one or more second coding slices that are defined by a second slice parameter and contain one or more first skip blocks, The first slice parameter includes at least one of a first slice size or a first slice position in the first encoded image frame, The second slice parameter includes at least one of a second slice size or a second slice position in the first encoded image frame, wherein at least one of the second slice size or the second slice position generates a first encoded image frame that is different from the one corresponding to at least one of the first slice size or the first slice position. A second encoded image frame, It includes one or more further second coded slices based on the coding of one or more second slices, defined by a second slice parameter, Generate a second encoded image frame containing one or more further first encoded slices, which are defined by the first slice parameter and contain one or more second skip blocks. An image processing device configured to transmit a first encoded image frame to a receiving device prior to a second encoded image frame over a communication network.

13. The image processing apparatus according to claim 12, wherein the processor is further configured to transmit a first instruction to a receiving device indicating that the first encoded image frame is not to be displayed.

14. The image processing apparatus according to claim 12, wherein the image processing apparatus is a video camera.

Images