Method and device for encoding image frames of an image stream and transmitting the encoded image frames over a communication network.

Abstract

To provide a device for encoding image frames of an image stream and transmitting encoded image frames on a communications network.SOLUTION: A method comprises: dividing an image frame into multiple slices wherein the multiple slices comprise first slices and second slices and prioritizing (503) the multiple slices such that the one or more first slices are prioritized over the one or more second slices; generating (504) a first encoded image frame comprising one or more second encoded slices (411a, 411c) comprising one or more first skip blocks; and transmitting (505) the first encoded image frame before the second encoded image frame on the communications network to a receiving device.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] Embodiments of the present specification relate to a method for transmitting image frames over a communication network and an image capture device. Corresponding computer programs and computer program carriers are also disclosed.

Background Art

[0002] Public surveillance using imaging, particularly video imaging, is common in many regions around the world. Areas that may require surveillance are, for example, banks, stores, and other areas that require security such as schools and government facilities. Other fields that require surveillance are processing, manufacturing, and logistics applications, and video surveillance is mainly used to monitor processing.

[0003] Videos are usually streamed over a communication network for remote access to the video. Video encoding is usually required to stream video over a communication network. Video encoding is, for example, a process of preparing video for output, and digital video is encoded to meet the appropriate formats and specifications for recording and playback through the use of video encoder software. In other words, video encoding is a process of converting RAW images into compressed video files so that they are stored as a fluid video rather than as individual images. Encoding can also be used as an umbrella term to describe similar processes related to changing the size of video files. Video encoding provides a way to compress image files without degrading video quality. Both video encoding and encoding often refer to the same process.

[0004] In the field of video compression, video frames can be compressed using different algorithms, each with different advantages and disadvantages, primarily centered on the amount of data compression. These different algorithms for video frames may be called picture types or frame types. For example, there may be three different picture types, also called prediction types, used in different video algorithms: I, P, and B. These differ in the following characteristics:

[0005] I-frames are the least compressible, but they do not require other video frames to decode.

[0006] P-frames can be decompressed using data from the previous frame and are more compressible than I-frames.

[0007] B-frames can achieve the highest data compression by using both the previous and subsequent frames for data referencing.

[0008] In the H.264 / MPEG-4 AVC standard, the granularity of prediction types is reduced to the slice level. A slice is a spatially distinct region of a frame that is encoded separately from any other region within the same frame. I-slices, P-slices, and B-slices are alternatives to I, P, and B frames.

[0009] However, streaming video over a communication network always carries the risk of dropping packets, and therefore, video frames or parts of video frames may also be dropped when streamed over a communication network. The effect of such packet loss is that parts of the video may be corrupted by errors. Since objects in a video image may be moving, errors may also be moving around in the video image, which may be perceived as interference. Examples of error effects include traces / debris left behind by moving objects, backgrounds near objects moving along with the objects, and inaccurate or illogical colors and image melting. Some players may refuse to play anything that has errors or relies on previous parts that had errors.

[0010] Video coding according to standards such as the H.264 standard may specify that slices must be encoded and transmitted in a specific order, for example, starting from the upper-left corner and ending at the lower-right corner of the image frame input to the encoder. Each slice may be placed in its own Network Abstraction Layer (NAL) unit, and each NAL unit may then be packetized into a series of Internet Protocol (IP) packets sent over a communication network. Statistically, the first packet has a higher probability of reaching the receiving device over the communication network than later packets. If later packets contain slices with content that is more relevant than the content of the slices in the earlier packets, the risk of losing higher-priority content is greater than the risk of losing lower-priority content. This is the case, for example, with slices from a surveillance camera that is typically oriented so that the most interesting part of the scene is shown in the center of the video frame, while the standard specifies that slices must be encoded and transmitted in a specific order, starting from the upper-left corner and ending at the lower-right corner of the image frame input to the encoder.

[0011] Video coding according to the baseline profile of the H.264 standard includes a solution to this problem, namely flexible macroblock ordering (FMO). FMO means that it is possible to freely choose which macroblocks to transmit first (and the macroblocks that are most likely to be transmitted successfully). U.S. Patent Application Publication 2003 / 0112867 discloses a method for coding and transmitting video data over a communication channel, in which a video frame is sliced ​​into a first tile group (slice 1) covering the central portion of the video frame corresponding to a region of interest (ROI), the region around the ROI is sliced ​​into a second tile group (slice 2) containing non-ROI data, and the ROI is coded and placed in the bitstream before the non-ROI data.

[0012] U.S. Patent Application Publication No. 2003 / 0112867 discloses a method for encoding and transmitting video frames, wherein a video frame is determined to contain areas of higher relevance, the video frame is sliced, and the slices are grouped into subgroups, the first group corresponding to areas of higher relevance within the video frame, the second group containing the background, and the first group being prioritized during transmission over a wireless network.

[0013] U.S. Patent Application Publication No. 2014 / 0036999 discloses slice group priority assignment, where higher-priority slice groups may be in the central portion of a video frame, and higher-priority slice groups are transmitted first (QoS).

[0014] However, FMO-based solutions are generally not supported due to their technical complexity. [Overview of the project] [Problems that the invention aims to solve]

[0015] Therefore, an objective of the embodiments described herein may be to prevent some of the above-mentioned problems, or at least reduce their effects. Specifically, an objective may be to improve packet transport over a communication network.

[0016] As mentioned above, when packet drops occur, it is sometimes preferable to be able to prioritize which parts of a video frame or package are dropped, at least at a statistical level. For example, a frame is transmitted in a bundle of packets. The probability of the first packet reaching the receiving device through the communication network may be higher than that of later packets. Therefore, the later parts of the frame may be dropped on the communication network. This is not ideal, as surveillance cameras are usually pointed so that the most interesting parts of the scene are shown in the center of the video frame. [Means for solving the problem]

[0017] Embodiments of this specification disclose solutions that are compatible with currently used standards, i.e., compatible with transmitting frame slices in a predefined order, and that reduce the probability of dropping slices with higher priority content.

[0018] According to one embodiment, this objective is achieved by a method performed by an image capture device for encoding image frames of an image stream and transmitting the encoded image frames over a communication network.

[0019] The method includes receiving image frames from an image sensor of an image capture device.

[0020] The method further comprises dividing an image frame into multiple slices, each slice comprising one or more first slices defined by a first slice parameter and one or more second slices defined by a second slice parameter.

[0021] The method further comprises prioritizing a plurality of slices such that one or more first slices are prioritized over one or more second slices.

[0022] The method comprises one or more first encoded slices defined by a first slice parameter, and based on encoding one or more first slices, comprises one or more second encoded slices defined by a second slice parameter, and further comprises generating a first encoded image frame comprising one or more first skip blocks.

[0023] The method comprises one or more further second encoded slices defined by a second slice parameter, and based on encoding one or more second slices, comprises one or more further first encoded slices defined by a first slice parameter, and further comprises generating a second encoded image frame comprising one or more second skip blocks.

[0024] The method further comprises transmitting the first encoded image frame to a receiving device on a communication network before the second encoded image frame.

[0025] For each encoded image frame, the encoded slices may be transmitted according to a first order, such as specified in a video compression standard.

[0026] According to another aspect, the above object is achieved by an image capture device configured to execute the above method.

[0027] According to a further aspect, this object is achieved by a computer program and a computer program carrier corresponding to the above aspect.

[0028] For each symbolized image frame, the encoded slices may be transmitted in a first (normal) order as specified by the video compression standard in order to meet the requirements regarding the transmission of the slices. On the other hand, the prioritization of the original slices is performed at the frame level by incorporating slices with different priorities into different encoded frames associated with the same input frame received from the image sensor and transmitting the encoded frames associated with the same input frame according to a second order. The second order may be based on the prioritization of the content of the slices of each input frame.

[0029] The first encoded image frame is transmitted to a receiving device on a communication network before the second encoded image frame, for example, according to the second order, so the probability of losing the priority slices is lower.

[0030] A further advantage of the embodiments herein is that when the slices are encoded such that the encoded slices are independent of other encoded slices resulting from the same received frame, for example, the same raw input frame, errors due to lost communication packets such as IP packets, and thus lost slices, do not spread to other slices.

Brief Description of the Drawings

[0031] In the figures, features appearing in some embodiments are indicated by dashed lines.

[0032] Various aspects including specific features and advantages of the embodiments disclosed herein will be readily understood from the following detailed description and the accompanying drawings.

[0033] <着 [Figure 1] FIG. 1 is a diagram showing an exemplary embodiment of an image capturing device. [Figure 2a] FIG. 2a shows an exemplary embodiment of a video network system. [Figure 2b] FIG. 2b shows an exemplary embodiment of a video network system and a user device. [Figure 3a] Figure 3a is a schematic block diagram showing an exemplary embodiment of the imaging system. [Figure 3b] Figure 3b is a schematic block diagram showing the reference method in the image capture device. [Figure 4a] Figure 4a is a schematic block diagram showing an embodiment of the method in an image capture device. [Figure 4b] Figure 4b is a schematic block diagram showing an embodiment of the method in an image capture device. [Figure 4c] Figure 4c is a schematic block diagram showing an embodiment of the method in an image capture device. [Figure 4d] Figure 4d is a schematic block diagram showing an embodiment of the method in an image capture device. [Figure 4e] Figure 4e is a schematic block diagram showing an embodiment of the method in an image capture device. [Figure 5] Figure 5 is a flowchart showing an embodiment of the method in an image capture device. [Figure 6] Figure 6 is a block diagram showing an embodiment of an image capture device. [Modes for carrying out the invention]

[0034] Embodiments of this specification may be implemented in one or more image capture devices, such as a digital camera. Figure 1 shows various exemplary image capture devices 110. The image capture device 110 may be, or include, any of the following: a video camera 120 such as a camcorder, network video recorder, camera, surveillance camera or monitoring camera; a digital camera; a wireless communication device 130 such as a smartphone including an image sensor; or an automobile 140 including an image sensor.

[0035] Figure 2a shows an exemplary video network system 250 in which embodiments of the herein may be implemented. The video network system 250 may include an image capture device, such as a video camera 120, which can capture digital images 201, such as digital video images, and perform image processing thereon. The video server 260 in Figure 2a may acquire images from the video camera 120 via a network, for example, as indicated by the bidirectional arrow in Figure 2a.

[0036] The video server 260 is a dedicated computer-based device for distributing video. Video servers are used in many applications and often have additional functions and capabilities to address the specific needs of those applications. For example, video servers used for security, surveillance, and inspection purposes are typically designed to capture video from one or more cameras and distribute the video over a computer network connection. In video production and broadcast applications, video servers may have the ability to record and play back recorded video and distribute many video streams simultaneously. Today, many video server functions are built into the video camera 120.

[0037] However, in Figure 2a, the video server 260 is connected to an image capture device 110, exemplified by a video camera 120, via a video network system 250. The video server 260 may be further connected to a video storage 270 for storing video images and / or to a monitor 280 for displaying video images. In some embodiments, the video camera 120 is directly connected to the video storage 270 and / or monitor 280, as indicated by direct arrows between these devices in Figure 2a. In some other embodiments, the video camera 120 is connected to the video storage 270 and / or monitor 280 via the video server 260, as indicated by arrows between the video server 260 and the other devices.

[0038] Figure 2b shows a user device 295 connected to the video camera 120 via a video network system 250. The user device 295 may be, for example, a computer or a mobile phone. The user device 295 may, for example, control the video camera 120 and / or display the video emitted from the video camera 120. The user device 295 may further include the functions of both a monitor 280 and video storage 270.

[0039] To better understand the embodiments described herein, the imaging system will be described first.

[0040] Figure 3a is a schematic diagram of an imaging system 300, which in this case is a digital video camera such as video camera 120. The imaging system captures a scene on an image sensor 301. The image sensor 301 may include a Bayer filter so that different pixels receive radiation in a specific wavelength range in a known pattern. Typically, each pixel of the captured image is represented by one or more values ​​that represent the intensity of the captured light within a certain wavelength band. These values ​​are usually called color components or color channels. The term “image” may refer to an image frame or video frame containing information derived from the image sensor that captured the image.

[0041] After reading the signals from individual sensor pixels of the image sensor 301, different image processing actions may be performed by the image processing pipeline 302. The image processing pipeline 302 may include an image processing unit 302a and a video post-processing unit 302b.

[0042] Typically, for video processing, images are included in an image stream. Figure 3a shows a first video stream 310 from an image sensor 301. The first image stream 310 may comprise a plurality of captured image frames, such as a first captured image frame 311 and a second captured image frame 312.

[0043] Image processing may include demosaicing, color correction, noise filtering (to remove spatial and / or temporal noise), distortion correction (e.g., to remove the effects of barrel distortion), global and / or local tone mapping (e.g., to enable imaging of scenes with a wide range of intensities), deformation (e.g., correction and rotation), flat-field correction (e.g., to remove the effects of vignetting), and application of overlays (e.g., privacy masks, descriptive text). The image processing pipeline 302 may also be associated with an analysis engine that performs object detection, recognition, alarms, etc.

[0044] The image processing unit 302a may, for example, perform image stabilization and apply noise filtering, distortion correction, global and / or local tone mapping, transformation, and flat-field correction. The video post-processing unit 302b may, for example, crop a portion of the image, apply an overlay, and include an analysis engine.

[0045] Following the image processing pipeline 302, the image may be transferred to the encoder 303, where the information within the image frame is encoded according to an encoding protocol such as H.264. The encoded image frame is then transferred to a receiving client, video server 260, storage 270, etc., for example, exemplified here by monitor 280.

[0046] The video encoding process generates several values ​​that can be encoded to form a compressed bitstream. These values ​​may include: Quantization transformation coefficients • Information that enables the decoder to recreate the prediction • Information regarding the structure of the compressed data and the compression tool used during encoding. • Information about the complete video sequence.

[0047] These values ​​and parameters (syntax elements) are converted into binary code using, for example, variable-length coding and / or arithmetic coding. Each of these coding methods produces an efficient and compact binary representation of the information, also called a coded bitstream. The coded bitstream can then be stored and / or transmitted.

[0048] As mentioned above, improving the encoding of video images in video streams and the transmission of video streams over communication networks is crucial to mitigate the impact of lost data packets on the network.

[0049] To better understand the embodiments of this specification, a reference method for encoding and transmitting an image frame is first presented with reference to Figure 3b.

[0050] As described above, the first captured image frame 311 is encoded by the encoder 303 by dividing the first captured image frame 311 into several parts called slices. In Figure 3b, the first captured image frame 311 is divided into two slices, namely the first slice 311a and the second slice 311b. Other image frames, such as the second captured image frame 312, are also divided into slices. Thus, the second captured image frame 312 is divided into two slices, namely the first slice 312a and the second slice 312b.

[0051] According to standard encoding techniques, slices may be encoded and transmitted in a specific order, for example, starting from the upper-left corner and ending at the lower-right corner of the image frame input to encoder 303. Each slice may be placed in its own Network Abstraction Layer (NAL) unit, and each NAL unit may then be packetized into a series of Internet Protocol (IP) packets to be sent over a communication network such as video network system 250.

[0052] However, lost data packets on the communication network can be a problem. The first part of an encoded image frame may have a higher probability of reaching the receiving device through the communication network than later parts. Therefore, later parts of the encoded image frame may be dropped on the communication network. This is not ideal, as surveillance cameras are usually pointed so that the most interesting part of the scene is shown in the center of the video frame.

[0053] One way to solve this problem is to provide an image capture device configured to slice (or tile) an input image into several parts, move each part (or group of parts) to a separate frame arranged in order of priority, and replace empty areas with pre-generated empty slices. Thus, for each encoded frame, the slices may be sent in a first (e.g., normal) order or according to a first set of priorities, such as that specified by the video compression standard, in order to satisfy the requirements for transmitting the slices, while the original slice prioritization is performed at the frame level by incorporating slices of different priorities into different encoded frames associated with the same input frame received from the image sensor, and transmitting the encoded frames associated with the same input frame according to a second order. The second order may be based on the prioritization of the contents of the slices for each input frame. The second order may be different from the first order. The second order may also be called a second priority set. Thus, first and second encoded frames associated with the same input frame are transmitted in a second order, for example, according to a second set of priorities. For example, based on the content of an input slice, the encoded slices are distributed among the set of encoded frames associated with the same input frame, such that the encoded slice with the highest priority according to its content is sent first. Within each encoded frame associated with the same input frame, the slices are sent according to their primary priority, for example, according to a standard specification.

[0054] Next, separate frames can be combined into a single frame on the decoder side, just as normal video frames are combined into a result at a specific time. Empty slices function like empty frames and can be generated like empty frames. An empty frame may be a frame that has no encoded difference with respect to the previous frame of the encoded image stream. In some embodiments, an empty frame may be a pre-computed encoded frame. A pre-computed encoded frame means that, instead of generating encoded frames using encoder 303, the encoded frames are generated in advance by a software algorithm, for example, by encoding all empty frames in the same manner. Thus, an empty frame may be a pre-computed encoded frame that has no encoded difference with respect to the previous frame of the encoded image stream. For example, such a pre-computed empty frame may include one or more macroblocks indicating that there was no change with respect to the previous frame. Such changes may relate to the image content of the previous frame, for example, pixel values. Thus, an empty slice may be a slice that has no encoded difference with respect to the previous slice of the encoded image stream.

[0055] For example, the normal QP value per slice in a slice header according to the H.264 / H.265 standard, or the base QP per frame, may be set to QP 51. For empty frames, e.g., frames without encoded block residuals, the QP is not used by the decoder, and QP 51 can be used as a "flag" to indicate an empty slice if it is never used by the encoder 303. The advantage of using a QP value is that it does not require any extra processing because it is part of the video that is already ready to be used.

[0056] All frames except the last frame in a group of frames created from the same single frame can be marked as hidden to avoid strange visual effects such as partial image updates. Furthermore, motion vector search can be restricted to within each slice / tile only.

[0057] Here, exemplary embodiments of this specification will be described with reference to Figures 4a, 4b, 4c, 4d, 4e, and 5, and further with reference to Figures 1, 2a, 2b, and 3a.

[0058] Figure 4a shows an embodiment of the encoding method in the image capture device 110. In particular, the embodiment may be performed by an encoder 303. The image frame 410 may be divided into a plurality of slices 410a, 410b, 410c, each comprising one or more first slices 410b defined by a first slice parameter and one or more second slices 410a, 410c defined by a second slice parameter. In Figure 4a, one or more first slices 410b are shown with a different filling pattern than one or more second slices 410a, 410c. In the example of Figure 4a, one or more first slices 410b are a single slice, and one or more second slices 410a, 410c include two slices.

[0059] Figure 5 shows a flowchart illustrating how the image capture device 110 performs the encoding of image frames 311 and 410 of the image stream 310 and the transmission of the encoded image frames 411 and 412 over a communication network such as a video network system 250.

[0060] Action 501 The image capture device 110 receives captured image frames 311 and 410 of the image stream 310 from its image sensor 301. For example, an encoder 303 may receive captured image frames 311 and 410 of the image stream 310 from the image sensor 301. The image capture device 110 can receive complete raw frames.

[0061] Action 502 The image capture device 110 divides image frames 311 and 410 into a plurality of slices 410a, 410b, and 410c. The plurality of slices 410a, 410b, and 410c include one or more first slices 410b defined by a first slice parameter and one or more second slices 410a and 410c defined by a second slice parameter. The slice parameter may include the size and position of the slice within the frame.

[0062] Multiple slices 410a, 410b, and 410c may be of equal size. In one embodiment, the input image 410 is divided into three slices of equal size and encoded. For example, multiple slices 410a, 410b, and 410c may further include one or more third slices 410c defined by a third slice parameter.

[0063] Image segmentation can be dynamic to better cover the most important parts of the video, which may change. For example, the number, size, and position of slices may be changed dynamically.

[0064] Action 502 can be performed by encoder 303.

[0065] Action 503 The image capture device 110 prioritizes the multiple slices 410a, 410b, and 410c such that one or more first slices 410b are preferred over one or more second slices 410a and 410c.

[0066] Prioritization may be based on the content of multiple slices 410a, 410b, and 410c. For example, one or more slices may be grouped into the same priority group. For example, one or more second slices 410a and 410c may be grouped into a second priority group having a lower priority than one or more first slices 410b within the first priority group.

[0067] In some embodiments of this specification, a raw frame is divided into three slices and encoded, with the middle slice receiving the highest priority and the other two slices receiving lower priority. Thus, in these embodiments, the middle slice may be included in one or more first slices 410b which have priority over the other two slices included in one or more second slices 410a, 410c.

[0068] In some other embodiments of this specification, prioritizing a plurality of slices 410a, 410b, 410c includes prioritizing one or more second slices 410a over one or more third slices 410c.

[0069] Action 504 Figure 4b shows the first and second encoded image frames 411 and 412.

[0070] The image capture device 110 generates a first encoded image frame 411 which includes one or more first encoded slices 411b defined by a first slice parameter and based on the encoding of one or more first slices 410b, and one or more second encoded slices 411a, 411c defined by a second slice parameter and including one or more first skip blocks.

[0071] A skip block may indicate that, for the portion of the encoded image frames 421 and 422 covered by the skip block, there is no encoded difference between the encoded image frames 421 and 422 containing the skip block and the previous encoded image frames 411 and 412.

[0072] The image capture device 110 further generates a second encoded image frame 412 which includes one or more further second encoded slices 412a, 412c defined by a second slice parameter and based on the encoding of one or more second slices 410a, 410c, and one or more further first encoded slices 412b defined by a first slice parameter and including one or more second skip blocks.

[0073] Generating encoded image frames 411 and 412 may include encoding multiple encoded slices 411a, 411b, and 411c such that each encoded slice 410a, 410b, and 410c is independent of all other encoded slices 411a, 411b, and 411c in the multiple encoded slices 411a, 411b, and 411c.

[0074] Therefore, each encoded slice 411a, 411b, 411c may be independent of other encoded slices 411a, 411b, 411c arising from the same received input image frame 410. For example, in some embodiments of this specification, one or more first encoded slices 411b may be independent of one or more second encoded slices 411a, 411c.

[0075] However, slices within an encoded image frame associated with a different input image frame, for example, a previous or subsequent input image frame, may depend on the previous or subsequent slice. For example, if an object moves across a slice, for one or more second encoded slices 411a, 411c, the encoder cannot use a motion vector pointing to one or more first encoded slices 411b. Instead, the moving object must be filled with I blocks when it enters one or more second encoded slices 411a, 411c. This is more costly in terms of computation and transmission resources, but limits the dependency. As soon as an object is in one or more second encoded slices 411a, 411c, for example, a motion vector may be used so that one or more second encoded slices 411a, 411c depend on a corresponding encoded slice associated with a previous or subsequent image frame, such as a previous or subsequent input frame or a previous or subsequent encoded frame. In embodiments of this specification, a previous or subsequent encoded image frame refers to a set of encoded image frames associated with the same received input frame.

[0076] For example, motion vector search can be restricted to within each slice only. This means there are no predictions across slice boundaries. This prevents errors from propagating between encoded slices.

[0077] The generation of encoded frames is illustrated by the following two approaches:

[0078] 1. The encoder 303 performs two passes on the input image frames 311 and 410, each of which generates encoded output frames 411 and 412 having multiple, for example, three encoded slices, and for each slice, it is requested to either generate actual slice data or fill it with skip blocks.

[0079] The first method is shown in Figure 4c, in which the input frame 410 is encoded twice. That is, the first time it is encoded into the first encoded frame 411, and the second time it is encoded into the second encoded frame 412.

[0080] 2. The frame is encoded into an intermediate encoded image frame 415 using multiple, for example, two, slices. The second approach is shown in Figure 4d. The intermediate encoded image frame 415 may contain multiple encoding slices, such as one or more first encoding slices 415b and one or more second encoding slices 415a, 415c, etc.

[0081] Next, multiple output frames, for example two, are constructed by selecting encoded slice data for some slices and filling in the remaining slices with skip blocks. Thus, the intermediate encoded image frame 415 can be used to generate the first encoded image frame 411 and the second encoded image frame 412.

[0082] Accordingly, in some embodiments of this specification, generating a first encoded image frame 411 includes encoding a plurality of slices 410a, 410b, 410c such that one or more first slices 410b are encoded into one or more first encoded slices 411b of the first encoded image frame 411, and one or more second encoded slices 411a, 411c of the first encoded image frame 411 are generated by replacing one or more second slices 410a, 410c with second skip blocks. Next, generating a second encoded image frame 412 involves encoding a plurality of slices 410a, 410b, 410c such that one or more second slices 410a, 410c are encoded into one or more further second encoded slices 412a, 412c of the second encoded image frame 412, and one or more further first encoded slices 411a, 411c are generated by replacing one or more first slices 410b with one or more first skip blocks.

[0083] In some other embodiments, generating a first encoded image frame 411 and generating a second encoded image frame 412 includes generating an intermediate encoded image frame 415 based on encoding image frames 311, 410 by encoding a plurality of slices 410a, 410b, 410c such that one or more first slices 410b are encoded into one or more first encoded slices 415b of the intermediate encoded image frame 415, and one or more second slices 410a, 410c are encoded into one or more second encoded slices 415a, 415c of the intermediate encoded image frame 415. Next, generating a first encoded image frame 411 further includes inserting one or more first encoded slices 415b of the intermediate encoded image frame 415 into the first encoded image frame 411 and inserting skip blocks in place of one or more second encoded slices 415a, 415c of the intermediate encoded image frame 415, and generating a second encoded image frame 412 further includes inserting one or more second encoded slices 415a, 415c of the intermediate encoded image frame 415 into the second encoded image frame 412 and inserting skip blocks in place of one or more first encoded slices 411b of the intermediate encoded image frame 415 into the second encoded image frame 412.

[0084] Figure 4e shows several embodiments when a plurality of slices 410a, 410b, 410c include one or more third slices 410c defined by a third slice parameter. The generated first encoded image frame 411 may further comprise one or more third encoded slices 411c having one or more third skip blocks defined by a third slice parameter, and the generated second encoded image frame 412 may further comprise one or more third encoded slices 411c.

[0085] These embodiments may further comprise generating a third encoded image frame 413 comprising one or more further third encoded slices 413c based on encoding one or more third slices 410c defined by a third slice parameter, one or more further first encoded slices 412b defined by a first slice parameter and comprising one or more second skip blocks, and one or more second encoded slices 411a defined by a second slice parameter and comprising one or more first skip blocks.

[0086] Action 505 To satisfy the requirement of transmitting encoded frame slices in a specific order not based on the priority of the slice content, the image capture device 110 transmits the encoded slices of each encoded image frame 411, 412 according to a first set of priorities, and transmits the encoded image frames 411, 412 generated from the same received image frame 410 according to a second set of priorities. As described above, the priority of the first set can be based on standard requirements, and the priority of the second set can be based on the content of the slices.

[0087] For example, one or more first encoded slices 411b are based on one or more first slices 410b that take precedence over one or more second slices 410a, 410c, for example, based on the contents of one or more first and second slices 410a, 410b, 410c. Therefore, one or more first encoded slices 411b should take precedence over one or more further second encoded slices 412a, 412c. This is done by transmitting one or more first encoded slices 411b as part of a separate first encoded frame 411 that takes precedence over a second encoded frame 412 containing one or more further second encoded slices 412a, 412c.

[0088] Therefore, the image capture device 110 transmits the first encoded image frame 411 to the receiving devices 260, 270, 280, and 295 on the communication network 250 before the second encoded image frame 412. Thus, since the image capture device 110 transmits the first encoded image frame 411 before the second encoded image frame 412, the image capture device 110 prioritizes the first encoded image frame 411 over the second encoded image frame 412.

[0089] The horizontal arrows in Figures 4b and 4e indicate the transmission time.

[0090] Figure 4b further illustrates how the image capture device 110 transmits a further first encoded image frame 421 before a further second encoded image frame 422, following the same principles as described above for the first encoded image frame 411 and the second encoded image frame 412. The time interval between the first and second sets of encoded image frames 411, 412 and the second set of further first and second encoded image frames 421, 422 may be set according to a specific desired frame rate, for example, 1 / 30th of a second.

[0091] Figure 4e further illustrates how the image capture device 110 transmits a further second encoded image frame 422 before a further third encoded image frame 423, following the same principle as described above.

[0092] The image capture device 110 can transmit a first instruction to the receiving devices 260, 270, 280, and 295 indicating that the first encoded image frame 411 should not be displayed.

[0093] In some embodiments of this specification, the first instruction is transmitted within the header of the first encoded image frame 411.

[0094] In general, all transmitted encoded frames except the last transmitted encoded frame may include, for example, an instruction in the encoded frame header stating that the encoded frame should not be displayed.

[0095] In some other embodiments of this specification, the first encoded frame 411 includes a first instruction indicating that only the last encoded frame 412 among the encoded frames related to the first encoded frame 411 should be displayed.

[0096] This display can be implemented as a flag, such as a 1-bit flag. By transmitting the first instruction, the image capture device 110 helps the receiving device 295 understand how to display the content of the encoded frame.

[0097] In some embodiments of this specification, the image capture device 110 transmits a second instruction to the receiving devices 260, 270, 280, 295 indicating how the image frames 311, 410 are divided into a plurality of slices 410a, 410b, 410c. By transmitting the second instruction, the image partitioning can be dynamic.

[0098] Additionally, the second instruction may be sent in one or more headers.

[0099] If there are three encoded image frames, the method may further comprise transmitting the second encoded image frame 412 to the receiving devices 260, 270, 280, and 295 over the communication network 250 before the third encoded image frame 413.

[0100] On the receiver side, the received encoded frame is decoded as usual. The decoded frames are combined into a single frame to be displayed. For example, the decoding of a second image may reuse blocks from the first decoded image if those blocks are skipped blocks. In other words, the decoded data from the first decoded image frame may be copied to the second decoded image frame, which is then displayed. Therefore, one or more decoded frames may not be displayed. In particular, in some embodiments, only the results after the last frame are displayed, including all slices. This may be indicated, for example, by the first instruction.

[0101] If a second encoded frame is dropped on the network and receivers 260, 270, 280, and 295 know that the first encoded frame should not be displayed, for example, if the first encoded frame contains a first indicator, then receivers 260, 270, 280, and 295 will display the next set of decoded frames correctly in any case.

[0102] Next, a schematic representation of the complete embodiment is shown. The input image is encoded into three slices of equal size. The slice is separated into three distinct frames, and empty areas are replaced with pre-generated empty slices, for example, indicated by a QP value of 51. • Frames are arranged so that the frame with the highest priority slice is placed first. • The first two frames are marked as hidden. Each frame is transmitted as one or more network packets. For example, each slice may result in a packet. The droplet is most likely to collide with frame #2 or #3. • Decode the frames as usual, and only the results after the last frame will be displayed, including all slices of data.

[0103] Accordingly, according to embodiments of this specification, multiple encoded image frames can be generated from a single input frame. The number of encoded image frames may correspond to or be equal to the number of priority groups of slices that make up the frame. Each encoded image frame corresponding to a particular priority group comprises a slice generated by the encoder by encoding the corresponding input slice from the corresponding priority group, and may skip blocks in place of encoded slices from other priority groups. On the receiver side, the multiple encoded frames are decoded, the determined slices are combined, and displayed as usual.

[0104] Therefore, some embodiments of this specification disclose a method in a video camera. The method is - Receiving frame data from the image sensor of the video camera, - Dividing the received frame into several slices, such as two slices, - Setting a priority for the aforementioned slices, - Constructing a first frame from video data in a preferred slice, skipping blocks in a non-preferred slice, for example, the preferred slice being image-encoded, i.e., updated with respect to the preceding video frame. - Constructing a second frame from the video data in the non-priority slice and the skip blocks in the priority slice, - Transmitting the first and second frames to a receiving device over the communication network. It can include this.

[0105] Referring to Figure 6, a schematic block diagram of an embodiment of the image capture device 110 is shown. As described above, the image capture device 110 is configured to encode image frames 311 and 410 of the image stream 310 and transmit the encoded image frames 411 and 412 over a communication network such as the video network system 250.

[0106] As described above, the image capture device 110 may include any of the following: a camera, a video camera 120, a surveillance camera, a surveillance camera or camcorder, a network video recorder, and a wireless communication device 130.

[0107] In particular, the image processing device 110 may be a video camera 120 such as a surveillance camera.

[0108] The image processing device 110 may include an encoder 303 configured to perform one or more of the actions of the method shown in Figure 5. For example, the encoder 303 may be configured to encode an input image frame into at least two encoded image frames based on the method described above in relation to Figure 5.

[0109] The image capture device 110 may further comprise a processing module 601, such as means for performing the method described herein. The means may be embodied in the form of one or more hardware modules and / or one or more software modules.

[0110] The image capture device 110 may further include a memory 602. The memory may include, for example, instructions in the form of a computer program 603, and the instructions may include a computer-readable code unit that, when executed on the image capture device 110, causes the image capture device 110 to perform a method of encoding image frames 311, 410 of the image stream 310 and transmitting the encoded image frames 411, 412 over the communication network 250.

[0111] According to some embodiments of this specification, the image capture device 110 and / or processing module 601 comprises a processing circuit 604 as an exemplary hardware module which may comprise one or more processors. Thus, the processing module 601 may be embodied in the form of the processing circuit 604, or "implemented by" the processing circuit 604. Instructions are executable by the processing circuit 604, thereby enabling the image capture device 110 to operate in the manner of Figure 5 as described above. In another example, instructions, when executed by the image capture device 110 and / or processing circuit 604, can cause the image capture device 110 to perform the manner of Figure 5.

[0112] Taking the above into consideration, in one example, an image capture device 110 is provided for encoding image frames 311 and 410 of an image stream 310 and transmitting the encoded image frames 411 and 412 over a communication network 250.

[0113] In this case as well, the memory 602 contains instructions that can be executed by the processing circuit 604, thereby causing the image capture device 110 to operate in accordance with the method shown in Figure 5.

[0114] Figure 6 further illustrates the carrier 605, or program carrier, which contains the computer program 603 described above. The carrier 605 can be one of the following: an electronic signal, an optical signal, a radio signal, or a computer-readable medium.

[0115] In some embodiments, the image capture device 110 and / or processing module 601 may include, as exemplary hardware modules, one or more of the following: a receiving module 610, a splitting module 620, a prioritizing module 630, an encoded frame generation module 640, and a transmitting module 650. In other examples, one or more of the above-described exemplary hardware modules may be implemented as one or more software modules.

[0116] In particular, the encoder 303 may comprise one or more, or even all, of the following: the receiving module 610, the splitting module 620, the prioritizing module 630, the encoded frame generation module 640, and the transmitting module 650.

[0117] Therefore, the encoder 303 may perform one or more, or all, of the functions of the receiving module 610, the splitting module 620, the prioritizing module 630, the encoded frame generation module 640, and the transmitting module 650.

[0118] Furthermore, the processing module 601 may include an input / output unit 606. According to one embodiment, the input / output unit 606 may include an image sensor configured to capture the aforementioned raw image frames, such as raw image frames included in a first video stream 310 from the image sensor 301.

[0119] According to the various embodiments described above, the image capture device 110 and / or processing module 601 and / or receiving module 610 and / or encoder 303 are configured to receive captured image frames 311, 410 of the image stream 310 from the image sensor 301 of the image capture device 110.

[0120] The image capture device 110 and / or processing module 601 and / or splitting module 520 and / or encoder 303 are configured to split image frames 311, 410 into a plurality of slices 410a, 410b, 410c. The plurality of slices 410a, 410b, 410c include one or more first slices 410b defined by a first slice parameter and one or more second slices 410a, 410c defined by a second slice parameter.

[0121] The image capture device 110 and / or the processing module 601 and / or the prioritization module 630 are configured to prioritize multiple slices 410a, 410b, 410c such that one or more first slices 410b are preferred over one or more second slices 410a, 410c.

[0122] The image capture device 110 and / or processing module 601 and / or encoded frame generation module 640 and / or encoder 303, The first encoded image frame 411 comprises one or more first encoded slices 410b defined by the first slice parameter and based on encoding the one or more first slices 411b, and one or more second encoded slices 411a, 411c defined by the second slice parameter and comprising one or more first skip blocks, The system is configured to generate the second encoded image frame 412, comprising one or more further second encoded slices 411a, 411c defined by the second slice parameter and based on encoding the one or more second slices 410a, 410c, and one or more further first encoded slices 412b defined by the first slice parameter and comprising one or more second skip blocks.

[0123] In some embodiments of this specification, the image capture device 110 and / or processing module 601 and / or encoded frame generation module 640 and / or encoder 303 are: The first encoded image frame 411 is generated by encoding the plurality of slices 410a, 410b, 410c such that one or more first slices 410b are encoded into one or more first encoded slices 411b of the first encoded image frame 411, and one or more second encoded slices 411a, 411c are generated by replacing the one or more second slices 410a, 410c with the second skip blocks, The system is configured to generate the second encoded image frame 411 by encoding the plurality of slices 411a, 411b, 411c such that the one or more second slices 410a, 410c are encoded into the one or more further second encoded slices 410a, 410c of the second encoded image frame 412, and the one or more further first encoded slices 410a, 411c are generated by replacing the one or more first slices 410b with the one or more first skip blocks.

[0124] In some embodiments of this specification, the image capture device 110 and / or processing module 601 and / or encoded frame generation module 640 and / or encoder 303 are further configured to generate a third encoded image frame 413 which includes one or more further third encoded slices 410b defined by a first slice parameter, which includes one or more second skip blocks, which includes one or more second encoded slices 412a defined by a second slice parameter, which includes one or more first skip blocks, based on encoding one or more third encoded slices 413c which includes one or more third encoded slices 411c defined by a third slice parameter.

[0125] The image capture device 110 and / or processing module 601 and / or transmission module 650 and / or encoder 303 are configured to transmit the first encoded image frame 411 to the receiving devices 260, 270, 280, and 295 before the second encoded image frame 412 over the communication network 250.

[0126] The image capture device 110 and / or processing module 601 and / or transmission module 650 and / or encoder 303 may be configured to transmit a first instruction to the receiving devices 260, 270, 280, 295 that the first encoded image frame 411 should not be displayed.

[0127] In some embodiments of this specification, the image capture device 110 and / or the processing module 601 and / or the transmission module 650 and / or the encoder 303 are configured to transmit a first instruction within the header of a first encoded image frame 411.

[0128] The image capture device 110 and / or processing module 601 and / or transmission module 650 and / or encoder 303 may be configured to transmit a second instruction to the receiving devices 260, 270, 280, 295 indicating how the image frames 311, 410 are divided into multiple slices 410a, 410b, 410c.

[0129] As used herein, the term “module” may refer to one or more functional modules, each of which may be implemented as one or more hardware modules and / or one or more software modules and / or a combined software / hardware module. In some examples, a module may represent a functional unit implemented as software and / or hardware.

[0130] As used herein, the terms “computer program carrier,” “program carrier,” or “carrier” may refer to one of the following: electronic signals, optical signals, radio signals, and computer-readable media. In some examples, a computer program carrier may exclude transient propagating signals such as electronic signals, optical signals, and / or radio signals. Thus, in these examples, a computer program carrier may be a non-transient carrier such as a non-transient computer-readable medium.

[0131] As used herein, the term “processing module” may include one or more hardware modules, one or more software modules, or a combination thereof. Any such module, which is a module of hardware, software, or a combination of hardware and software, may be a connection means, providing means, configuring means, responding means, disabling means, etc., as disclosed herein. For example, the expression “means” may refer to a module corresponding to one of the modules listed above with reference to the figures.

[0132] As used herein, the term “software module” can refer to a software application, a dynamic link library (DLL), a software component, a software object, an object by the Component Object Model (COM), a software component, a software function, a software engine, an executable binary software file, and the like.

[0133] The terms “processing module” or “processing circuit” as used herein may encompass a processing unit including, for example, one or more processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. A processing circuit, etc., may comprise one or more processor kernels.

[0134] Where used herein, the expression “configured to do so” may mean that a processing circuit is configured, for example adapted or operable by a software configuration and / or hardware configuration to perform one or more of the actions described herein.

[0135] As used herein, the term “action” can refer to a movement, step, operation, response, reaction, activity, etc. Note that, where applicable, an action may be divided into two or more sub-actions. Furthermore, note that, where applicable, two or more movements described herein may be merged into a single action.

[0136] As used herein, the term "memory" may refer to hard disks, magnetic storage media, portable computer diskettes or disks, flash memory, random access memory (RAM), etc. It may also refer to the internal register memory of a processor, etc.

[0137] As used herein, the term “computer-readable media” may include, but is not limited to, Universal Serial Bus (USB) memory, DVD-Discs, Blu-ray Discs, software modules received as a stream of data, flash memory, hard drives, memory sticks, multimedia cards (MMC), secure digital (SD) cards, and other memory cards. One or more of the aforementioned examples of computer-readable media may be provided as one or more computer program products.

[0138] As used herein, the term “computer-readable code unit” can be the text of a computer program, part or all of a binary file representing a computer program in a compiled format, or any part or part of these.

[0139] As used herein, the terms “number” and / or “value” may refer to any type of number, such as a binary number, a real number, an imaginary number, or a rational number. Furthermore, “number” and / or “value” may refer to one or more characters, such as a single character or a string of characters. “Number” and / or “value” may also be represented by a string of bits, i.e., 0s and / or 1s.

[0140] Where used herein, the phrase "in some embodiments" is used to indicate that features of the embodiments described may be combined with any other embodiments disclosed herein.

[0141] While various embodiments have been described, many different changes and modifications will be apparent to those skilled in the art. Therefore, the embodiments described are not intended to limit the scope of this disclosure.

Claims

1. A method for encoding image frames of an image stream and transmitting the encoded image frames over a communication network, performed by an image capture device, The image sensor of the image capture device receives an image frame from the image stream, The image frame is divided into a plurality of slices, each of which includes one or more first slices defined by a first slice parameter and one or more second slices defined by a second slice parameter. Prioritize the plurality of slices such that the one or more first slices have priority over the one or more second slices. A first encoded image frame, One or more first encoded slices based on the encoding of one or more first slices defined by the first slice parameter, and Includes one or more second coding slices that include one or more first skip blocks defined by the second slice parameter, The first slice parameter includes at least one of a first slice size or a first slice position in the first encoded image frame, and further The second slice parameter includes at least one of a second slice size or a second slice position in the first encoded image frame, wherein at least one of the second slice size or second slice position generates a first encoded image frame that is different from the corresponding first slice size or first slice position. A second encoded image frame is generated, which includes one or more further second encoded slices based on the encoding of one or more second slices defined by the second slice parameter, and one or more further first encoded slices including one or more second skip blocks defined by the first slice parameter. A method for transmitting a first encoded image frame to a receiving device on a communication network prior to a second encoded image frame.

2. The method according to claim 1, further comprising transmitting a first instruction to the receiving device indicating that the first encoded image frame is not to be displayed.

3. The method according to claim 2, wherein the transmission of the first instruction is further comprising transmitting the first instruction in the header of the first encoded image frame.

4. The method according to claim 1, wherein the generation of the encoded image frame comprises encoding the plurality of slices such that each of the plurality of encoded slices is independent of all the other encoded slices of the plurality of encoded slices.

5. The method according to claim 1, further comprising transmitting a second instruction to the receiving device indicating how the image frame is divided into the plurality of slices.

6. The generation of the first encoded image frame comprises encoding a plurality of slices, wherein one or more first slices are encoded into one or more first encoded slices of the first encoded image frame, and one or more second slices are generated by replacing the second skip blocks with the first or more second encoded slices. The method according to claim 1, wherein the generation of the second encoded image frame comprises encoding a plurality of slices, wherein one or more second slices are encoded into one or more further second encoded slices of the second encoded image frame, and one or more further first encoded slices are generated by replacing one or more first slices with one or more first skip blocks.

7. The generation of the first encoded image frame and the generation of the second encoded image frame are performed by encoding an image frame by encoding a plurality of slices to generate an intermediate encoded image frame, wherein one or more first slices are encoded into one or more first encoded slices of the intermediate encoded image frame, and one or more second slices are encoded into one or more second encoded slices of the intermediate encoded image frame. The generation of the first encoded image frame further involves inserting one or more first encoded slices of intermediate encoded image frames into the first encoded image frame, and inserting one or more first skip blocks in place of one or more second encoded slices of intermediate encoded image frames. The method according to claim 1, wherein the generation of a second encoded image frame further includes inserting one or more second encoded slices of intermediate encoded image frames into the second encoded image frame, and inserting one or more second skip blocks into the second encoded image frame in place of one or more first encoded slices of intermediate encoded image frames.

8. The plurality of slices includes one or more third slices defined by a third slice parameter, Prioritizing the plurality of slices includes prioritizing one or more second slices over one or more third slices, The generated first encoded image frame further includes one or more third encoded slices defined by a third slice parameter and containing one or more third skip blocks, The generated second encoded image frame further includes one or more third encoded slices, and the method further A third encoded image frame is generated, comprising one or more further third encoded slices defined by a third slice parameter and generated based on the encoding of one or more third slices, and one or more further first encoded slices defined by a first slice parameter and containing one or more second skip blocks, and one or more second encoded slices defined by a second slice parameter and containing one or more first skip blocks. The method according to claim 1, wherein a second encoded image frame is transmitted to a receiving device on a communication network prior to a third encoded image frame.

9. The method according to claim 1, wherein each skip block of one or more first skip blocks and one or more second skip blocks indicates that there is no encoded difference between the encoded image frame containing the skip block and the previous encoded image frame with respect to a portion of the encoded image frame covered by the skip block.

10. The method according to claim 1, wherein the plurality of slices are of equal size.

11. A non-temporary computer program storage device including a computer-readable code unit that, when executed on an image processing device, causes the image processing device to perform the method described in Claim 1.

12. An image processing apparatus, A processor configured to perform a method for encoding image frames of an image stream and transmitting the encoded image frames over a communication network, wherein the processor The image sensor of the image capture device receives an image frame from the image stream. The aforementioned image frame is divided into multiple slices, The plurality of slices includes one or more first slices defined by a first slice parameter and one or more second slices defined by a second slice parameter. Prioritize the aforementioned multiple slices such that one or more first slices have priority over one or more second slices. A first encoded image frame, One or more first encoded slices defined by a first slice parameter and based on the encoding of one or more first slices, It includes one or more second coding slices that are defined by a second slice parameter and contain one or more first skip blocks, The first slice parameter includes at least one of a first slice size or a first slice position in the first encoded image frame, The second slice parameter includes at least one of a second slice size or a second slice position in the first encoded image frame, wherein at least one of the second slice size or the second slice position generates a first encoded image frame that is different from the one corresponding to at least one of the first slice size or the first slice position. A second encoded image frame, It includes one or more further second coded slices based on the coding of one or more second slices, defined by a second slice parameter, Generate a second encoded image frame containing one or more further first encoded slices, which are defined by the first slice parameter and contain one or more second skip blocks. An image processing device configured to transmit a first encoded image frame to a receiving device prior to a second encoded image frame over a communication network.

13. The image processing apparatus according to claim 12, wherein the processor is further configured to transmit a first instruction to a receiving device indicating that the first encoded image frame is not to be displayed.

14. The image processing apparatus according to claim 12, wherein the image processing apparatus is a video camera.

Images