Method and digital video camera for forming a combined image frame of a combined video stream - Patents.com

Abstract

To provide a method for forming a combined image frame of a combined video stream.SOLUTION: In the method, an image frame of a first video stream is encoded into first encoded data comprising a plurality of rows, each of which has a height of a single coding unit and a width equal to a width of the image frame and is encoded as one or more slices. An image frame of a second video stream is encoded into second encoded data comprising a plurality of rows. Each of the rows has a height of a single coding unit and a width equal to a width of the image frame and is encoded as one or more slices, and rows of the first encoded data of the image frame of the first video stream and rows of the second encoded data of the image frame of the second video stream are interleaved into combined encoded data.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to the formation of combined image frames of a combined video stream. [Background technology]

[0002] In recent years, the number of surveillance cameras, such as network camera surveillance systems, has increased. Many surveillance cameras use multiple image sensors to capture images of a scene to cover a wide area of ​​the scene and / or capture more detail of the scene. Due to the large amount of image data in the video stream generated by each of the image sensors, the image data from each of the image sensors is typically handled separately on separate video processing circuits. Each such video processing circuit includes, for example, an encoder for separately encoding each of the video streams captured by each of the image sensors. Thus, separate video streams are generated from the different image sensors.

[0003] Therefore, there is a need to be able to generate a combined video stream from separate video streams. The combined video stream should be decodable by a standard decoder. One example of generating such a combined video stream is disclosed by Axis AB in EP18190932. However, the solution disclosed in EP18190932 may face problems using codecs such as H.264 and MJPEG. Therefore, an alternative way of generating a combined video stream from separate video streams is required. Summary of the Invention

[0004] In view of the above, it is therefore an object of the present invention to overcome or at least mitigate the above problems, in particular to exploit the concept of slices in encoding, such as in H.264, to generate a single encoded image stream from multiple image streams captured by multiple image sensors.

[0005] According to a first aspect, a digital video camera is provided, comprising a first image sensor configured to capture image frames of a first video stream, a second image sensor configured to capture image frames of a second video stream, a first encoder configured to encode image data for the image frames of the first video stream, where each of the image frames of the first video stream is encoded into first encoded data comprising a plurality of rows, each row having a height of a single coding unit and a width equal to a width of the image frame, and encoded as one or more slices, and a second encoder configured to encode image data for the image frames of the second video stream, where each of the image frames of the second video stream is encoded into first encoded data comprising a plurality of rows, each row having a height of a single coding unit and a width equal to a width of the image frame, and encoded as one or more slices. and a data combination unit configured to combine the first encoded data for the image frames of the first video stream and the second encoded data for the image frames of the second video stream into combined encoded data comprising rows of first encoded data for the image frames of the first video stream and rows of second encoded data for the image frames of the second video stream, each row having a height of a single coding unit and a width equal to a width of an image frame, and configured to combine the first encoded data for the image frames of the first video stream and the second encoded data for the image frames of the second video stream into combined encoded data by interleaving the rows of first encoded data for the image frames of the first video stream and the rows of second encoded data for the image frames of the second video stream, such that the combined encoded data represents a combined image frame of the combined video stream.

[0006] The decoder is configured to decode and display the video stream. However, for interoperability reasons, it is generally not possible for the decoder to read information from two different sources and present it as a single video stream. The decoder is configured to follow a raster scan, where data is read row by row following a Z pattern. The disclosed digital video camera presents one solution, where each image frame (from one image sensor) is encoded by a separate encoder. Each encoder is configured to encode each image frame in rows, where each row is one coding block high and has the width of the image frame. Each row is encoded as one or more slices, i.e., a single row may contain one or more slices. Following encoding by the separate encoders, the encoded data from the separate encoders are interleaved and combined to form a combined image frame. Thus, the encoded data from the separate encoders is shuffled so that the decoder sees the information as multiple rows from a single combined image frame. Such combined image frames may be decoded and displayed by a decoder. Thus, the disclosed digital video camera allows a combined video stream to be generated from separate video streams captured by separate image sensors and processed by separate video processing circuits, each of which includes an encoder. In particular, the encoding (and subsequent decoding) may be embedded according to the H.264 or MJPEG codec. However, it is understood that other encoding codecs may be used. Some non-limiting examples include H.265, H.266, EVE, VP9, ​​or AV1. The disclosed digital video camera utilizes a slicing concept for encoding. Each of the encoders is configured to slice each of the video streams into a large number of amounts.These slices are set to have a height of only one coding unit.

[0007] Each row in the combined encoded data may include a single row of the first encoded data and a single row of the second encoded data.

[0008] The first and second encoders may be configured to encode image data of the first and second video streams in parallel, where in parallel means that the first and second encoders may process different data at the same time.

[0009] The first and second image sensors may be configured to synchronously capture the first and second video streams.

[0010] The combined image frames may represent a panoramic image.

[0011] The data combination unit may be configured to output the combined encoded data as a single bitstream.

[0012] The first and second encoders may be configured to encode image data for the first and second image frames, such that each row includes a number of slices, where each slice has a height of one single coding unit.

[0013] The first and second encoders may be configured to encode image data of the first and second image frames, each of the rows constituting a single slice. Such encoding may reduce the amount of metadata for the combined image frames, which may reduce the video size, because the number of slice headers may be reduced.

[0014] The data combining unit may be configured to check whether two or more immediately following slices in the combined encoded data include only skip encoded coding units. If so, the data combining unit may be configured to replace the two or more immediately following slices that include only skip encoded coding units with a new slice that includes the skip encoded coding unit and has a length of the two or more immediately following slices that include only skip encoded coding units. Such replacement of subsequent slices that include only skip encoded coding units may reduce the amount of metadata of the combined image frame, which may reduce the video size, since the number of slice headers may be reduced.

[0015] According to a second aspect, there is provided a method for forming a combined image frame of a combined video stream, comprising: capturing an image frame of a first video stream with a first image sensor of a multi-sensor camera; capturing an image frame of a second video stream with a second image sensor of the multi-sensor camera; encoding image data of the image frames of the first video stream with a first encoder of the multi-sensor camera, where each of the image frames of the first video stream is encoded into first encoded data including a plurality of rows, each row having a height of a single coding unit and a width equal to a width of the image frame, and encoded as one or more slices; encoding image data of the image frames of the second video stream with a second encoder of the multi-sensor camera, where each of the image frames of the second video stream is encoded into first encoded data including a plurality of rows, each row having a height of a single coding unit and a width equal to a width of the image frame, and encoded as one or more slices. and combining, by a data combination unit, the first encoded data for the image frames of the first video stream and the second encoded data for the image frames of the second video stream into combined encoded data by interleaving the rows of the first encoded data for the image frames of the first video stream with the rows of the second encoded data for the image frames of the second video stream, such that the combined encoded data represents a combined image frame of the combined video stream.

[0016] The combining may be performed such that each row in the combined encoded data includes a single row of the first encoded data and a single row of the second encoded data.

[0017] The encoding of the image data of the first video stream and the image data of the second video stream may be performed in parallel.

[0018] The capturing of the first and second video streams may be performed synchronously.

[0019] The method may include checking whether two or more immediately following slices in the combined encoded data contain only skip-encoded coding units, and if so, replacing the two or more immediately following slices that contain only skip-encoded coding units with a new skip slice that contains the skip-encoded coding unit and has a length of the two or more immediately following slices that contain only skip-encoded coding units.

[0020] Furthermore, the above features of the digital video camera, where applicable, also apply to this second aspect, see above in order to avoid unnecessary repetition.

[0021] According to a third aspect, there is provided a non-transitory computer readable storage medium having stored thereon instructions which, when executed on a device having processing capability, perform the method according to the second aspect.

[0022] Further scope of applicability of the present invention will become apparent from the following detailed description. It should be understood, however, that the detailed description and specific examples, while indicating preferred embodiments of the present invention, are given for purposes of illustration only, since various changes and modifications within the scope of the invention will become apparent to those skilled in the art from this detailed description.

[0023] Thus, it is to be understood that the invention is not limited to the particular components of the described devices or the particular operation of the described methods, as such devices and methods may vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting. It should be noted that, as used in this specification and the appended claims, the articles "a," "an," "the," and "said" are intended to mean that there are one or more elements, unless the context clearly dictates otherwise. Thus, for example, reference to "a unit" or "the unit" may include several devices, and the like. Further, the words "comprising," "including," "containing," and the like do not exclude other elements or steps. [Brief description of the drawings]

[0024] These and other aspects of the invention will now be described in more detail with reference to the accompanying drawings, which should not be considered limiting, but are instead used for purposes of explanation and understanding.

[0025] As shown in these figures, the sizes of each layer and region may be exaggerated for illustrative purposes and are therefore provided to show the general structure, and like reference symbols refer to like elements throughout the figures.

[0026] [Figure 1] FIG. 1 shows a digital video camera configured to generate a combined video stream. [Diagram 2] FIG. 2 shows an alternative digital video camera configured to generate a combined video stream. [Diagram 3]FIG. 3 illustrates the concept of combining the first and second encoded data into a combined encoded data by interleaving rows of the first encoded data of an image frame of the first video stream with rows of the second encoded data of an image frame of the second video stream. [Figure 4] FIG. 4 illustrates the concept of replacing two or more immediately subsequent slices in the combined encoded data of FIG. 3 that contain only skip-encoded coding units with a new slice that contains skip-encoded coding units and has a length of two or more immediately subsequent skips that contain only skip-encoded coding units. [Diagram 5] FIG. 5 is a block diagram of a method for forming a combined image frame of a combined video stream. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] The present invention will now be described in more detail with reference to the accompanying drawings, in which there are shown presently preferred embodiments of the invention. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth below. Rather, these embodiments are provided for completeness and completeness, and to fully convey the scope of the invention to those skilled in the art.

[0028] Methods for encoding video, such as Motion JPEG (MJPEG), Advanced Video Coding (AVC) (also called H.264), and High Efficiency Video Coding (HEVC) (also called H.265), introduce the concept of slices. Slices allow for dividing an image into separate regions that can be decoded / encoded independently. These separate regions may be a grid of rectangular regions. To be able to code independently, no prediction should use pixels from more than one slice, and results from predictions should only be used in the same slice. This concept may be used for parallel processing purposes. This concept may also be used as described herein to realize a single encoded image in an imaging device that includes multiple image sensors and multiple encoders, i.e., stitching image data encoded by multiple encoders into one encoded image.

[0029] FIG. 1 illustrates a digital video camera 100 implementing the present concept. The digital video camera 100 includes a first image sensor 102a and a second image sensor 102b. However, it is understood that the digital video camera 100 may include more than two image sensors 102. Each of the image sensors 102 is configured to capture a respective image frame of a video stream. Each of the video streams depicts a respective scene. The image sensors 102 may be configured to capture different parts of the same scene. Thus, when the video streams captured by the first and second image sensors 102a-b are stitched together, a panoramic video stream may be formed. This is illustrated in FIG. 1. As a result, each of the image sensors 102 may be pointed in the same direction from the digital video camera 100. It should be noted that any type of image sensor 102 may be used for the purposes described herein, such as, for example, image sensors 102 employing visible or infrared (IR) light, thermal sensors, time-of-flight (ToF) sensors, etc.

[0030] The first and second image sensors 102a-b may be configured to capture the first and second video streams synchronously, i.e. the first and second image sensors 102a-b may be configured to capture the first and second video streams substantially simultaneously. According to one example, the image frames are captured synchronously such that a moving object that is partially depicted in the image frame captured by the first image sensor 102a and partially depicted in the image frame captured by the first image sensor 102b is correctly depicted. Thus, the image frames of a pair of image frames, where the first image frame of the pair originates from the first video stream and the second image frame of the pair originates from the second video stream, may be captured synchronously, i.e. substantially simultaneously.

[0031] The digital video camera 100 may include a time controller 302 in communication with each of the image sensors 102. The time controller 302 may be configured to synchronize image capture settings between the image sensors 102. For example, the time controller 302 may be configured to synchronize the points in time at which the image frames of the pair are captured.

[0032] The first image sensor 102a is associated with a first encoder 106a. The first encoder 106a is configured to receive image data 104a of an image frame of a first video stream captured by the first image sensor 102a. The first encoder 106a is configured to encode each of the image frames of the first video stream into first encoded data. In other words, the first encoder 106a is configured to encode the image data 104a captured by the first image sensor 102a. The first encoder 106a is configured to encode each of the image frames of the first video stream such that the first encoded data comprises a number of rows. The first encoder 106a is configured to perform the encoding such that each of the rows has a height of a single coding unit and a width equal to the width of an image frame of the first video stream. The coding unit may be, for example, a macroblock or a coding tree block. The size of the coding unit may depend on the encoding codec used. In the case of H.264, a coding unit of size 16x16 pixels may be used. In the case of MJPEG, a coding unit of size 8x8 pixels may be used. In the case of H.265, the size of the coding unit is flexible, but for example 32x32 pixels or 64x64 pixels may be used. In the case of H.266 or AV1, the size of the coding unit is flexible, but for example 128x126 pixels or 256x256 pixels may be used. Each of the rows is encoded as one or more slices. Each of the rows may constitute a single slice. Thus, each of the rows may be one slice. Alternatively, each of the rows may include multiple slices.

[0033] The second image sensor 102b is associated with a second encoder 106b. The second encoder 106b is configured to receive image data 104b of image frames of the second video stream captured by the second image sensor 102b. The second encoder 106b is configured to encode each of the image frames of the second video stream into second encoded data. In other words, the second encoder 106b is configured to encode the image data 104b captured by the second image sensor 102b. The second encoder 106b is configured to encode each of the image frames of the second video stream such that the second encoded data comprises a number of rows. The second encoder 106b is configured to perform the encoding such that each of the rows has a height of a single coding unit and a width equal to the width of an image frame of the second video stream. The coding unit may be, for example, a macroblock or a coding tree block. The size of the coding unit may depend on the encoding codec used. In the case of H.264, a coding unit of size 16x16 pixels may be used. In the case of MJPEG, a coding unit of size 8x8 pixels may be used. In the case of H.265, the size of the coding unit is flexible, but for example 32x32 pixels or 64x64 pixels may be used. In the case of H.266 or AV1, the size of the coding unit is flexible, but for example 128x126 pixels or 256x256 pixels may be used. Each of the rows is encoded as one or more slices. Each of the rows may constitute a single slice. Thus, each of the rows may be one slice. Alternatively, each of the rows may include multiple slices.

[0034] The first and second encoders 106a-b are configured to encode using coding units of the same size, and thus both the first encoder 106a and the second encoder 106b may be configured to encode using macroblocks of size 16x16 pixels, or coding units of any other suitable size.

[0035] The first and second encoders 106a-b may be configured to encode the image data 104a-b of the first and second video streams synchronously, i.e. substantially simultaneously. In other words, the first and second encoders 106a-b may be configured to operate in parallel. Furthermore, the first and second encoders 106a-b may be configured to operate independently of each other, since the first encoder 106a is configured to encode the image data 104a of the first video stream and the second encoder 106b is configured to encode the image data 104b of the second video stream.

[0036] The first encoder 106a may be implemented as hardware or software, or a combination thereof. The first encoder 106a may be implemented, at least in part, as a hardware circuit including logical hardware circuit components. The first encoder 106a may be implemented, at least in part, as software implemented as program code running on a processor.

[0037] The second encoder 106b may be implemented as hardware or software, or a combination thereof. The second encoder 106b may be implemented, at least in part, as a hardware circuit including logical hardware circuit components. The second encoder 106b may be implemented, at least in part, as software implemented as program code running on a processor.

[0038] The digital video camera 100 further includes a data combination unit 110. The data combination unit 110 is configured to receive the first encoded data 108a encoded by the first encoder 106a and the second encoded data 108b encoded by the second encoder 106b. The data combination unit 110 is further configured to combine the first and second encoded data 108a-b into a pair of first and second video streams. Preferably, the image frames of the pair of first and second video streams are captured synchronously, i.e., the image frames of the pair of image frames are captured substantially simultaneously, respectively. The data combination unit 110 is configured to combine the first and second encoded data 108a-b into a combined encoded data 112. The combined encoded data 112 represents a combined image frame of the combined video stream. Thus, the data combination unit 110 is configured to splice the first and second image streams into a combined video stream. The combined video stream is an encoded and combined video stream. The encoded and combined video stream may represent a panoramic video stream.

[0039] The data combination unit 110 may be implemented as hardware or software, or a combination thereof. The data combination unit 110 may be implemented, at least in part, as a hardware circuit including logical hardware circuit components. The data combination unit 110 may be implemented, at least in part, as software implemented as program code running on a processor.

[0040] The data combination unit 110 may be implemented as such for the purpose of a dedicated unit, as shown in Figure 1. Alternatively, the data combination unit 110 may be implemented in the first encoder 106a, as shown in Figure 2. Thus, the first encoder 106a may be a master encoder and the second encoder 106b may be a slave encoder.

[0041] The data combination unit 110 is configured to combine the first and second encoded data 108a-b into combined encoded data 112 by interleaving rows of the first encoded data 108a of the image frames of the first video stream with rows of the second encoded data 108b of the image frames of the second video stream, as will be explained in more detail below in relation to FIG.

[0042] In the upper part of Fig. 3, a representation of the first and second encoded data 108a-b is shown. Both a representation of an image frame and a representation of a bit stream are shown. A representation of an image frame of the first encoded data 108a, which represents an image frame of the first image stream encoded by the first encoder 106a, is shown on the left. A representation of an image frame of the second encoded data 108b, which represents an image frame of the second image stream encoded by the second encoder 106b, is shown on the right. The first encoded data 108a comprises a number of rows R11-R15. The second encoded data 108b comprises a number of rows R21-R25. Each of the rows R11-R15, R21-R25 has a height of a single coding unit and a width equal to the respective width of an image frame. The width of the image frame of the first video stream and the width of the image frame of the second video stream may be equal. This is the case in Fig. 3. However, it is understood that the width of the image frame of the first video stream may be different from the width of the image frame of the second video stream. The coding unit may be, for example, a macroblock. The coding unit may have a size of 16x16 pixels. As in the example shown in Figure 3, each of the rows may constitute a single slice. Thus, each of the rows may be one slice. Alternatively, each of the rows may include multiple slices. However, the multiple slices of one row are treated as respective rows of the encoded image frame.

[0043] Directly below the representation of the image frame of the first encoded data 108a is a representation of the first encoded data 108a in the form of a bitstream. In the bitstream diagram, the rows R11 to R15 of the first encoded data are arranged in a sequential order. Directly below the representation of the image frame of the second encoded data 108b is a representation of the second encoded data 108b in the form of a bitstream. In the bitstream diagram, the rows R21 to R25 of the second encoded data are arranged in a sequential order.

[0044] As mentioned above, the data combination unit 110 is configured to combine the first and second encoded data 108a-b into the combined encoded data 112. A representation of the combined encoded data 112 is shown at the bottom of FIG. 3. Both an image frame representation and a bit stream representation are shown. As can be seen in the image frame representation of the combined encoded data 112, each row of the combined encoded data 112 includes a single row of the first encoded data and a single row of the second encoded data. This is because the data combination unit 110 is configured to form the combined encoded data 112 by interleaving rows of the stream of the first encoded data 108a and rows of the second encoded data 108b. Thus, as can be seen in the bit stream representation of the combined encoded data 112, a row from the first encoded data is followed by a row of the second encoded data.

[0045] As a result, the data combination unit 110 is configured to combine the first encoded data 108a in the form of a bitstream received from the first encoder 106a and the second encoded data 108b in the form of a bitstream received from the second encoder 106b into combined encoded data 112, represented as a single bitstream.

[0046] The data combination unit 110 may further be configured to modify metadata of the combined encoded data 112, in particular such that the metadata of the combined encoded data 112 indicates that the width of the image frame represented by the combined encoded data 112 has the combined width of the image frame of the first video stream and the image frame of the second video stream. The metadata may for example be a header of the combined encoded data 112.

[0047] The data combination unit 110 may further be configured to modify metadata of slices of the combined encoded data 112. In particular, a start position of the slice in the combined encoded data 112 is indicated in the metadata of each slice in the combined encoded data 112. The start position of each slice may for example be given in terms of a number of coding units.

[0048] The data combination unit 110 may further be configured to insert a restart marker at the end of the rows R11 to R15 of the first encoded data, which is particularly required if the encoding is performed according to the MJPEG codec, to indicate that the MJPEG image frame does not stop at the end of the rows R11 to R15 of the first encoded data.

[0049] As described herein in connection with FIG. 4, the data combining unit 110 may further be configured to check whether two or more immediately following slices in the combined encoded data 112 contain only skip encoded coding units. Such slices are hereinafter referred to as skip encoded slices. Checking whether a slice in the combined encoded data 112 contains only skip encoded coding units may be performed in different ways. For example, the size of the slice may be checked and compared to a threshold size. If the size of the slice is less than the threshold size, the slice may be considered as a skip encoded slice. According to another example, metadata may be read that contains information about the slice containing only skip encoded coding units. According to a further alternative, the slice may be compared with known skip encoded slices. If two or more immediately following slices in the combined encoded data 112 contain only skip encoded coding units, the data combining unit 110 is configured to replace the two or more immediately following skip encoded slices with a new skip encoded slice. The new skip encoded slice contains only skip encoded coding units. The new skip encoded slice has the length of the two or more immediately following skip encoded slices. In the example shown in FIG. 4, it is assumed that each of the rows R11-15, R21-R25 of the first and second encoded data 108a-b is a single slice. The top row R11 and the third row R13 of the first encoded data 108a are skip encoded slices. Furthermore, the second row R22 and the third row R23 of the second encoded data 108b are skip encoded slices.Subsequently, in the combined encoded data, the three immediately following skip encoded slices R22, R13, and R23 may be replaced by a new skip encoded slice S', which has a length that is the combined length of R22, R13, and R23. In this manner, the number of skip encoded slices may be reduced by combining them, thereby reducing the number of slice headers. As a result, the file size of the combined encoded data 112 may be reduced. The skip encoded slices contain substantially no data other than the headers.

[0050] 5, a method 500 for forming a combined image frame of a combined video stream is described. Some of the steps of method 500 may be performed by digital video camera 100. Digital video camera 100 may also be referred to as a multi-sensor video camera because digital video camera 100 includes two or more image sensors. However, it is equally understood that some of the steps of method 500 may be performed by another device. Method 500 includes the following steps. These steps may be performed in any suitable order.

[0051] Capturing an image frame of a first video stream S502. The image frame of the first video stream is captured by a first image sensor 102a of the digital video camera 100.

[0052] Capturing image frames of the second video stream S504. The image frames of the second video stream are captured by the second image sensor 102b of the digital video camera 100.

[0053] Capturing image frames S502, S504 of the first and second video streams may be synchronized. Thus, capturing image frames S502, S504 of the first and second video streams may be synchronized. That is, capturing S502, S504 may occur substantially simultaneously. The first and second video streams may depict different portions of a common scene. Thus, after splicing the first and second video streams into a combined video stream, a panoramic video stream may be formed.

[0054] Encoding image data of the image frames of the first video stream S506. The encoding S506 is performed by the first encoder 106a described above. Each of the image frames of the first video stream is encoded into first encoded data including a number of rows. Each row has a height of a single coding unit and a width equal to the width of the image frame. Furthermore, each row is encoded as one or more slices.

[0055] Encoding S508 image data of the image frames of the second video stream. The encoding S508 is performed by the second encoder 106a described above. Each of the image frames of the second video stream is encoded into second encoded data including a number of rows. Each row has a height of a single coding unit and a width equal to the width of the image frame. Furthermore, each row is encoded as one or more slices.

[0056] Combining S510 the first encoded data of the image frames of the first video stream and the second encoded data of the image frames of the second video stream into combined encoded data. The combining S510 may be performed by the data combination unit 110 described above. The combining S510 is performed by interleaving rows of the first encoded data of the image frames of the first video stream and rows of the second encoded data of the image frames of the second video stream. The combined encoded data represents a combined image frame of the combined video stream.

[0057] The encoding S506, S508 may be performed according to H.264, H.265, or MJPEG.

[0058] The combining S510 may be performed such that each row in the combined encoded data includes a single row of the first encoded data and a single row of the second encoded data.

[0059] The encoding of the image data of the first video stream and the image data of the second video stream S506, S508 may be performed in parallel, i.e., the encoding of S506, S508 may be performed substantially simultaneously.

[0060] Method 500 may further include checking whether two or more immediately following slices in the combined encoded data include only skip encoded coding units. If so, method 500 may further include replacing S512 the two or more immediately following slices that include only skip encoded coding units with a new slice that includes only skip encoded coding units. Thus, all coding units of the new slice are skip encoded. The new skip encoded slice may have a length of the two or more immediately following skip encoded slices.

[0061] Those skilled in the art will understand that the present invention is in no way limited to the preferred embodiments described above, but on the contrary, many modifications and variations are possible within the scope of the claims.

[0062] For example, three or more video streams captured by three or more image sensors and encoded by three or more encoders may be combined into a combined video stream using the present invention. If four video streams captured by four image sensors and encoded by four encoders are used, each row in the combined encoded data may include a single row of encoded data from each of the four video streams.

[0063] Moreover, different image sensors may be arranged next to each other in substantially the same plane. However, according to some embodiments, at least some of the image sensors may be arranged in an angled setup compared to each other. For example, one of the image sensors may be at a 90° angle compared to another one of the image sensors. According to one specific embodiment, a digital video camera may include four image sensors arranged in a rectangular or quadratic setup.

[0064] Additionally, variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

Claims

1. A digital video camera, a first image sensor (102a) configured to capture image frames of a first video stream; a second image sensor (102b) configured to capture image frames of the second video stream; a first encoder (106a) configured to encode image data (104a) of the image frames of the first video stream, each of the image frames of the first video stream being encoded into first encoded data (108a) comprising a plurality of rows (R11 to R15), each of the rows having a height of a single coding unit and a width equal to a width of the image frame, and encoded as one or more slices; a second encoder (106b) configured to encode image data (104b) of the image frames of the second video stream, each of the image frames of the second video stream being encoded into second encoded data (108b) comprising a plurality of rows (R21 to R25), each of the rows having a height of a single coding unit and a width equal to a width of the image frame, and encoded as one or more slices; a data combination unit (110) configured to combine the first encoded data (108a) of the image frames of the first video stream and the second encoded data (108b) of the image frames of the second video stream into combined encoded data (112) by interleaving rows of the first encoded data of the image frames of the first video stream and rows of the second encoded data of the image frames of the second video stream, such that the combined encoded data represents a combined image frame of a combined video stream; Including, The data combination unit (110) is configured to check whether two or more slices immediately following in bitstream order in the combined encoded data contain only skip encoded coding units, and if so, to replace the two or more slices immediately following in bitstream order that contain only the skip encoded coding units with a new slice including a skip encoded coding unit having a length of the two or more slices immediately following in bitstream order that contain only the skip encoded coding units.

2. 2. The digital video camera of claim 1, wherein rows in the combined encoded data include a single row of the first encoded data and a single row of the second encoded data.

3. 3. The digital video camera of claim 1 or 2, wherein the first and second encoders (106a-b) are configured to encode image data of the first and second video streams in parallel.

4. 4. The digital video camera of claim 1, wherein the first and second image sensors (102a-b) are configured to synchronously capture the first and second video streams.

5. 5. The digital video camera of claim 1, wherein the data combination unit (110) is configured to output the combined and encoded data as a bitstream.

6. 6. A digital video camera as claimed in claim 1, wherein the first and second encoders (106a-b) are configured to encode image data for image frames of the first and second video streams, each of the rows including a plurality of slices, each of the slices having a height of one single coding unit.

7. 6. A digital video camera as claimed in any one of claims 1 to 5, wherein the first and second encoders (106a-b) are configured to encode image data for image frames of the first and second video streams, each of the rows constituting a single slice.

8. 8. The digital video camera of claim 1, wherein the combined image frames represent a panoramic image.

9. 1. A method for forming a combined image frame of a combined video stream, comprising: capturing an image frame of a first video stream with a first image sensor of a multi-sensor camera (S502); capturing image frames of a second video stream with a second image sensor of the multi-sensor camera (S504); encoding (S506) image data for the image frames of the first video stream by a first encoder of the multi-sensor camera, wherein each of the image frames of the first video stream is encoded into first encoded data including a plurality of rows, each of the rows having a height of a single coding unit and a width equal to a width of the image frame, and encoded as one or more slices; encoding (S508) image data for the image frames of the second video stream by a second encoder of the multi-sensor camera, where each of the image frames of the second video stream is encoded into second encoded data including a plurality of rows, each of the rows having a height of a single coding unit and a width equal to a width of the image frame, and encoded as one or more slices; combining (S510) by a data combining unit the first encoded data of the image frames of the first video stream and the second encoded data of the image frames of the second video stream into combined encoded data by interleaving rows of the first encoded data of the image frames of the first video stream and rows of the second encoded data of the image frames of the second video stream, wherein the combined encoded data represents the combined image frames of the combined video stream; checking whether two or more immediately following slices in bitstream order in the combined encoded data contain only skip encoded coding units, and if so, replacing the two or more immediately following slices in bitstream order that contain only the skip encoded coding units with a new slice containing a skip encoded coding unit having a length of the two or more immediately following slices in bitstream order that contain only the skip encoded coding units (S512); A method comprising:

10. 10. The method of claim 9, wherein the combining is performed such that each row in the combined encoded data includes a single row of the first encoded data and a single row of the second encoded data.

11. 11. The method of claim 9 or 10, wherein encoding the image data of the first video stream and the image data of the second video stream is performed in parallel.

12. 12. The method of claim 9, wherein capturing the first and second video streams is performed synchronously.

13. A non-transitory computer readable storage medium having stored thereon instructions which, when executed on a device having processing capabilities, perform the method of any one of claims 9 to 12.

Images