encoding images from separate cameras

Abstract

This relates to encoding images from separate cameras. A computer-implemented method (400) for encoding images captured by separate video cameras is provided. The method includes: receiving (S410) a corresponding image from each of two separate video cameras; determining (S440) that a region in one of the images has an image texture similar to regions in other images according to at least one condition; and encoding the image (S460) as part of one or more encoded video streams, wherein encoding the region of the first image in the images includes using the region of another second image in the images as a reference. A corresponding video camera system, video camera, and computer program and computer program product are also provided.

Description

Technical Field

[0001] This disclosure generally relates to video camera systems and the field of encoding video streams generated from images captured by such cameras. Specifically, this disclosure relates to solutions for encoding images captured by separate video cameras as part of generating one or more encoded video streams. Background Technology

[0002] To cover the same scene from different angles and / or capture different parts of the same scene, contemporary video camera systems typically include multiple video cameras positioned at different locations or at least configured to have different fields of view (FOV). For many cameras, and especially many high-resolution cameras, encoding and transmitting video streams from the cameras can be challenging because as resolution increases, greater pressure is placed on the computing hardware and often higher bit rates are required. Summary of the Invention

[0003] To at least partially overcome some of the aforementioned challenges of contemporary technology, this disclosure provides, as defined in and by the appended independent claims, a method for encoding images captured by separate video cameras, a video camera system, a video camera, and corresponding computer programs and computer program products. Embodiments of these entities are defined in and by the appended dependent claims.

[0004] According to a first aspect, a computer-implemented method is provided for encoding images captured by separate video cameras (e.g., captured by two or more separate video cameras). The method includes receiving a corresponding image from each of the two separate video cameras. The method includes determining, based on at least one condition, that a region in one of the images has an image texture similar to regions in other images. The method further includes encoding the image as part of one or more encoded video streams, wherein encoding the region of a first image of the image includes using regions of other second images in the image as a reference (i.e., encoding the region of one image with reference to the regions of other images).

[0005] The proposed solution improves upon contemporary technology by considering a system-wide perspective, where texture similarity between images captured by separate cameras can be taken into account and utilized to improve coding efficiency. This contrasts with conventional solutions, where images from each camera are typically encoded separately. It should be noted that for the proposed solution to work—that is, to provide at least some improvement in coding efficiency (e.g., a reduction in the number of bits required to encode two images and / or a reduction in the bit rate required to transmit two images as part of one or more encoded video streams)—a perfect match between regions and their respective image textures is not required. Even a rather coarse match, for example, with respect to image texture, can still be beneficial because regions with at least some similar image textures may share at least some redundant information between images.

[0006] In one or more embodiments of the method, encoding the region of the first image using the region of the second image as a reference can be performed in response to determining that the two regions have similar image textures (e.g., by confirming this before attempting to encode the region of the first image). In other embodiments, confirming the texture similarity of these regions may be performed only once or, for example, only occasionally, and the region of the first image may be encoded with reference to the second image under the assumption that these regions are still texture similar.

[0007] In one or more embodiments of the method, determining that two regions have similar image textures may include performing image segmentation on each of the images, and at least one condition may include that the image segmentation indicates that the two regions belong to the same category. Image segmentation can be performed using conventional methods such as machine learning-based methods, and can determine that two regions have texture similarity even if, for example, objects of the same category (e.g., trees, roads, asphalt, grass, soil, pebbles, etc.) may have redundant information due to their texture similarity.

[0008] In one or more embodiments of the method, the method may further include acquiring the spatial relationship and field-of-view (FOV) relationship between the two cameras. This at least one condition may include the spatial relationship and FOV relationship indicating that the two regions correspond to the same part of the scene captured by the two cameras. Based on how the cameras are aligned and what the cameras may capture, assuming that the two regions are texturally similar can, for example, eliminate or supplement the need for image segmentation, etc.

[0009] In one or more embodiments of the method, the method may further include: obtaining (e.g., determining) a perspective transformation that, when applied to a second image, improves at least one of the spatial alignment and orientation of the region in the second image and the region in the first image. The reference may be to the second image transformed by the perspective transformation. Improving the spatial alignment and orientation between the regions can, for example, help reduce the amount of work required to perform a motion vector search in the second image to find macroblocks, for example, macroblocks similar to the region in the first image. It should also be noted that in some embodiments, applying such a perspective transformation may not be necessary, at least for some regions, and particularly for regions containing image textures with little or no spatial alignment / orientation, such as those depicting, for example, ordinary sky, asphalt, leaves, and other more spatially homogeneous textures.

[0010] In one or more embodiments of the method, the method may further include providing data indicating perspective transformation via an interface accessible to the receiver of one or more encoded video streams, such as a client, a receiver, and a decoder. Therefore, the second image does not need to be encoded in its transformed state, but the receiving decoder can apply the transformation to the second image (after it has already been decoded), and then use the transformed version to decode the first image using motion vectors, for example, those intended to point to a portion of the transformed version of the second image.

[0011] In one or more embodiments of the method, the method may further include selecting a first image as an image with a higher pixel density in its region and selecting a second image as an image with a lower pixel density in its region. This may have the advantage of sending the lower resolution image over the network and decoding it at another camera, which may be “cheaper.” Additionally, the quality suitable for, for example, scalable video encoding can be improved, since otherwise it might be necessary to first downscale the higher resolution image to a lower resolution image so that the then-newer lower resolution image can be used as a reference for encoding the then-newer higher resolution image.

[0012] In one or more embodiments of the method, the method may further include encoding two images (i.e., both the first image and the second image) as part of the same multi-layer video stream. This may be particularly useful, for example, in terms of coding efficiency, if a newer standard such as Low Complexity Enhanced Video Coding (LCEVC) is used, thereby allowing the hardware-coded base layer information to be supplemented with efficient, low-complexity enhancement layer information.

[0013] In one or more embodiments of the method, the method may further include encoding a second image in a base layer of the video stream and encoding a first image in an enhancement layer of the video stream. For example, the image in the base layer may be an image with a lower pixel density in its region, while the image in the enhancement layer may be an image with a higher pixel density in its region.

[0014] In one or more embodiments of this method, the fields of view (FOV) of the two cameras may not overlap. This can be advantageous because even if the FOVs do not overlap, one camera can still capture objects of the same category, for example, with image textures similar to those captured by the other camera. Therefore, the envisioned solution is still able to identify regions with similar textures in the two images and leverage this to improve coding efficiency and, for example, the bit rate requirements for transmitting the two images. For example, two cameras could be deployed to record a football match. Even if the camera FOVs do not overlap, the two cameras may still capture at least some areas containing, for example, grass (or any material forming a sports field thereon), and thus coding efficiency and bit rate requirements can be improved by referencing one area to encode the others, rather than encoding each image separately without a system viewpoint.

[0015] According to a second aspect, a computer-implemented method for encoding images captured by a video camera is provided. The method is performed in / by the video camera (e.g., by the video camera's processing circuitry) and includes obtaining corresponding images from the video camera and other (video) cameras. The method includes determining, based on at least one condition, that a region in one of the images has an image texture similar to regions in other images. The method further includes encoding the images as part of one or more encoded video streams, wherein encoding the region of a first image in the images includes using the region of another second image in the images as a reference. In some examples, encoding the region of the first image may be performed in response to the determination, wherein the determination may be performed repeatedly (for each new image captured by the video camera) or at least once (e.g., after the installation and / or setup of the video camera).

[0016] According to the third aspect, a video camera system is provided. The video camera system includes two separate video cameras and processing circuitry configured to perform the methods of the first aspect (or any example embodiments thereof as described herein).

[0017] According to the fourth aspect, a video camera is provided. The video camera includes at least one image sensor for capturing images. The video camera includes an interface for receiving other images from other (video) cameras. The video camera further includes processing circuitry configured to perform the methods of the second aspect (or any example embodiments thereof as described herein).

[0018] According to the fifth aspect, a computer program including instructions is provided that, when executed by the processing circuitry of a video camera system comprising two separate cameras (e.g., the system of the third aspect), causes the video camera system to perform the operation of the method of the first aspect (or any example embodiment thereof as described herein).

[0019] According to the sixth aspect, a computer program including instructions is provided, which, when executed by the processing circuitry of a video camera (e.g., the video camera system of the fourth aspect), causes the video camera to perform the operation of the method of the second aspect.

[0020] According to the seventh aspect, a computer program product is provided, which includes a computer-readable storage medium storing a computer program of at least one of the fifth and sixth aspects.

[0021] As used herein, a computer-readable storage medium can be, for example, non-transitory and can be provided as, for example, a hard disk drive (HDD), a solid-state drive (SDD), a USB flash drive, an SD card, a CD / DVD, and / or any other storage medium capable of non-transitory storage of data. In other embodiments, a computer-readable storage medium can be transient and can correspond, for example, to signals (electrical, optical, or mechanical, etc.) present on, for example, a communication link, a wire, or a similar signal transmission device, in which case the computer-readable storage medium is, of course, more of a data carrier than a data storage entity.

[0022] Other objects and advantages of this disclosure will become apparent from the following detailed description, drawings, and claims. Within the scope of this disclosure and the appended claims, all features and advantages described with reference to, for example, the method of the first aspect are contemplated, applicable to, and can be used in conjunction with the method of the second aspect, the video camera system of the third aspect, the video camera of the fourth aspect, the computer program of each of the fifth and sixth aspects, and the computer program product of the seventh aspect, and vice versa. Attached Figure Description

[0023] The following description of exemplary embodiments will be made with reference to the accompanying drawings, in which:

[0024] Figure 1An example embodiment of a video camera system according to the present disclosure is illustrated in perspective view, wherein the system is configured to capture one or more scenes from different angles;

[0025] Figure 2A , Figure 2B and Figure 2C Schematic diagram of by Figure 1 The image shown in the diagram is an example of an image captured by the camera of the system.

[0026] Figure 3A and Figure 3B The illustration schematically depicts an example of a perspective transformation according to the present disclosure, which can be applied to an image to improve the spatial alignment and / or orientation between regions of the image and regions of other images identified as having similar image textures;

[0027] Figure 4 The flowcharts schematically illustrate one or more example embodiments of a method according to the present disclosure for encoding images captured by separate video cameras;

[0028] Figure 5 A schematic diagram illustrating the decoding process. Figure 4 A flowchart of one or more example embodiments of the method according to this disclosure, showing images of one or more encoded video streams generated by the method.

[0029] Figure 6A and Figure 6B The illustrations schematically depict various example embodiments of video camera systems and video cameras according to the present disclosure;

[0030] Figure 7A and Figure 7B Components and functional blocks of various example embodiments of a device according to this disclosure, such as those used in a video camera system, video camera, or decoder as envisioned herein, are schematically illustrated; and

[0031] Figure 8 The illustrations schematically depict example embodiments of computer programs, computer program products, and computer-readable storage media according to the present disclosure.

[0032] In the accompanying drawings and figures thereon, similar reference numerals will be used for similar elements unless otherwise stated. Unless the contrary is explicitly indicated, the drawings only show elements necessary to illustrate the exemplary embodiments, while other elements may be omitted or merely implied for clarity. As illustrated, for illustrative purposes, the (absolute or relative) dimensions of elements and regions may be exaggerated or understated relative to their true values, and are therefore provided to illustrate the general structure of the exemplary embodiments.

[0033] List of reference numerals

[0034] 100: Example setups / scenes including separate cameras

[0035] 110a-c: Video camera

[0036] 120a-c: Camera FOV

[0037] 210a-c: Images captured by separate cameras

[0038] 220a, 220b: Regions with similar image textures

[0039] 230a-c, 231c: Regions with similar image textures

[0040] 240a-c, 241a: Regions with similar image textures

[0041] 250a-c: Regions with similar image textures

[0042] 300, 301: Perspective Transformation

[0043] 400: Method Flowchart

[0044] S410-S460: Operation of Method 400

[0045] 500: Method Flowchart

[0046] S510-530: Operation of Method 500

[0047] 600, 601: Video camera system

[0048] 610a, 610b: Encoding modules

[0049] 611(b): Decoding module

[0050] 620a, 620b: Spatial data and FOV data

[0051] 622: Data exchange between cameras

[0052] 630(b): Perspective Transformation Module

[0053] 640(b): Texture Similarity Determination Module

[0054] 642, 644: Encoded video stream

[0055] 650: User Client

[0056] 660: Additional Information / Data

[0057] 670 interface module

[0058] 672, 674: Data obtained from the interface module

[0059] 680: Centralized Unit

[0060] 700: Device

[0061] 710, 710a-d: Processing circuit, functional block entity

[0062] 712, 714, 716: Memory, Additional Entities, Communication Interfaces

[0063] 810a, 810b; 820a, 820b: Computer program products; computer programs

[0064] 830: Computer-readable storage medium Detailed Implementation

[0065] Now refer to Figure 1 A more detailed description of examples of the settings to which the envisioned solutions apply.

[0066] Figure 1 The arrangement 100 of multiple video cameras 110a, 110b, and 110c, configured to monitor portions of a scene from different angles, is illustrated schematically in perspective view. In the following text, unless otherwise stated, the terms "video camera" and "camera" will be used interchangeably.

[0067] In this particular example, the scene is an urban landscape, and the cameras are mounted on, for example, buildings and lampposts in the city, and configured such that they each have a corresponding FOV 120a, 120b, and 120c. Cameras 110a-c form part of the video camera system envisioned herein. In this particular example, the FOVs 120a and 120b of cameras 110a and 110b overlap, while the FOV 120c of camera 110c does not overlap with either FOV 120a or 120b. It should be noted that, as will be described in more detail later in this document, the exact setup and number of cameras ( The operation of the solution envisioned in this paper is not critical, and the scene captured by the camera can of course be any scene, in which monitoring the scene from different angles is desirable. In fact, one or more cameras do not need to be fixed and can be, for example, dashcams in cars, wearable cameras, autonomous driving cameras, or other cameras mounted on non-fixed objects (e.g., people, animals, and vehicles).

[0068] Each camera 110a-c is configured to capture an image, and therefore each captured image will correspond to a portion of the scene within the corresponding FOV 120a-c of the camera. Since FOVs 120a and 120b partially overlap, and since none of the FOVs 120a-c are equal, although a portion of the scene captured by, for example, camera 110a will at least partially overlap with a portion of the scene captured by camera 110b, cameras 110a-c will capture different portions of the scene, and vice versa. However, due to its non-overlapping FOV 120c, camera 110c will capture different portions of the scene that do not overlap with those captured by cameras 110a and 110c.

[0069] Now will also refer to Figure 2A , Figure 2B and Figure 2C Here are some examples of images captured by camera 110a-c.

[0070] Figure 2A The illustration schematically depicts image 210a captured by camera 110a. Due to the complexity of the scene, image 210a will include regions with different image textures. For example, for illustrative purposes only, in Figure 2A Five such areas are highlighted in the diagram. The first area 220a corresponds to a portion of a brick wall (of a building) and has an image texture with a brick wall pattern. The second area 230a corresponds to a portion of a tree and has an image texture with a leaf pattern. The third area 240a corresponds to a portion of a lawn and has an image texture with a grass pattern. For illustrative purposes, a fourth area 241a is also included, which corresponds to a portion of the same lawn and also has an image texture with a grass pattern. Finally, the fifth example area 250a corresponds to a portion of a sidewalk / pavement and, in this particular example, has an image texture with a pattern such as slate or cobblestones. Of course, image 210a could include different and / or similar image textures and additional such areas of different and / or similar shapes and sizes, but to avoid clutter, only a few possible areas are highlighted.

[0071] Figure 2B The schematic illustration shows image 210b captured by camera 110b. Image 210b captures a portion of the scene that is almost identical to that of image 210a, but because the field of view (FOV) 120a and 120b are not exactly the same, image 210b was captured from a different angle / position than image 210a, and therefore there are some visual differences. However, in image 210b, features with similar characteristics can be identified. Figure 2AThe highlighted areas are multiple regions with similar image textures. For example, the first region 220b of image 210b has an image texture similar to region 220a of image 210a because both regions correspond to a portion of the same brick wall. Similarly, the second region 230b has an image texture similar to region 230a of image 210a because both regions correspond to a portion of a tree; the third region 240b has an image texture similar to region 240a because both regions correspond to a portion of a lawn; and the fourth region 250b has an image texture similar to region 250a because both regions correspond to a portion of a road surface / sidewalk / pedestrian walkway, etc.

[0072] Figure 2C The illustration schematically depicts image 210c captured by camera 110c. Due to the non-overlapping FOV 120c, image 210c captures a portion of the scene that differs from portions of the scene in either image 210a or 210b. However, there are still regions in image 210c that have image textures similar to corresponding regions in one or more of images 210a and 210b. For example, region 230c (and / or, for example, region 231c) of image 210c has an image texture similar to that of regions 230a and 230b, as they all correspond to trees; region 240c of image 210c has an image texture similar to that of regions 240a, 241a, and 240b, as they all correspond to lawns; and region 250c of image 210c has an image texture similar to that of regions 250a and 250b, as they all correspond to pavements / sidewalks / pedestrian paths, etc. Therefore, even if the FOV of camera 110c does not overlap with either FOV 120a or 120b, there may still be regions in image 210c (captured by the third camera 110c) that have image textures similar to those found in one or more regions of other images, especially if similar (but not necessarily identical) objects are found in all images.

[0073] Generally, the solution envisioned in this paper is based on the understanding that when regions with the same or similar image texture exist in multiple images, encoding an image can be more efficient by using one of the other images as a reference. Specifically, this is true not only when the image textures are completely identical, with the same orientation and spatial alignment, but also, at least to some extent, as long as there is at least some visual similarity between two regions in different images. In other words, even if two regions do not have completely identical image textures, and / or even if the image textures of these regions are not perfectly aligned and oriented, encoding other regions by referencing one region is still advantageous in terms of, for example, bit rate requirements, because, for example, encoding other regions by referencing one region, plus the final difference / residual term, may still be more efficient than encoding these regions individually without any such reference.

[0074] As envisioned in this paper, whether a region has an image texture similar to other regions will be determined by checking the satisfaction of at least one condition. As used in this paper, "image texture" can correspond to a (small-scale) structure perceived on an image based on, for example, the spatial arrangement of color or intensity.

[0075] For example, image textures can be segmented based on features such as spatial frequency and average gray level, and the image textures of separated regions can be compared using structured and / or statistical methods. Structured methods can, for example, involve viewing the image texture as a set of primitive texture elements in a regular or repeating pattern, and the representation of the spatial relationships of the texture elements can be collected using, for example, Voronoi tessellation. Statistical methods can, for example, involve viewing the image texture as a quantitative measurement of the arrangement of intensity in regions, and can use or at least draw upon methods such as edge detection, co-occurrence matrix, texture energy measurement, and autocorrelation. Textures can be segmented using, for example, region-based and / or boundary-based methods.

[0076] Other examples may alternatively or additionally include: if two regions are determined to include (or correspond to) one or more objects of the same (object) category, then these regions are considered to have the same or similar image texture. Therefore, alternatively or in addition to texture segmentation as described above, determining the category of an object / part of an image can be achieved by performing one or more other forms of image segmentation (i.e., segmenting the image into multiple image fragments (e.g., image regions or image objects)). For example, the pixels of an image can be labeled, and pixels sharing the same label can be considered to share certain features, i.e., corresponding to the same object category, and therefore (at least statistically likely) share the same (or similar) image texture. In some examples, semantic image segmentation can be used, where all objects of the same type are assigned the same label (or category ID). Object types / categories can include, for example, sky, grass, soil, gravel, stone, rock, pebbles, tile / slate, brick / brick wall, tarpaulin / asphalt, and of course, any of a variety of other possible texture types. In some examples, object types can be further subdivided based on, for example, lighting and color. Using "grass" as an example of an object category (or type), this category can be further subdivided, for example, into well-lit grass, moderately lit grass, and poorly lit / dark grass, etc. Other examples include providing different subclasses / labels for, for example, green grass, brown grass, yellow grass, or any other suitable division that can be conceived. In other examples, instance segmentation can be used, where different objects of the same type / category can be identified. In still other examples, combinations such as panorama segmentation can be used, which are capable of identifying not only object type / category but also, for example, different instances of the same object type / category. It should be noted that for the solution envisioned in this paper to work, objects identified as belonging to the same category do not need to be the same instance of that category; that is, it can be assumed that even if two objects correspond to, for example, two different trees, the image textures are similar enough simply because they are both trees.

[0077] Image segmentation can be achieved, for example, using computer vision-based methods and / or by employing artificial intelligence (AI) techniques. Image segmentation can be achieved, for example, by using or drawing upon methods such as thresholding, data clustering (e.g., K-means algorithm), motion and interaction segmentation, compression-based methods, histogram-based methods, edge detection, outlier detection, dual clustering methods, region growing methods, partial differential equation (PDE)-based methods (e.g., parametric methods, level set methods, and fast marching methods, etc.), variational methods, graph segmentation methods (e.g., Markov random fields, maximum a posteriori probability methods, simulated annealing, iterative conditional mode / gradient descent, etc.), watershed transform, pattern-based segmentation, multi-scale segmentation, and / or by utilizing trainable segmentation ideas from artificial neural networks (ANNs). Examples of the latter include, for example, self-organizing (feature) maps (SOM or SOFM) / Kohonen networks, U-net (or other convolutional neural network-based methods), ResU-Net / R2U-Net, SegNet, ENet, BoxENet, or any other suitable architecture, such as one or more transformer architectures or encoder-decoder architectures. As envisioned in this paper, some other examples of how to determine whether two regions of two images captured by different cameras are sufficiently similar in texture may include, for example, simply assuming they are sufficiently similar and examining the bitrate of the encoded video stream as a result to determine whether such an assumption is correct, since texture similarity can be assumed to result in a reduced bitrate compared to the absence of such texture similarity. Other examples include, for example, using feature matching via one or more algorithms based on Scale Invariant Feature Transform (SIFT).

[0078] Taking images 210a and 210b as examples, image segmentation can be performed on the two images, thereby classifying, for example, regions 220a and 220b as, for example, "brick wall"; regions 230a and 230b as, for example, "tree" or "shrub"; regions 240a / 241a and 240b as, for example, "grass" / "lawn"; and regions 250a and 250b as, for example, "road surface," "tile," "stone slab," and "pebbles." In some examples, such classification can lead to regions being further classified as corresponding to the same or similar lighting conditions (e.g., dark, medium brightness, bright, etc.) and / or colors (e.g., brown, red, green, yellow, etc.).

[0079] Therefore, at least one condition for determining that two regions of two images 210a and 210b have the same or similar image texture may include requiring that the two regions are associated with / classified as the same object category and / or object subcategory. If each object category and / or subcategory is assigned a specific number, the condition may include that the two regions have the same number. If each object category and / or subcategory is assigned a probability, the condition may include that the probabilities of the two regions are sufficiently high for the same category and / or subcategory (e.g., above a probability threshold) and, for example, equal or sufficiently equal (e.g., the absolute difference between them is below a difference threshold), and so on. Other examples of how to determine whether two regions are classified as belonging to the same or similar object category are of course possible. For example, determining that, for example, regions 240a and 240b have similar image texture may require that both regions be classified as belonging to the same object category and / or subcategory, for example, both regions are classified as "grass," and optionally also as grass with the same lighting conditions and / or color, etc. It should be noted that explicit labels such as “grass,” “rock,” “brick wall,” etc. are of course not required, and any way of distinguishing different categories (e.g., different bits, integers, floating-point numbers, or whatever is appropriate) is sufficient for the purposes of this disclosure.

[0080] Furthermore, it may or may not be considered whether the image textures of the two regions are also spatially aligned and oriented. As envisioned in this paper, as long as the image textures have the same or similar type, the solution can still work even if the spatial alignment and / or orientation of the image textures of the two regions are not the same, i.e., to reduce, for example, computational complexity, the total amount of data required to encode two images, the bit rate required to transmit the encoded images, etc. This is based on the assumption that as long as the image textures are the same or similar, there may be at least some visual information / data shared between the regions, thus allowing the encoding of other regions to be referenced to one region, even if there is no perfect spatial alignment and / or the same or similar orientation between the image textures of the two regions. In other words, attempting to reference one region to encode other regions can reduce the size of the difference / residual term between regions, thereby providing more efficient encoding.

[0081] In addition, for example Figures 2A to 2CThe exact dimensions of the regions illustrated are, of course, merely examples. For instance, regions 220a and / or 220b could be expanded to cover a larger portion of the building's brick wall, or could be shrunk to cover a smaller portion, for example. Regions can be defined such that they cover, for example, the largest possible area with the same image texture, or, for example, the largest possible area with the same image texture but also with the same spatial arrangement and / or orientation, and so on. The size / shape of a region can be determined, for example, by maximizing regions containing the same type of image texture and depicting specific objects, etc. In other examples, regions can be maximized under the constraint that they still present approximately the same size, shape, and / or position in both images. The size / shape of a region can, of course, also be determined in other ways (e.g., by predetermined size and / or shape values ​​and historical analysis based on whether a particular combination of region size and / or shape leads to improved coding efficiency, etc.). For example, a conceived solution could include comparing rectangles to rectangles, but it is envisioned that any available assistance from matching regions could contribute to more efficient image encoding, regardless of the exact form / shape of overlapping regions with sufficiently similar image textures. For example, an optimization problem can be solved where the goal is to increase the size of two regions (i.e., each region in each corresponding image), with the cost function constrained to penalize candidate solutions where a larger proportion of one region now has texture-similar counterparts in other regions, and vice versa. Other solutions are also envisioned, of course, including, for example, increasing the regions until (no longer) a significant reduction in the bitrate of the resulting encoded video stream is observed, or, for example, until the resulting bitrate begins to increase again, and so on. Other examples include directly considering the resulting difference / residual terms arising from attempting to reference one region to encode other regions, where these terms are expected to increase as texture similarity decreases across all regions, and vice versa.

[0082] As another example envisioned in this paper, whether two regions in different images captured by different cameras are considered to have the same or similar image texture can be based solely on, or additionally on, the spatial relationship and FOV relationship between the cameras. Such a relationship can include, for example, knowledge about differences in the physical location, physical orientation / heading, zoom level, image sensor properties, and / or lens arrangement properties (such as, for example, focal length) between the cameras. Parameters can be provided individually for each camera, and the differences between cameras can be calculated based on these parameters, or, for example, the differences can be provided directly (such that only the relative differences between the cameras are known / considered). For example, data can be provided for each camera indicating its position (e.g., x, y, and z coordinates), its orientation (e.g., pan, tilt, and yaw angles), and its zoom level, or any other parameters from which the FOV (i.e., what the camera sees) can be derived. In other examples, data can be provided that indicates (only) differences in position, orientation, and / or zoom level (i.e., differences in FOV) rather than a single value.

[0083] Based on such data, it can be inferred that a specific region in an image captured by one camera corresponds to a portion of the same scene (e.g., the same object) as a specific region in another image captured by other cameras. It can then be assumed that, since these regions (likely) capture the same part of the scene, they may also have the same or similar image texture. This can be assumed with or without further analysis of the images (e.g., texture and / or image segmentation). One advantage, of course, is that without performing any further such analysis, the amount of computational power and resources required to implement the solution disclosed herein can be further reduced, thereby freeing up such resources for other purposes or allowing, for example, the use of simpler and / or more cost-effective hardware to construct the cameras and / or devices required to process images from the cameras.

[0084] As an example, based on knowledge of the spatial and FOV relationships between cameras 110a and 110b, it can be assumed that region 220a found in image 210a may correspond to a part of the same scene as region 220b found in image 210b, and it can be assumed that the image textures of the two regions 220a and 220b are the same or similar. Furthermore, it is decided to refer to other images and regions in regions 220a and 220b to encode one of regions 220a and 220b to provide more efficient encoding. The same applies to other pairs of regions such as 230a, 230b, 240a / 241a and 240b, as well as 250a and 250b.

[0085] Figure 2CTo illustrate the proposed solution in action, the FOVs of the two cameras do not necessarily need to overlap. While it might be more difficult to determine if two regions have the same or similar image textures because they depict the same part of the scene, it is still possible to infer that two regions have the same or similar image textures by depicting, for example, objects of the same category / type, or by using some other metric (e.g., texture-based segmentation) to confirm that their image textures are the same or similar. In other words, using images 210a and 210c as examples, since all regions 230a, 230c, and 231c correspond to the part of the scene that includes trees, even though regions in different images do not necessarily correspond to, for example, the same single tree (i.e., the same instance of the same object category), one or more of these regions in one image can be encoded by referring to one or more such regions in other images. This is because, of course, although, for example, two trees are different individuals, their visual attributes (i.e., image textures) may still be the same or similar, and this fact can be used to reference other trees to encode a tree. Of course, this also applies to other areas that depict other types of objects, such as areas 240a / 241a and 240c that depict grass / lawn, and areas 250a and 250c that depict pavement, etc.

[0086] As envisioned in this paper, when encoding a region of an image with reference to other images, such a procedure may include performing a so-called motion vector search, wherein the region of the image to be encoded is divided into, for example, one or more macroblocks (e.g., 4×4, 8×8, 16×16 pixels, etc.), and wherein, for each such macroblock of the region to be encoded, a search is performed in other images to find whether a visually similar macroblock exists in those other images. If yes, a motion vector indicating where the corresponding macroblock will be found in the other images can be provided. However, such motion vector searches are often computationally expensive, especially when the two images are not well aligned (due to, for example, movement of the cameras capturing the two images or due to, for example, different FOVs of the separate cameras capturing the images), because the search area in the other images needs to be expanded before a matching corresponding macroblock can be successfully found, or, for example, it is inferred that no such matching macroblock exists and, alternatively, the particular region needs to be encoded without reference to other images (because the residual as a result may not be small enough). As envisioned in this paper, motion vector search (or "motion estimation") can be performed, for example, using various block matching algorithms, phase correlation, pixel recursion algorithms, and optical flow. Block matching algorithms can, for example, use mean difference or mean absolute difference (MAD) and mean squared error (MSE) as evaluation metrics for the existence of matching macroblocks, and if the spatial alignment and / or orientation between regions of two images is not optimal, it may be necessary to expand the search parameters indicating the search area used. To at least partially alleviate this problem, in some examples of this solution, it is envisioned that one or more perspective transformations are obtained (e.g., by computation) that, when applied to other images, improve at least one of the spatial alignment and orientation of one or more regions of the other image with one or more regions of the other image to be referenced for encoding. References will also be made to... Figure 3A and Figure 3B Here are some further examples illustrating such a transformation.

[0087] Figure 3A This schematic diagram illustrates the area used in (second) image 210b (image " "the area in" Perspective transformation between the region "j" in image "a" and the region "j" in image "a". Example set 300. To avoid cluttering the image, only some possible perspective transformations are illustrated. For example, perspective transformations... It is considered to represent the perspective transformation from region 220b to region 220a (i.e., aligning the region of brick wall 220b with region 220a). It is a transformation used to align region 230b with region 230a, and and These are transformations used to align region 240b with regions 240a and 241a, respectively. Such transformations for various regions can, of course, be defined as transformations used to align the image… (This is 210b) area "aligned" to the image " Single transformation of regions such as (here is 210a) It is part of. Of course, an inverse transform can also be defined. or Alternatively, the region of image 210a is aligned with the region of image 210b. It should be noted that in this document, which image is labeled "first image" or "second image" may vary depending on the implementation and / or details of each image / camera.

[0088] Conceptually, perspective transformations, when applied to an image that is to be used as a reference for encoding other images, make regions of that reference image better aligned with regions of other images (i.e., spatially and / or about orientation). In terms of motion vector search, it becomes easier to find matching macroblocks in the reference image for macroblocks of regions in the image to be encoded, because it can be expected, for example, that the search area required to find such matching macroblocks does not need to be expanded (if any) as much before a matching macroblock can be found. In other words, by first applying one or more perspective transformations to the reference image, it is computationally more efficient to find motion vectors for encoding one or more regions of other images with reference to that reference image than if such motion vectors were to be found without first applying one or more perspective transformations. Similarly, even if a perspective transformation does not necessarily change the spatial alignment (i.e., position) of one or more regions in the reference image, but only their orientation, it is conceivable that, for example, motion vector searching in the reference image (after one or more transformations have been applied) can still be made easier, because the chance of finding a matching macroblock is still increased, since finding a macroblock with a physical orientation similar to the target macroblock (i.e., the macroblock of the image to be encoded) is easier than trying to find a macroblock with a physical orientation dissimilar to the target macroblock. From another perspective, the time required to perform a motion vector search is not the only aspect. Even if the time required to find a matching macroblock remains, for example, constant, improving spatial alignment and / or orientation by applying one or more perspective transformations can lead to better matching macroblocks, resulting in smaller difference / residual terms that need to be encoded, and thus more efficient encoding not only in terms of speed but also in terms of the required size (e.g., bits) or the required bit rate. Therefore, identifying and applying one or more perspective transformations can be beneficial in multiple ways.

[0089] As envisioned in this article, one or more perspective transformations (or such as...) The transformation (of motion vectors) can be found based on knowledge of the spatial relationships and FOV relationships between the cameras used to capture the image. For example, it can be inferred from such relationships that, in order to better align a region of the reference image with a region of the image to be encoded, thereby improving the efficiency of motion vector search, the region of the reference image may need to be translated, scaled, and / or rotated in a specific manner based on known differences between the cameras, such as their positions, orientations, and zoom levels. Figure 3A In the diagram, examples of such transformations are illustrated by arrows, indicating how, for example, the corners of region (2Xb) of image 210b are transformed to align with the corners of region (2Xa) of image 210a. It is also illustrated how, for example, region 240b of image 210b can be used to encode one or both of regions 240a and 240b of image 210a, since both regions 240a and 240b correspond to grass-like image textures.

[0090] Figure 3B The schematic illustration depicts, for example, the region 2Xc of image 210c to the region 2Xa of the first image 210a, i.e., the image " The region “i” and the image “ "area" One or more "perspective" transformations between "". Another example set is 301. Specifically, Figure 3B In a reference image (e.g., image 210c), there may be multiple regions with image textures similar to or identical to those in a region of the image to be encoded (e.g., image 210a), and therefore one of regions 230c and 231c can be used to encode the corresponding region 230a of image 210a with reference to image 230c. If multiple regions in an image can be used as references for encoding regions in other images, then which of the multiple regions to use can be determined based on some suitable criteria, including, for example, which region is most spatially aligned with the region to be encoded and which region has an image texture considered most similar to the region to be encoded. For example, such a decision can be made using any of the similarity measures described herein, or, for example, by examining the bitrate and / or the difference / residual terms of the encoded video stream as a result when attempting to encode other regions with reference to one region.

[0091] Figure 3BAlso used in illustration, in the envisioned solution, "perspective transformation" does not necessarily require actual overlap of the FOV between the two cameras; that is, the "perspective" transformation (in such a case) does not need to be an actual perspective transformation to transform the portion of the scene visible in the two images. Instead, it can be a scaling, translation, and / or rotation of a portion of the reference image, such that, if, for example, it is located above the first image, the translated region of the reference image is ultimately more aligned and / or oriented with one or more regions to be encoded in other images. This makes searching for motion vectors in the (transformed) reference image to find features similar to those of the one or more regions to be encoded more efficient / faster, for example. In other words, such as... The matrix does not need to be a physically correct perspective transformation based on camera parameters. Instead, it can correspond to one or more scaling / translation / rotation operations to better align these areas, even if the regions between images do not belong to the same part of the scene (e.g., the same instance of an object, etc.).

[0092] As an example, as envisioned in this article, how to obtain, for example Figure 3A and Figure 3BAn alternative to the transformation illustrated in the diagram involves generating homography between overlapping camera FOVs (e.g., between FOV 120a and FOV 120b). This can be done, for example, manually as part of the calibration process for cameras 110a and 110b, or automatically by using contemporary feature matching methods (like SIFT) or, for example, AI-based feature matching methods, to identify several (e.g., three to four) points / features present in both images 210a and 210b. As those skilled in the art of image analysis will understand, based on these points and how they differ between the images, matrices describing the perspective transformation between FOV 120a and FOV 120b can be generated. As another alternative, a three-dimensional (3D) model of the local scene can be constructed. For example, intrinsic parameters (including, for example, lens distortion, focus width, zoom level, pixel pitch, etc.) and extrinsic parameters (including, for example, camera tilt, roll, and mounting height, etc.) of the two cameras (such as cameras 110a and 110b) can be obtained, or obtained as part of camera calibration. The 3D model of the scene can be estimated, for example, by importing from an external system such as a Geographic Information System (GIS) or by, for example, by using a depth model implemented with AI. For each pixel in an image captured by, for example, camera 110a (e.g., a pixel defined by x and y coordinates), its corresponding position in the 3D model, such as the position defined by X, Y, and Z coordinates, can then be obtained by calculating its projection direction into the 3D model and looking at which point in the 3D model it intersects with based on the intrinsic and extrinsic parameters of camera 110a. For other cameras 110b, backtracking from the position (X, Y, Z) to the corresponding pixel (X', Y') in the image captured by camera 110b can be performed using the intrinsic and extrinsic parameters of camera 110b, and then perspective transformation can be calculated based on several (e.g., 3 to 4) such matching points between the two images.

[0093] In some examples, it is envisioned that motion vector search may not always be necessary, and knowledge of the spatial and FOV relationships between the two cameras used to capture the two images can be used to directly obtain the motion vectors, thereby reducing the computational effort otherwise consumed by such a search. For example, if it is determined that a region in one image depicts a part of a scene also found in other images, then finding where to find similar image textures in the reference image can be calculated / estimated based on known differences between the cameras, such as position, orientation, zoom level, etc. As already mentioned in this paper, even without a perfect mapping between the region to be decoded and selected regions in the reference image—that is, even if the motion vectors are not perfectly computed—as long as there is at least some similarity in image texture between the region to be encoded and regions in other images (as referenced by, for example, motion vectors), the envisioned solution can be beneficial because such similarity can still be used to encode the region more efficiently. If the spatial relationship and FOV relationship between the two cameras are not used to obtain motion vectors directly without performing a motion vector search, then knowledge of such a relationship can / can still be used, for example, to help the encoder by offsetting the search window and / or zero-cost point based on such a relationship, in order to reduce the time and / or workload required for motion vector search.

[0094] Figure 4 The flowcharts schematically illustrate one or more examples of a method 400 for encoding images captured by separate video cameras (such as encoding any combination of two or more images 210a-c captured by cameras 110a-c). This method is performed by, for example, processing circuitry of a video camera or video camera system. Reference will be made to... Figure 1 , Figures 2A to 2C , Figure 3A and Figure 3B The example provided describes method 400, but method 400 is of course applicable to any situation involving at least two images, each of which has been captured by separate video cameras.

[0095] As part of operation S410, method 400 includes receiving images from two separate cameras. For example, one received image may be image 210a captured by camera 110a, and the other received image may be image 210b captured by camera 110b. As contemplated herein, “receiving images” may include: receiving one or two images directly from the respective cameras as, for example, raw image data; receiving one or two images, not as raw image data, but as one or more images obtained by at least processing such raw image data; receiving one or more images from memory, or by any other means resulting in the reception of at least two images, wherein the images will be encoded as part of one or more encoded video streams.

[0096] As part of operation S440, method 400 further includes: as already referenced Figure 1 and Figures 2A to 2C For example, it can be determined that a region of one image has a similar image texture to regions of other images. For instance, regions 220a and 220b can be determined to have similar image textures (since they both show part of a brick wall, etc.).

[0097] As described herein, whether regions have similar image textures is determined based on at least one condition, i.e., at least one condition must be met to conclude that the regions have similar image textures. For example, method 400 may optionally include operation S420 prior to operation S440, operation S420 including performing image segmentation on the image, and one such condition may include the region being determined to depict an object (or texture) that, for example, belongs to the same category or at least belongs to a category that has been determined to be sufficiently "similar" (e.g., has image textures that are expected to be sufficiently similar), or may include texture segmentation indicating that the image textures are sufficiently similar. For example, the condition may be satisfied if the region to be encoded depicts at least a portion of a tree, if other regions also depict trees, or, for example, if other regions depict objects such as shrubs or hedges that are considered sufficiently similar to trees. As envisioned herein, similarity may imply belonging to the same category under the assumption that all instances of the same category will have sufficiently similar image textures, or, for example, by actually confirming that the image textures are sufficiently similar (as a result of, for example, texture segmentation). In fact, for the envisioned solution, even billboards, posters, or other images showing at least a portion of, for example, a tree or anything with a similar texture would be candidates for use as references, because the texture similarity between, for example, a tree and anything depicting something, or similarity to a tree, could at least somewhat help encode the region more efficiently with a tree. Of course, choosing a region in a reference image that has the highest possible texture similarity to the region to be encoded can be advantageous, as the increase in efficiency in encoding that region can be expected to be maximized. However, it is important that perfect similarity is not required, and even a fairly coarse match is sufficient to improve the encoding efficiency of the region to be encoded.

[0098] As another example of how to determine (e.g., decide) that two regions are sufficiently similar in texture, instead of, for example, operation S420, or in addition to, for example, operation S420, method 400 may optionally include operation S430, which includes, as also described earlier herein, considering the spatial relationships and FOV relationships between cameras. In other words, in addition to performing image segmentation, or instead of performing image segmentation, it is possible to estimate (or, for example, “guess”) in which part of the reference image is a region with a similar image texture to the region to be encoded likely to be found, based on knowledge of the differences between cameras, such as position, orientation, and zoom level. For example, if there is a tree at a location in the image, whether a tree (or some other tree or some other object with a similar texture to a tree) is found in the reference image can be determined based on knowledge of the spatial relationships and FOV relationships between the cameras.

[0099] As part of operation S460, method 400 further includes: encoding an image as part of one or more encoded video streams; and encoding a region of a (e.g., first) image using a region of a second image as a reference, for example, by encoding a region of a region of a second image using a motion vector pointing to a region of another (e.g., second) image.

[0100] As envisioned herein, any suitable multi-layer coding scheme can be used to encode an image, for example, as part of a multi-layer image stream. Examples include, but are not limited to, MPEG-5 Part 2 / Low Complexity Enhanced Video Coding (LCEVC); Scalable Video Coding (SVC) AV1 (AV1-SVC); SVC extensions of H.264 / AVC and Scalable High Efficiency Video Coding (SVHC) extensions of H.265 / HEVC, etc. If this is done, a reference image can be inserted, for example, into the base layer, and other images can be inserted, for example, into the enhancement layers. Of course, when it is said herein that an image references other images to more effectively encode a specific region, there can certainly be other references. For example, if a specific region of an image is encoded with reference to other reference images, there may be references to regions of the reference images, but there may also be references to, for example, one or more other reference images in the same or different layers. For example, if an image including the region to be encoded is placed in an enhancement layer, it may also reference one or more (temporally) subsequent and / or preceding images in the enhancement layer and / or one or more other enhancement / non-base layers and / or one or more subsequent and / or preceding images in the base layer. The reference to the reference image does not necessarily have to be a reference image depicting the same temporal instance of the scene; that is, the reference image can be before or after the image including the region to be encoded, or can be aligned with it. The image including the region to be encoded can be, for example, a prediction frame (P-frame or B-frame). For example, if LCEVC encoding is used, the reference frame can be encoded as part of the base layer using any suitable (preferably hardware-based) codec (e.g., AVC, HEVC, VP9, ​​AV1, EVC, and VVC, etc.), and the image including the region to be encoded can be encoded using difference / residual as part of a (lower complexity) enhancement layer, and, for example, upsampling methods based on standard signal notification can be used to correct artifacts produced by the base layer video codec and add detail and sharpness to the final output video. In other examples, such as if a single-layer coding architecture is used, the image can be encoded as part of a single layer. If only a single layer is used, appropriate flags can be provided to indicate which images (or image frames) originate from which camera / are associated with which camera, and the decoder can be configured to organize and display images from each camera to create separate video feeds from the respective cameras, etc.

[0101] As envisioned in this paper, two images can be encoded as part of the same encoded video stream, such as part of the same multi-layer video stream or the same single-layer video stream. In other examples, images can be encoded as part of separate video streams, including images of the region to be encoded that are then referenced to other video streams and reference images and (reference) regions found therein.

[0102] As previously described herein, in some examples, method 400 may include, as part of optional operation S450, determining / obtaining, such as One or more perspective transformations, when applied to a reference image, improve at least one of the spatial alignment and orientation of the region of the reference frame to be used as a reference and the region to be encoded thereas. If such one or more transformations are used, then referencing the reference image thus means referencing a reference image that has been correspondingly transformed. In other words, in the envisioned method 400 of such an example, a motion vector search (if used) is therefore performed on the transformed reference image, and thus the region of the (first) image is encoded with reference to the transformed reference image. In such an example, the transformed reference image thus becomes the "reference image". However, since the reference image is preferably encoded itself as part of one or more video streams in its untransformed state (i.e., only the transformation is performed to encode the first image, without encoding the reference second image itself), information about such one or more transformations should be known to the decoder responsible for decoding one or more encoded video streams, as will be referred to later, for example... Figure 5 This will be described in more detail. If necessary, method 400 may include, for example, instructions for one or more perspective transformations (e.g., The information / data is provided via an interface accessible to the receiver of one or more decoded video streams, such as a client computer / server, receiver, and decoder. In some embodiments of method 400, information about one or more (perspective) transformations is therefore not provided as part of the one or more encoded video streams themselves, but needs to be accessed separately in order to properly decode the one or more encoded video streams. For example, such data / information may be provided via an application programming interface (API), such as the API “VAPIX” developed by the applicant. As envisioned herein, it is assumed that such one or more perspective transformations may remain static over time, and by avoiding, for example, encoding and / or providing them as part of the video streams themselves, and by alternatively allowing them to be retrieved from the interface on a per-request basis, the amount of data transmitted over the network interconnecting the cameras can be reduced, etc.

[0103] In method 400, in some examples, what constitutes a “first” image (i.e., the image including the region to be encoded) and a “second” image (i.e., the reference image referenced when encoding a region of the first image) can be determined based on, for example, the pixel density of the respective regions. As used herein, “pixel density” can refer to the number of pixels used to describe a particular object or region, or a particular portion of an object or region, as indicated by, for example, pixels per area or dots per inch (DPI). In other words, for overlapping regions, “pixel density” can refer to, for example, the total number of pixels used to depict the overlapping regions. For non-overlapping regions, “pixel density” can refer to, for example, the number of pixels used to depict regions or objects of, for example, the same physical size.

[0104] For example, if a multi-layer coding scheme is used, method 400 may include, for example, selecting a first image as the image with a higher pixel density in its region and selecting a second image as the image with a lower pixel density in its region after comparing (or knowing) the pixel densities of two images.

[0105] Figure 5 The illustrative diagrams are flowcharts of one or more examples of method 500, performed by, for example, a user client / decoder, to decode one or more encoded video streams generated by method 400. Method 500 may be performed, for example, by the processing circuitry of a decoder, by a video processing server, or by any other computer-based entity responsible for decoding one or more video streams.

[0106] Method 500 includes: as part of operation S510, acquiring one or more video streams encoded using method 400.

[0107] Method 500 further includes, as part of operation S530, determining that there are images of one or more video streams that include at least one specific region that has been encoded with reference to one or more other images, and decoding the at least one specific region using the other images as references.

[0108] In some examples of method 500, optional operation S520 may include: determining that at least one specific region has been encoded with a (perspective) transformed version of another image; and acquiring information / data indicating the perspective transformation used, and performing a (perspective) transformation on the other image using that data before using it as a reference for decoding one or more specific regions, i.e., before operation S530 of method 500. Therefore, if operation S520 is performed, the "reference image" mentioned in operation S530 is a reference image with a (perspective) transformation obtained directly by decoding one or more video streams obtained in operation S510. Here, why one or more (perspective) transformations may be needed before one or more specific regions can be correctly decoded has already been described herein and will not be repeated when describing method 500.

[0109] As envisioned herein, several options may be available so that, for example, the decoder knows that an image used as a reference for decoding other images needs to be transformed first. As an example, the transformed reference frame can be encoded as part of the encoded video stream, thereby eliminating the need to determine whether a transformation is needed and / or to obtain information about how to perform such a transformation before the video stream can be decoded. As another, "cheaper" example, the transformation itself can be encoded, for example, as an additional frame with motion vectors using an auxiliary frame disclosed by the same applicant as the applicant of this disclosure in US 10798418 B2. As yet another example of even higher bit-rate efficiency, special / dedicated client software can be provided to perform the necessary transformations and feed the correct data to the decoder.

[0110] Figure 6A Various examples of a video camera system 600, as envisioned herein, including camera 110a and camera 110b, are schematically illustrated. Camera 110a captures image 210a using one or more image sensors (not shown), provided as part of camera 110a. Camera 110a is configured to transmit image 210a as image 210a' to camera 110b. Before sending image 210a' to camera 110b, this is done either by sending image 210a as raw data (or possibly after some preprocessing) or by first encoding image 210a using a subsequently provided encoder module 610a of camera 110a to generate image 210a'.

[0111] Camera 110b is configured to capture its own image 210b, such as a scene (part of) the same or different from that of camera 110a, using one or more image sensors (not shown) provided as part of camera 110b. Camera 110b is further configured to receive image 210a' from camera 110a. If image 210a' is encoded, camera 110b may include a decoding module 611b to decode the received image 210a' (hereinafter referred to as 210a' for the purposes of subsequent operations involving camera 110b).

[0112] Camera 110b is further configured to determine, using determination module 640b, that in at least one of images 210a' and 210b, there exists at least one specific region having an image texture similar to one or more regions of the other images 210a' and 210b, i.e., as described earlier in reference to method 400 herein. In response to such determination, or, for example by assuming that this is always the case, camera 110b is further configured to reference the one or more regions of the other images 210a' and 210b to encode the one or more specific regions of the at least one image 210a' and 210b as part of the encoding of one or more video streams 642b output from camera 110b.

[0113] The determination module 640b may be configured, for example, to perform image segmentation on images 210a', 210b' to perform the determination, and / or to make such a determination based on information about the position, orientation, zoom level (e.g., FOV) of each of cameras 110a and 110b, or at least based on information indicating the relative differences in position, orientation, zoom level (e.g., FOV) between cameras 110a and 110b. For example, camera 110a may provide information 620a indicating its spatial and FOV information, which may be received by camera 110b (which may already have such internal information about itself). Then, in some examples, the internal information and information 620a may (e.g., by module 640b) be used, for example, by assuming that a portion of a scene depicted in a region of an image captured by one camera may correspond to a portion of a scene depicted in a region of an image captured by another camera and therefore the two regions may have texture similarity, to determine the existence of regions in images 210a', 210b' that (may) have similar image textures.

[0114] As another example, if it is deemed appropriate / desirable to first perform one or more perspective transformations on the image to be used as a reference (or at least a region of the reference image to be referenced), then camera 110b can be configured to perform such transformations using, for example, perspective transformation module 630b. To obtain knowledge about which transformation to use, camera 110b can be configured to use data / information 620a along with its own internal information to determine which transformation to apply to the reference image.

[0115] For example, camera 110a may provide an API accessible from camera 110b, through which data / information 620a may be provided to camera 110b upon request from camera 110b, or camera 110a may be configured to push such information to camera 110b based on its own decision or as part of a request / subscription to such data from camera 110b.

[0116] If needed, camera 110b can also be configured to provide additional information 660, such as information on whether the decoded video stream 642b requires one or more perspective transformations (and, for example, data 620a), to a client 650 responsible for decoding one or more encoded video streams 642b. Such additional information 660 can also or additionally indicate, for example, which images in the encoded video stream 642b correspond to which camera 110a, 110b, etc. In other examples, such additional information 660 may be encoded / included alternatively or additionally as part of the video stream 642b, etc. Information 660 can also or additionally include, for example, information corresponding to data 620a received from camera 110a and internal data of camera 110b, or, for example, an indication of where the client 650 can access information such as an API address.

[0117] In some examples, a separate interface module 670, such as an API module, can be provided, through which data, for example, 620a can be made available to camera 110b as part of data 672. Camera 110a can be configured to provide information 620a to interface module 670, for example, on its own decision or upon request. Camera 110b can also be configured to provide its internal data, such as data 620b sent to interface module 670, for example, on its own decision or upon request. Interface module 670 can be configured to provide data 672, etc., indicating, for example, the perspective transformation required for decoding video stream 642b as described herein.

[0118] If it is indicated that a (perspective) transformation of the reference image is required before decoding a specific region with reference to the reference image (such as that encoded in video stream 642b), then camera 110b can be configured to access data / information 620a directly from camera 110 or from interface module 670, which is responsible for collecting information from camera 110a (e.g., data / information 620a) and / or from camera 110b (e.g., data / information 620b), and to provide such information (e.g., 672) to client 650 upon request. In other examples, client 650 may alternatively or additionally obtain information corresponding to data 620a, for example, directly from camera 110b, as part of, for example, information 660 and / or as part of video stream 642b. For example, client 650 may receive information about the FOV of, for example, camera 110a via interface module 670 and receive such information about camera 110b directly from camera 110b (e.g., as part of information 660), or may receive such information about both cameras 110a and 110b from interface module 670 as part of data 672.

[0119] Furthermore, it is envisioned that, in some instances, the two cameras 110a and 110b can be configured, if needed, to exchange as much information as possible required for executing method 400, as indicated by dashed arrow 622. Cameras 110a and 110b can be connected, for example, via wires or wirelessly, and can be configured to exchange data as needed, for example, by exposing respective APIs / interfaces accessible to the other cameras and / or by each camera sending information to the other cameras in a push-like manner, or by a combination of both options. Cameras 110a and 110b can be configured, for example, to communicate via Bluetooth, WiFi, or other proprietary or open protocols / standards for such communication. Cameras 110a and 110b can be configured, for example, to be connected via wires using, for example, Ethernet standards or similar open or proprietary protocols for wired communication. Of course, it is envisioned that each camera 110a and 110b includes the hardware and / or software required to implement such communication, if needed.

[0120] Cameras 110a and 110b may include, for example, processing circuitry configured to perform various operations performed by each camera to execute method 400.

[0121] Generally speaking, Figure 6AThe illustration shows one or more examples of a video system in which one camera receives images from other cameras and uses this information to encode one or more of its own images and one or more of the images received from other cameras to output one or more video streams that encode the outputs of both cameras. The advantages are as already referenced herein. Figure 1 , Figures 2A to 2C , Figure 3A , Figure 3B , Figure 4 and Figure 5 As stated above.

[0122] Figure 6A Also used for illustration, this document envisions a camera, such as camera 110b, that can also independently execute method 400. Of course, the choice... Figure 1 Which of the cameras 110a-c in the diagram is designated as the "first camera" or the "second camera" depends on the circumstances. Terms such as "first camera" / "first image" and "second camera" / "second image" are merely labels used to indicate, respectively, from which camera the image is selected for reference and from which area on which the image is selected to be encoded using the "second image".

[0123] Figure 6B One or more other examples of the video camera system 601 envisioned herein are schematically illustrated. (See references) Figure 6A In contrast to the described system 600, in system 601, the responsibility for determining the existence of matching / similar image textures in two regions and, for example, encoding the images, is instead transferred from one of the cameras 110a and 110b to a centralized unit 680, such as a video processing / encoding server. In system 601, each camera 110a and 110b sends its respective captured images 210a and 210b as images 210a' and 210b' to unit 680. In some examples, images 210a' and 210b' are raw / unencoded data, while in other examples, at least one of the cameras 110a and 110b uses corresponding encoding modules 610a and 610b to encode the images before sending them to unit 680.

[0124] Unit 680 includes a receiving module 611 configured to receive images 210a' and 210b' from cameras 110a and 110b. In some examples, the receiving module 611 may also provide the function of decoding one or more of images 210a' and 210b' (if they are encoded), if desired. As used herein, therefore images 210a' and 210b' refer to images after they have been encoded, i.e., image data to which further analysis can be performed to detect regions of similar texture if necessary.

[0125] Unit 680 includes a determining module 640 configured, as described herein multiple times, to determine, for example, the presence of textureally similar regions in at least one image of each of images 210a' and 210b', and also, as described herein, to decode at least one region of at least one image of either 210a' or 210b' by referencing at least one region of at least one image of the other 210a' and 210b'. Therefore, module 680 may also include the function of encoding the one or more regions (referencing the one or more other regions due to texture similarity) to output one or more encoded video streams 644, such that these can be received and decoded, for example, by a user client 650. As described herein, module 680 may include, if desired, means necessary for performing, for example, image segmentation and / or for considering the spatial and FOV relationships of cameras 110a, 110b (based on data 620a, 620b).

[0126] In some examples, module 680 further includes a transformation module 630 for performing one or more perspective transformations based on data 620a and 620b, for example, indicating the spatial and FOV relationships of cameras 110a and 110b. Therefore, transformation module 630 is configured to transform at least one image (or at least one region of at least one image) of at least one of 210a' and 210b' before transmitting 210a' and 210b' to determination module 640. Of course, also in example system 600, an interface module 670 may be present, through which client 650 can collect information 674 as needed to perform one or more perspective transformations on the reference image, if necessary, for example, before being able to decode the image (region) of the reference image. Generally speaking, Figure 6A and Figure 6B For illustration purposes, as envisioned herein, the operations to be determined and encoded can be performed by one of the cameras or by a centralized function (e.g., module 680) that is not part of any of the cameras.

[0127] More generally, it is not critical in which particular entity method 400 is executed, and method 400 may be executed jointly by multiple entities in a distributed manner, such as by multiple cameras and / or by one or more cameras and a centralized server (e.g., unit 680). Of course, a “centralized server” may also include, for example, a cloud-based service, to which the cameras may communicate, for example, wirelessly or through one or more interfaces to which the cameras are wired and / or wirelessly connected.

[0128] In general, it should be noted that two cameras, such as 110a and 110b, mentioned in the description of method 400, can also form part of, for example, the same multi-view camera and / or dual-channel traffic camera. For example, one camera may be a fisheye camera configured to obtain a wider field of view of the scene, while the other camera may be a PTZ camera configured to provide a narrower / variable focal length field of view of the same or some other scenes. Even if two such cameras can be housed together in the same housing and offered / sold as a single multi-view camera, such examples will fall within the scope of this disclosure and the appended claims. For the purposes of this disclosure, it is sufficient to use different cameras (e.g., using different image sensors, etc.) to capture images in order to consider the system-wide field of view at encoding time, rather than simply considering each image from each camera individually. In other words, the envisioned solution is applicable both to multi-view cameras and to physically separate cameras, such as those located on different buildings, walls, and lampposts, arranged in physically different locations (i.e., not in the same location and / or within the same housing).

[0129] Figure 7A The illustration schematically depicts one or more examples of apparatus 700 for performing a method 400, as contemplated herein, for encoding images captured by separate video cameras. Apparatus 700 may be, for example, a video camera (e.g., a surveillance camera), or other apparatus external to such a camera but capable of acquiring images captured by the camera. As contemplated herein, “apparatus 700” may also be a collection of multiple apparatuses (e.g., various entities of a video camera system, such as multiple video cameras and / or a centralized server) configured to jointly perform method 400. Hereinafter, apparatus 700 will be described as a single apparatus, but the statements herein can of course be extended to multiple apparatuses, for example, by replacing a single processing circuit with multiple processing circuits, a memory with multiple memories, an interface with multiple interfaces, etc.

[0130] The apparatus 700 includes at least one processor (or “processing circuitry”) 710 and optional memory 712. As used herein, a “processor” or “processing circuitry” can be, for example, any combination of one or more suitable central processing units (CPUs), multiprocessors, microcontrollers (µCs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), etc., capable of executing software instructions stored in memory 712. Memory 712 may be external to processor 710 or internal to processor 710. As used herein, “memory” can be any combination of random access memory (RAM) and read-only memory (ROM) or any other type of memory capable of storing instructions. Memory 712 contains (i.e., stores) instructions that, when executed by processor 710, cause apparatus 700 to perform the methods described herein (i.e., method 400 or any embodiment thereof). Apparatus 700 may further include one or more additional items 714, which in some cases are useful for performing the method. In the case where device 700 is a camera, the additional item 714 may include, for example, an image sensor and one or more lenses for focusing light from the scene onto the image sensor, such that the camera can capture an image of the scene as part of performing the envisioned method. The camera may be a video camera configured to capture a stream of images. The additional item 714 may also include various other electronic components required for capturing the scene, such as for properly operating the image sensor and / or lenses as needed. Performing this method in a surveillance camera may be useful because processing is moved to the "edge," i.e., closer to where the actual scene is captured compared to performing, for example, image analysis elsewhere (e.g., at a more centralized processing server).

[0131] Device 700 may be connected to a network, for example, so that the results from performing method 400 can be transmitted to, for example, a user / operator and / or another device such as a server. For this purpose, device 700 may include a network interface 716, which may be, for example, a wireless network interface (as defined in any of the IEEE 802.11 or subsequent standards supporting, for example, Wi-Fi) or a wired network interface (as defined in any of the IEEE 802.3 or subsequent standards supporting, for example, Ethernet). Network interface 716 may also support, for example, any other wireless standard capable of transmitting encoded video, such as Bluetooth. Various components 710, 712, 714, and 716 (if present) may be connected via one or more communication buses 720, enabling these components to communicate with each other and exchange data as needed.

[0132] Device 700 may be, for example, a surveillance camera mounted on or potentially mounted on a building or other supporting structure, such as a PTZ camera capable of providing a wider field of view of the scene, or a fisheye camera, or any other type of surveillance / monitoring camera. Other examples include device 700 being a dashcam camera for a vehicle, a wearable camera for a person or animal, an unmanned camera, and similar cameras for non-stationary objects. In any such example of device 700, it is contemplated that device 700 may include all necessary components (if any) other than those already described herein required to perform the method 400 contemplated herein. In some examples, the various components of device 400 may be further configured to implement the method operations described herein (at least such as S410, S240, S460). As mentioned above, in other examples, device 700 may be distributed across multiple physical and / or logical entities to form, for example, a computer system, a video camera system, etc., wherein, for example as part of a distributed computing process, two or more of the operations (and / or two or more different sub-operations of the same operation) may be performed on / by different physical and / or logical entities.

[0133] Figure 7B One or more embodiments of the apparatus 700 are schematically illustrated based on multiple functional blocks / computation blocks 710a, 710b, and 710c. Figure 4As shown in the flowchart, each of these blocks 710a-c is responsible for performing a function according to a specific operation of method 400. For example, one such functional block 710a can be configured (as in operation S410) to acquire / receive images from separate cameras. Block 710a can be referred to as an image acquisition block / module, etc. Another block 710b can be configured (as in operation S440) to determine the texture similarity between regions. Block 710b can be referred to as a (texture similarity) determination block / module, etc. Another block 710c can be configured to encode an image (as in operation S460) as part of one or more encoded video streams, including encoding regions of an image by referencing texture-similar regions of other reference images. Block 710c can be referred to, for example, an encoding block / module or an encoder, etc. The apparatus 700 may optionally include, for example, one or more additional functional blocks, such as one or more blocks 710d, for example, blocks for encoding images captured therefrom before sending them to, for example, other cameras or central units (e.g., unit 680), for decoding images received from other cameras, for performing image segmentation (operation S420), for considering spatial relationships and FOV relationships between cameras (operation S430), and / or for determining perspective transformations (operation S450), etc., to provide / perform any operation of the method 400 envisioned herein, but not yet covered by any of blocks 710a-c.

[0134] Generally, each functional block 710a-d can be implemented in hardware or software. Preferably, one or more or all functional blocks 710a-d can be implemented by processing circuitry 710 in cooperation with storage medium / memory 712 and / or communication interface 716. Thus, processing circuitry 710 can be arranged to retrieve instructions provided by functional blocks 710a-d from memory 712 and execute those instructions, thereby performing any operation by means of apparatus 110b / 700 as disclosed herein / method 400 performed in apparatus 110b / 700 as disclosed herein or any embodiment thereof.

[0135] Similarly, device 700 or a device including similar hardware components may alternatively be configured as described in the reference. Figure 6B The video camera system 601 described herein executes method 500 in a distributed manner. For example, device 700 may then be configured to implement at least the functions of centralized module 680, namely, receiving images from cameras 110a, 110b, determining regions with texture similarity, and encoding a region with reference to other regions, as already described herein. This is achieved through a corresponding configuration of processing circuitry 710, memory 712, and / or, for example, additional entity 714 and communication interface 716.

[0136] Of course, means for decoding one or more encoded video streams generated by method 400 (e.g., for performing method 500) may also be provided. Although not illustrated in any of the figures, such means may include processing circuitry, memory, communication interfaces, and / or one or more other entities required to perform method 500.

[0137] Figure 8 Examples of computer program products 810a and 810b, including a computer-readable device / storage medium 830, are schematically illustrated. At least one of computer programs 820a and 820b (including computer code) may be stored on the computer storage medium 830. Computer programs 820a and 820b (when the code is executed) can cause processor 710 and entities and means operatively coupled thereto (e.g., communication interface 716 and memory 712) of device 700 to perform method 400. Therefore, computer program 820a and / or computer program product 810a can provide means for performing any operation of method 400 (or any embodiment thereof) jointly executed in a video camera system (e.g., 601), and computer program 820b and / or computer program product 810b can provide means for performing operations in a single camera (e.g., ...). Figure 6A The apparatus for any operation of method 400 (or any embodiment thereof) performed in the camera 110b shown in the figure.

[0138] exist Figure 8 In the examples, computer program products 810a, 810b and computer-readable storage medium 830 are illustrated as optical discs such as CDs (compressed optical discs), DVDs (digital versatile optical discs), or Blu-ray discs. Computer program products 810a, 810b and computer-readable storage medium 830 can also be implemented as one or more of random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM), and more specifically, as non-volatile storage media of devices in external storage such as USB (Universal Serial Bus) memory or flash memory such as flash memory. Therefore, although computer programs 820a, 820b are schematically illustrated herein as tracks on the depicted optical disc, computer programs 820a, 820b can be stored in any manner suitable for computer program products 810a, 810b and computer-readable storage medium 830. Storage medium 830 can be non-transitory or transient.

[0139] In summary, the improvement of this disclosure over the prior art lies in providing an improved method for encoding images captured by separate video cameras, wherein it takes into account the existence of regions with similar textures in the images captured by the separate cameras, and benefits from the knowledge that one such region can be encoded with reference to other regions. Such a system-wide perspective can, for example, improve the encoding efficiency of at least one of such regions, and save bitrate, for example, by encoding from a system-wide perspective rather than just from the individual perspective of each camera. This can be particularly beneficial as camera resolution increases, and it is necessary to limit computational complexity, the number of bits used to store the video stream, and the bitrate used to transmit the video stream.

[0140] Although features and elements may have been described above in specific combinations, each feature or element may be used alone without other features and elements, or may be used in various combinations with or without other features and elements. Furthermore, by studying the accompanying drawings, the disclosure, and the appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the claimed invention.

[0141] In the claims, the words “comprising” and “including” do not exclude other elements, and the indefinite articles “a” or “an” do not exclude plural. The fact that certain features are stated in different dependent claims does not mean that combinations of these features cannot be used beneficially.

Claims

1. A computer-implemented method for encoding images captured by separate video cameras, comprising: Receive the corresponding image from each of two separate video cameras; It is determined that a region in one of the images has an image texture similar to that of regions in other images; as well as The image is encoded as part of one or more encoded video streams, wherein encoding of the region of a first image in the image includes using the region of another second image in the image as a reference.

2. The method according to claim 1, wherein, The determination includes performing image segmentation on each of the images and confirming that the image segmentation indicates that two regions belong to the same category.

3. The method according to claim 1, wherein, The method further includes: acquiring the spatial relationship and field of view (FOV) relationship between the two cameras, wherein the spatial relationship and the FOV relationship include knowledge of one or more differences in physical location, physical orientation, zoom level, image sensor attributes and / or lens arrangement attributes between the two cameras, and wherein the determination includes confirming that the spatial relationship and the FOV relationship indicate that the two regions correspond to the same part of the scene captured by the two cameras.

4. The method according to claim 1, wherein, The method further includes: obtaining a perspective transformation, which, when applied to the second image, improves at least one of the spatial alignment and orientation of the region of the second image and the region of the first image, and wherein the reference is to the second image transformed by the perspective transformation.

5. The method according to claim 4, wherein, The method further includes providing data indicative of the perspective transformation via an interface accessible to the receiver of the one or more encoded video streams, such as a client, a receiver, and a decoder.

6. The method according to claim 1, wherein, The method further includes: selecting the first image from the images as an image with a higher pixel density in its region and selecting the second image as an image with a lower pixel density in its region.

7. The method according to claim 1, wherein, The method includes encoding two images as part of the same multi-layer video stream.

8. The method according to claim 7, wherein, The method includes: encoding the second image in the base layer of the video stream and encoding the first image in the enhancement layer of the video stream.

9. The method according to claim 1, wherein, The field of view (FOV) of the two cameras do not overlap.

10. A computer-implemented method for encoding images captured by a video camera, the method being performed by the video camera, the method comprising: Obtain corresponding images from each of the video camera and the other camera; It is determined that a region in one of the images has an image texture similar to that of regions in other images; as well as The image is encoded as part of one or more encoded video streams, wherein encoding of the region of a first image in the image includes using the region of another second image in the image as a reference.

11. A video camera system, comprising: Two separate video cameras; as well as The processing circuit is configured as follows: Receive corresponding images from each of the two separate video cameras; It is determined that a region in one of the images has an image texture similar to that of regions in other images; and The image is encoded as part of one or more encoded video streams, wherein encoding of the region of a first image in the image includes using the region of another second image in the image as a reference.

12. A video camera, comprising: At least one image sensor for capturing images; An interface for receiving additional images from other cameras; as well as The processing circuit is configured as follows: The image and the other image are obtained from the at least one image sensor and the other camera, respectively; Determine that a region in one of the images has an image texture similar to that of regions in the other images; and The image is encoded as part of one or more encoded video streams, wherein encoding of the region of a first image in the image includes using the region of another second image in the image as a reference.

13. A computer program comprising instructions that, when executed by processing circuitry of a video camera system comprising two separate video cameras, cause the video camera system to: Receive corresponding images from each of the two separate video cameras; It is determined that a region in one of the images has an image texture similar to that of regions in other images; and The image is encoded as part of one or more encoded video streams, wherein encoding of the region of a first image in the image includes using the region of another second image in the image as a reference.

14. A computer program comprising instructions that, when executed by processing circuitry of a video camera, cause the video camera to: Obtain images captured by at least one image sensor of the camera and other images captured by other cameras; Determine that a region in one of the images has an image texture similar to that of regions in the other images; and The image is encoded as part of one or more encoded video streams, wherein encoding of the region of a first image in the image includes using the region of another second image in the image as a reference.

15. A computer program product comprising a computer-readable storage medium storing a computer program according to at least one of claims 13 and 14.

Images