Real-time multi-source video pipeline

Abstract

The present disclosure provides an image processing apparatus and method, comprising one or more memories storing instructions and one or more processors, the one or more processors being configured, upon execution of the instructions, to: acquire one or more video sources of video data; specify a layout that identifies positions within a frame to place the acquired video sources; generate an output video stream based on the specified layout using pixel mapping to a target output frame displayed on a display of the computing device; provide the output video stream as input to one or more buffers; and dynamically render frames of the output video stream from the one or more buffers, where each frame is output in the specified layout, for each frame of the output video stream exiting the one or more buffers.

Description

[Background technology]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority from U.S. Provisional Patent Application Serial No. 63 / 292758, filed December 22, 2021, the entire contents of which are incorporated herein by reference. Field

[0002] FIELD OF THE DISCLOSURE This disclosure relates to image processing, and more particularly to generating a composite image from multiple image sources. 2. Description of Related Art

[0003] There are many applications that allow multiple different video data streams to be rendered in a composite layout. One such type of application is a video conferencing application that can receive real-time video from separate sources and display it in a single user interface. However, there is a need to improve the speed of compiling and rendering these layouts for real-time video processing. The systems and methods in the present disclosure remedy these shortcomings. Summary of the Invention

[0004] The present disclosure provides an image processing apparatus and method, comprising one or more memories storing instructions and one or more processors, the one or more processors being configured, upon execution of the instructions, to: obtain one or more video sources of video data; specify a layout identifying positions within a frame to place the obtained video sources; generate one or more pixel mappings to one or more output frame pixel locations, the one or more pixel mappings mapping each source of video data to the specified layout; obtain an output frame from one or more stream buffers; and dynamically render one or more output frame pixels from the one or more stream buffers via each of the one or more pixel mappings.

[0005] In one embodiment, the one or more buffers include at least a first buffer and a second buffer such that the second buffer is provided with subsequent frames of the output video stream while previous frames of the output video stream are being dynamically rendered. In another embodiment, the one or more stream buffers include at least a first stream buffer and a second stream buffer such that the second stream buffer is provided with subsequent frames of the output video stream while previous frames of the output video stream are being dynamically rendered. In another embodiment, the one or more pixel mappings to the one or more output frame pixel locations each include one or more pixel locations of one of the video sources of the video data. In another embodiment, the one or more pixel locations of one of the video sources of the video data include a predetermined number of neighboring pixels in the source video. In some examples, the pixel mappings further include respective weighting information associated with each of the predetermined number of neighboring pixels in the source video.

[0006] In another embodiment, the output video stream stores, for each target pixel in the pixel list, a pixel mapping of each target pixel back to a pixel in a respective one of the sources of video data. In this embodiment, the pixel mapping may include information mapping each pixel to a source pixel and a predetermined number of neighboring pixels proximate to the source pixel. In addition, the pixel mapping further includes weighting information associated with each of the predetermined number of neighboring pixels used to interpolate the source pixel to the target pixel.

[0007] In another embodiment, the image processing device generates a specified layout that includes a source boundary definition that identifies a portion of a video frame to be included as each one of the video sources, a target boundary definition that describes the location within the image buffer using relative pixel coordinates where each one of the video sources is stored, and orientation information that identifies an orientation of each one of the video sources as seen in the target frame.

[0008] In yet another embodiment, the image processor is configured to calculate the transformation from the source pixels to the target pixels using a two-dimensional transformation corresponding to a two-dimensional rotation and a two-dimensional translation from the coordinate space of the source pixels to the coordinate space of the target pixels.

[0009] These and other objects, features, and advantages of the present disclosure will become apparent from a reading of the following detailed description of illustrative embodiments thereof when considered in conjunction with the accompanying drawings and the appended claims. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 shows the image processing algorithm in this disclosure.

[0011] [Diagram 2] FIG. 2 shows the image processing algorithm in this disclosure.

[0012] [Diagram 3] FIG. 3 shows a hardware configuration in this disclosure.

[0013] Throughout the figures, unless otherwise stated, the same reference numerals and characters are used to denote like features, elements, components or portions of the illustrated embodiments. Moreover, while the subject disclosure will now be described in detail with reference to the figures, it is done so in connection with the exemplary embodiments. It is intended that changes and modifications can be made to the exemplary embodiments described without departing from the true scope and spirit of the subject disclosure as defined by the appended claims. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It should be noted that the following exemplary embodiments are merely examples for implementing the present disclosure, and can be appropriately modified or changed according to the individual structure and various conditions of the device to which the present disclosure is applied. Thus, the present disclosure is by no means limited to the following exemplary embodiments, and the described embodiments can be applied / implemented in situations other than those described below as examples, according to the figures and embodiments described below.

[0015] The present disclosure describes a multi-source video processing pipeline that takes one or more video or image sources and composes them into a video stream according to a provided layout. The system advantageously uses a predefined layout and resolution for the target output composite image, thus allowing these predefined characteristics to allow most of the computations required to perform image / video rendering to be pre-computed, and run-time (inter-frame) computations to be minimized to minimize real-time processing overhead. This advantageously reduces image processing costs and increases the speed at which video data streams can be rendered by a computing device.

[0016] The present disclosure provides image processing algorithms that compile multiple image sources (real-time video, still images, or a combination thereof) into a single output data stream, such that the images are presented in a predefined layout when output and displayed within a user interface on a display device. The algorithms are embodied as a set of computer-executable instructions that configure a processor (CPU) to execute the instructions. The algorithms may reside on any computing device, such as a laptop, desktop, or server computer, as long as the computing device is capable of acquiring image data from one or more imaging devices. Exemplary algorithms are described below.

[0017] FIG. 1 illustrates an exemplary embodiment of an image processing algorithm for creating an image pipeline that is rendered on a display device. Initially, at S101, the algorithm creates a new video pipeline by adding one or more desired image sources to be composited into a single composite display. The one or more image sources include one dynamic image content and static content. Exemplary types of dynamic content include, but are not limited to, video streams and / or images shared from a screen sharing feature of a computing device, such as when a user selects information from a window displayed on a display device and makes that information available to other computing devices. In some embodiments, the video stream includes a live capture video data stream being captured by an imaging device (e.g., a camera - still images or video). In other embodiments, the video stream may be a pre-recorded video that has been stored in memory for later use. Static content that may be included as one of the image sources of the newly created video pipeline includes a still image captured by an imaging device. In one embodiment, the still image is provided as an overlay image that is superimposed on top of another type of image source, such as a video data stream. In other embodiments, the static content may also include watermarks or other types of still images that include an alpha channel, whereby alpha blending is performed to allow all still images to be viewed on the display on top of other image sources such as video.

[0018] At S102, layout information is specified from one or more image sources to identify and process these sources according to a particular layout in the final output video stream for display. A source may specify a layout that includes one or more of: (a) an area in source image coordinates to be placed in the output (target) video stream; (b) an area in the output (target) video stream where the source is placed; (c) a Z-order of the source in the target, which identifies which image sources are rendered on top of each other if an overlay is one of the sources; and (d) a rotate and flip status to be applied to the source, which specifies that the source images are flipped and rotated from left to right (or vice versa) before being placed / rendered in the target video frame.

[0019] In some embodiments, the layout information of a source includes a source rectangle describing the crop rectangle of the source image to be placed in the target frame, a target rectangle describing the position of the rectangle in the target image buffer in relative coordinates (from 0.0 to 1.0) where the source should be placed, and the orientation of the source as viewed in the target frame. The source layout can be modified by several functions that generate new values ​​for the parameters that make up the source layout information. For example, the source may undergo operations such as, but not limited to, (a) rotate 90 degrees clockwise, (b) rotate 90 degrees counterclockwise, (c) flip left to right, (d) flip top to bottom, (e) move position by delta x and delta y, (f) resize to fit target frame (fitting to top or side borders as necessary so that the entire source is displayed in the target frame at its highest resolution while preserving the aspect ratio of the source), (g) resize to fill target frame (filling the window with the source while preserving the aspect ratio, thereby preventing the top and bottom of the source from being displayed, or the left and right sides from being displayed if the aspect ratio of the target does not match the aspect ratio of the source), (h) resize by percentage while preserving the aspect ratio, (i) resize the source to its original size and center it in the target frame, (j) crop the source according to the new source rectangle (updating the target rectangle appropriately), and (k) uncrop the source.

[0020] The Source Layout class also provides a mechanism to compute the transformations from source pixels to target pixels. These transformations are expressed as 2D transformations that correspond to a 2D rotation and a 2D translation from one coordinate space to another. The 2D transformations can be parameterized as follows:

number

number

number

number

number

[0021] In S103, the algorithm generates an output video stream that includes all source streams and compiles the video streams according to the provided layout to pre-calculate pixel mappings for the target video frames. In S103, the compilation of the streams may separate the target pixels into different types of target pixels. A first type of target pixel includes pixels that are purely static content, that is, pixels from image sources that never change between frames and only need to be rendered once into the target image buffer(s) and do not need to be updated for each frame of the video. A second type of target pixel is a pixel that is composed of dynamic content. These pixels may change between frames and are repeatedly rendered via a rendering function that is called every frame. A third type of target pixel is a combination of dynamic and static content. The third type of target pixel includes pixels that include static content (e.g., watermarks and overlay images) and alpha-blended dynamic content (e.g., video). In certain embodiments, the third type and the second type of target pixel are combined together. The fourth type includes a pixel of dynamic content that is alpha blended with one or more pixels of dynamic and / or static content.

[0022] The compiled video stream pipeline stores, for each target pixel in the list, a pixel mapping of the target pixel back to the source pixel(s). In one embodiment, the mapping includes information mapping the nearest pixel of the appropriate source (the source with the highest Z-order). In another embodiment, the mapping includes information mapping the four nearest neighbors and their associated weightings used for bilinear interpolation of the source pixel to the target pixel. In another embodiment, the mapping may include information for mapping additional source pixels and their associated weightings used for other interpolation methods (e.g., bicubic or Lanczos).

[0023] Now that the video data stream has been generated with the appropriate source-to-target mapping information, a buffer is initialized in S104 to receive successive frames of the video data stream that are output and communicated for display on a display device. In some embodiments, multiple buffers may be used, such that one buffer can be updated while a previously rendered buffer is read by the video consumer. The buffer or buffers are initialized using a buffer initialization function that renders pixels associated with static content (only once) and may also clear (set to a constant color) all other pixels that are not associated with any content.

[0024] At S105, the video frames in the one or more buffers are dynamically rendered in the specified layout for display on the display device. In doing so, for each frame of the video call, a Render function is called to render the output pixels of all the dynamic content. In some embodiments, a second rendering operation is performed to perform a second pass on the pixels to perform alpha blending of the dynamic content and the watermark pixels to output a visible watermark. Since the rendering of each output pixel is independent, the rendering of the output pixels in the target frame may be performed via an independent thread or GPU core.

[0025] When rendering a target frame, source image buffers may be acquired so that they are marked in a read-only state. In this way, dynamic input buffers may be locked from updates while they are being read to ensure that all data rendered from the source originates from the same frame. Once the read is complete, the buffer is released by the renderer and the buffer's read count (the number of requested read instances) is decremented. Information may be written to the image buffer simultaneously, but the image buffer should have separate buffers for reading and writing. A thread / process requesting read access (e.g., for rendering) gets the most recently updated buffer in the image buffer that is not held with write access. A thread / process requesting write access (to update a frame) gets any buffer that is not currently being written to or read from.

[0026] In an exemplary rendering operation such as that described above with respect to FIG. 1, where a two-dimensional transform is determined based on the source layout, and an advanced interpolation mode (such as bilinear interpolation) is used, a pixel mapping from each target pixel to the nearest neighbor(s) of the appropriate source pixel can be calculated with appropriate weighting for the source pixels. In some embodiments, the four nearest source pixels are found for each target pixel, with the weighting corresponding to the pixel to be used for bilinear interpolation. The source pixel that is closest to the inverse transform map of the target pixel is the "nearest neighbor." This pixel has the highest weight when performing a weighted average of the four nearest neighbors with bilinear interpolation.

[0027] For example, consider a source image (say the third source) of size 100x100 with 3-byte pixels [R,G,B]. If the four nearest source pixels corresponding to a target pixel with offset 6000 have source buffer offsets of [30,33,330,333] with corresponding weights of [0.1,0.3,0.15,0.45], then the offset of the nearest source pixel in the source image is 333 since it has the highest weight. In some embodiments, for each target pixel being rendered, the offset of the target pixel (6000 in the above example), the offsets of the four nearest source pixels ([30,33,330,333] in the above example), the weights of the four nearest source pixels ([0.1,0.3,0.15,0.45] in the above example), and the buffer ID of the source image (3 since it is the third source in the above example) are stored. This speeds up the image rendering process. In another embodiment, the index of the most highly weighted pixel is stored, and the difference to the next row and each neighboring pixel is stored. Let [p1,p2,p3,p4] be the indexes of the four source pixels, with p1 being the nearest neighbor pixel, p2 being the pixel in the row adjacent to p1, p3 being the pixel next to p1 (same row as p1), and p4 being the pixel next to p2 (same row). Then [p1,dr,dc] may be stored, with dr=p2-p1 and dc=p3-p1. Furthermore, the pixel index p1 may be a relatively large number, while dr is relatively small since it is at most + / - the magnitude of one row of pixels, and dc has a value of + / -1. In the above example, we may store [333 (highest weighted pixel) as a 32-bit unsigned integer, -300 (row offset as a 16-bit signed integer), -3 (as an 8-bit signed integer)], and these values ​​can be used to compactly store and efficiently restore p1,p2,p3,p4. Furthermore, in some embodiments, the weights are stored as unsigned integers (e.g., 8-bit integers), in which case 4 bytes may be used to store the weightings. Of course, since the weights should sum to 256 and the last weight can be inferred, 3 bytes may be used.If one weight is 256, then the mapping of the pixel can be changed to [p1,p1,x,y] with weights [255,1,0,0], where the values ​​of x and y do not matter.

[0028] Additional embodiments reorder the list of nearest neighbor pixels and weights to ensure that the nearest neighbors (e.g., source pixels carrying the highest weights) are listed first. In the above example, swapping the first and last entries would change the source offset to [333,33,330,30] and the weight to [0.45.0.3,0.15,0.1]. In this way, the first pixel in the source offset list always represents the nearest neighbor pixel, allowing the rendering process to use either bilinear or nearest neighbor interpolation without additional computation.

[0029] This allows the rendering loop to be optimized for quick source pixel lookup and weighting, as described below. An exemplary rendering loop for N target pixels using bilinear interpolation is shown below. From this, it is clear that the total number of operations occurring during this pixel rendering loop is minimized, and most operations are pre-computed at the time of pipeline compilation. During this process, for each destination pixel to be updated, a reference to the destination pixel is obtained along with information about pixel mapping information that maps the source to the target pixel. In obtaining the pixel mapping information, a pointer to the image buffer of the pixel source is obtained and read based on the source information. The source pixel index [p1,p2,p3,p4] and pixel weights [w1,w2,w3,w4] are retrieved, and for each channel of the pixel (e.g., red, green, blue, and alpha channels), the destination channel is set equal to the sum of each corresponding source pixel index multiplied by the pixel weight corresponding to that pixel.

[0030] Another exemplary rendering loop for N target pixels using nearest neighbor interpolation is shown below. In this embodiment, no operations occur in this pixel rendering loop other than memory indexing. In this embodiment, for each destination pixel to be updated, a reference to the destination pixel is obtained along with the mapping information for that pixel. A pointer to the image buffer of the pixel source is retrieved and loaded based on the source information, and for each channel of the pixel (e.g., red, green, blue, and alpha channels), the destination channel is set equal to the index of the source pixel.

[0031] The two example rendering loops above advantageously use similar pre-computed pixelInfo information. The first technique is more computationally expensive than the second, but provides better quality bilinear interpolation at the expense of more render-time operations due to the weighted average of four pixels. The second example always uses the nearest neighbor pixel, which is the first enumerated pixel offset. However, either the first or second example above offers different advantages over current compilation algorithms by relying on pre-computed operations and mappings from source to target pixels.

[0032] The above algorithms are provided as examples. Operations can be further accelerated by treating pixels as multi-byte objects (e.g., 32-bit pixels) and processing the 32 bits simultaneously, rather than byte-by-byte (per channel). In some embodiments, the above algorithms are implemented using single-input / multiple-data instructions (SIMD). Yet other embodiments perform these tasks in parallel via a GPU. The exemplary algorithm loads four source pixels (32 bits each) into 128-bit registers (one byte for each channel) as uint8s (8-bit unsigned integers). [[r1,g1,b1,a1],[r2,g2,b2,a2],[r3,g3,b3,a3],[r4,g4,b4,a4]] The unsigned bytes are converted to unsigned 16-bit integers and stored in a new 256-bit register as unit16. The 32-bit weight vector [X,X,X,[w1,w2,w3,w4]] is loaded into a 128-bit register and the weights are expanded to 128 bits as follows: [[0,0,0,w1],[0,0,0,w2],[0,0,0,w3],[0,0,0,w4]] Each 32-bit integer with an 8-bit weight is multiplied by 0x01010101 and expanded as follows: [[w1,w1,w1,w1],[w2,w2,w2,w2],[w3,w3,w3,w3],[w4,w4,w4,w4]] This is then expanded as a unit16 and each short is multiplied as follows, with the lowest 16 bits preserved as follows: [[w1*r1, w1*g1, w1*b1, w1*a1],[w2*r2, ....]=[[wp1],[wp2],[wp3],[wp4]] Then, 64-bit words are added horizontally (e.g., four consecutive 64-bit integers are summed): [[wp1+wp2+wp3+wp4]]. In this exemplary embodiment, the four nearest neighbor pixels are weighted according to a predefined weighting via a series of special 128-bit and 256-bit instructions, which allows many operations to be performed simultaneously.

[0033] According to another embodiment, the rendering process performed as described above may be performed according to a list of rendering tasks based on the priority according to which the rendering process must be performed. This is illustrated in Figure 3, which shows a frame rendering pipeline organized by rendering priority and source rendering according to the rendering method to be performed. The pipeline breaks down a video frame rendering into one or more lists (sometimes homogeneous) of pixel mapping tasks, where each pixel mapping task contains the data and operations required to render one or more pixels in a target output frame.

[0034] As an example, a target output frame pixel may be a linear combination of multiple pixels from the source images. The combination or weights of the source pixels used may be pre-calculated for the mapping task. As an example, a target pixel may be the average of the four nearest neighboring pixels in the source images, with equal weighting, such that the red channel of the target pixel is the average of the red channels of the corresponding four source pixels. In this example, the weighting of each target pixel is one-fourth. However, in general the weightings may vary and be unequal, but essentially the weightings sum to one. Similarly, weightings are applied to the green, blue, and sometimes the alpha channels of the four source pixels. The result is as follows: r_target=(r1+r2+r3+r4) / 4 g_target=(g1+g2+g3+g4) / 4 b_target=(b1+b2+b3+b4) / 4 α_target=(α1+α2+α3+α4) / 4

[0035] According to this framework, bilinear interpolation may be achieved by appropriately selecting the weighting of each pixel mapping. All target pixels that originate from the same source and follow the same mapping technique (e.g., nearest neighbor mapping, bilinear mapping, bicubic mapping, mapping implemented by parallel execution, etc.) may be placed in a mapping task list. A collection of mapping task lists may also be organized such that a system consisting of parallel processors and processors capable of vectorized data processing may be able to efficiently execute video rendering tasks. Tasks of the same priority are generally formed as independent tasks and do not depend on each other, but may depend on tasks of higher priority. Because tasks of the same priority are independent of each other (e.g., each operation and each modification of an independent output pixel), these tasks can be distributed across computing CPU cores or GPU cores. For example, if a processor has four cores, four tasks can be executed simultaneously because all four tasks are modifying different destination pixels (different memory locations).

[0036] Additionally, when rendering pixels in a target output frame, some pixels may combine pixels from multiple sources. For example, one or more target pixels may be composed via alpha blending of multiple source pixels, which may include weighting of source pixels according to some interpolation scheme. In these cases, it may often be advantageous to render the pixel incrementally, starting with the source with the lowest z-order, and rendering partially transparent sources "on top" of pixels of lower z-order according to a specified z-order or source. In this case, a mechanism can be provided for the system to render all lower z-order portions of a pixel first before rendering the semi-transparent pixel on top of the previous rendering.

[0037] This is illustrated in FIG. 2, whereby a first list 201 is shown having a first priority level 202 associated therewith. As shown in this example, this rendering task list includes images from sources 203a, 203b, 203c, respectively, each of which is rendered according to the same rendering methodology. The image from source 203a has three rendering tasks associated therewith, while the images from sources 203b and 203c have two and one rendering tasks associated therewith. FIG. 2 also illustrates a second list 210 having a second priority level 211 associated therewith. As shown in this example, this rendering task list 210 includes images from sources 213a, 213b, 213c, respectively, each of which is rendered according to the same rendering methodology. The image from source 213a has three rendering tasks associated therewith, while the images from sources 213b and 213c have two and one rendering tasks associated therewith. In the above task allocation structure, homogeneous (by method) mapping tasks are organized according to priority and input source. The start of frame rendering begins with the highest priority rendering. By defining the rendering priorities, the system ensures that all higher priority renderings are performed before the next highest priority rendering is performed. This allows rendering of content with some transparency (specified in the alpha channel of the pixels) that is rendered and blended into the result obtained through the higher priority rendering. For each source-based task list for the selected priority (these are independent of all other pixel mapping tasks and can be executed in parallel), a source to target mapping method is determined, along with whether the determined mapping method uses the source's alpha channel to blend with previously rendered content.Since each task is responsible for mapping an independent target pixel, e.g., through threading or parallel processing, such as a GPU, pixel rendering is performed according to the use of alpha blending and a mapping method that allows pixel rendering to be performed in parallel. In other embodiments, pixel rendering may use vectorization to process red, green, blue, and alpha channels in parallel. Or, it may process multiple pixels in one operation via vectorization. This process of parsing and processing the various task lists shown in FIG. 2 is repeated until there are no lower priority lists. At that time, rendering stops.

[0038] With respect to the various rendering tasks that may be associated with a particular source video, there are several different embodiments that implement these tasks. In one embodiment, the rendering task may be nearest neighbor interpolation, where the task includes a source pixel offset to nearest source pixel mapping.

[0039] In another embodiment, the rendering task may be bilinear interpolation, whereby the task includes four nearest neighbor offsets in the source image and at least three weights for the first three nearest neighbors (the fourth weight is implied since the weights will generally sum to a fixed number so that all pixels are configured with a constant total weighting). The weights may be specified as floating point or integer numbers. In one embodiment, the weights are given as numbers between 0 and 256, with the sum of the weights being 256. Furthermore, since the first weight cannot be 0 if the sum of all weights is 256 and the first weight is the maximum weight, some embodiments store the pixel with the highest weight first, allowing the first weight of 256 to be stored as 0. Thus, the four weights may be stored in memory as 32 bits (8 bits per weight, 4 weights). In another embodiment, the four weights are stored as four 16-bit short integers. In yet another embodiment, the weights are provided as 16-bit short integers between 0 and 256 for each channel of up to four of the corresponding four nearest neighbor pixels in the source image. Thus the weights are given as 16 16-bit integers, w = [w0, w0, w0, w0, w1, w1, w1, w1, w2, w2, w2, w2, w3, w3, w3, w3] A total of 256 bits. While storing the weights in this manner is not memory efficient, some embodiments use this structure to optimize real-time processing of pixels, as described further below.

[0040] In some configurations, the processor of the mapping device can support single instruction multiple data operations. For example, some processors support 256-bit operations. One such embodiment loads the four source pixels into 256-bit registers, storing each pixel channel as a 16-bit word of the following format: p = [r0, g0, b0, a0, r1, g1, b1, a1, r2, g2, b2, a2, r3, g3, b3, a3] = [q0, q1, q2, q3] where qi is the 64-bit concatenation of each pixel channel as the 16 bits of the i-th pixel. Multiplying each 16-bit unit by p and w produces: p*w = [w0*r0, w0*g0, w0*b0, w0*a0, w1*r1, …] In this embodiment, the maximum value of any weight is 256 or 0x100 (hexadecimal), and the value of each pixel channel is 0 to 255 (0x00 to 0xff hexadecimal), so the result of the multiplication will never overflow a 16-bit boundary. Calculating p*w and summing the elements as 64-bit unsigned integers determines: x = sum_64(p*w) = [w0*r0+w1*r1+w2*r2+w3*r3, w0*g0+w1*g1+w2*g2+w3*g3, …] Because the sum of the pixel weights never exceeds 256, and the value of each pixel channel never exceeds 255, this sum of 64-bit unsigned integers effectively performs the sum of four 16-bit unsigned integers. Therefore, there is no possibility for the addition to cross a 16-bit boundary. The sum above includes the final pixel channel value (multiplied by the sum weight of 256) shifted by 8 bits. This means that selecting the first, third, fifth, and seventh bytes of the 64-bit unsigned integer x returns the r, g, b, and a channels of interest in 8-bit chunks. By using large CPU registers (256-bit in this case) to process the pixel channels simultaneously, the calculations can be performed more efficiently than via standard programming methods.

[0041] In another embodiment, the rendering task includes a bicubic interpolation method that calculates additional source pixels and weights. For example, the bicubic interpolation method may store the locations and weights of the 16 closest source pixels to estimate the destination pixel value via bicubic interpolation. The source pixel offsets and weights may be pre-calculated in a manner similar to how the bilinear pixel offsets and weights are pre-calculated.

[0042] In another embodiment, the rendering task includes alpha blending (nearest neighbor, bilinear, or bicubic). If the source specifies that alpha blending is to be used, the task first calculates an interpolated pixel and then uses the alpha channel to alpha blend the pixel against existing pixel values ​​that were rendered with a higher priority.

[0043] Other rendering tasks include pixel image adjustments that are applied once pixel values ​​are calculated. Such adjustments can change brightness, contrast, saturation, and hue. These adjustments are typically performed via pixel lookup tables or real-time computation of resulting pixel values ​​taking into account current intermediate pixel values. These adjustments do not have memory requirements for each mapping, but typically require storage of commonly used pixel lookup tables and pixel value transformation parameters. Pixel adjustments are typically performed at the lowest priority to ensure that all prerequisite pixel value computations have already been performed before pixel adjustments are made.

[0044] Another rendering task is the rendering of static content, whereby sometimes the rendered pixels are generated from static content, for example a background image. If the target frame pixels depend only on static content (e.g. inputs that do not change between frames), then these pixels may be rendered only once in the output buffer. These pixels do not need to be updated every frame. However, if image adjustments are applied, the mapping of the static inputs needs to be updated according to the new picture settings. Thus, the rendering task may further query the output frame (according to the internal frame flags) to determine if the static content of the frame is up to date. If not, the rendering task updates the static content when the frame is marked dirty or "stale". For this to work properly, the processing pipeline must tell the output frame when to mark the internal buffer as dirty, so that the rendering process can determine if the static content needs to be redrawn for any given frame. This is done by providing out-of-date information associated with each still image information, ensuring that what is finally rendered is appropriate in terms of time and image source.

[0045] FIG. 3 shows an exemplary hardware representing an apparatus 300 that can be used to execute the image processing algorithms described above and implement the present disclosure described above. The apparatus includes a CPU 301, a RAM 302, a ROM 303, an input unit 304, an external interface 305, and an output unit 306. The CPU 301 controls the apparatus 300 using a computer program (one or more stored sequences of instructions executable by the CPU) and data stored in the RAM 302 or the ROM 303. Here, the apparatus may include one or more dedicated hardware or a graphic processing unit (GPU) different from the CPU 301, and the GPU or dedicated hardware may execute a part of the processing by the CPU 301. Examples of dedicated hardware include an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), and the like. The RAM 302 temporarily stores computer programs or data read from the ROM, data supplied from the outside via the external interface 305, and the like. In the application described herein, the external interface 305 receives information in the form of image data from multiple image sources. In one embodiment, the imaging device 307 captures image data according to a particular image capture region and provides the captured image to the apparatus 300 via the external interface 305. Although the imaging device 307 is described generally, it is understood that the imaging device is a camera configured to capture real-time video images (and audio) or still images, or both. In another embodiment, an image source 308 is provided, and the apparatus 300 can obtain pre-recorded image data from the image source 308 via the external interface 305. For example, the image source may be an external storage having pre-recorded image data stored therein. In another example, the image source 308 may be the output from a particular application running on an external device, such as another computer, where a user of that computer has selected a window or user interface displayed on the user's computer for sharing with other users.The external interface 305 communicates with external devices such as a PC, a smartphone, a camera, etc. The ROM 303 stores computer programs and data that do not need to be modified and can control the basic operation of the device. The input unit 304 is composed of, for example, a joystick, a jog dial, a touch panel, a keyboard, a mouse, etc., and receives user operations and inputs various instructions to the CPU 301. The communication with the external device may be performed wired using a local area network (LAN) cable, a serial digital interface (SDI) cable, a WIFI connection, etc., or wirelessly via an antenna. The output unit 306 is composed of, for example, a display unit that displays a graphical user interface (GUI) thereon such that a display of information is provided to the display device 309. The output unit 306 further includes an audio output unit such as a speaker. The output unit 306 may also include a network connection, whereby the video stream generated from the video processing embodiment described in FIG. 1 and FIG. 2 is transmitted to a remote system.

[0046] The scope of the present disclosure includes non-transitory computer-readable media that store instructions that, when executed by one or more processors, cause the one or more processors to perform one or more embodiments of the present invention described herein. Examples of computer-readable media include hard disks, floppy disks, magneto-optical disks (MOs), compact disk-read-only memories (CD-ROMs), compact disk-recordables (CD-Rs), CD-rewritables (CD-RWs), digital versatile disk ROMs (DVD-ROMs), DVD-RAMs, DVD-RWs, DVD+RWs, magnetic tapes, non-volatile memory cards, and ROMs. Computer-executable instructions can also be provided to the computer-readable recording media by being downloaded over a network.

[0047] The use of "a" and "an" and "the" and similar references in the context of this disclosure describing one or more embodiments of the invention (particularly in the context of the claims below) is to be construed as covering both the singular and the plural, unless otherwise indicated herein or otherwise clearly contradicted by context. The terms "having," "comprising, including, containing" are to be construed as open-ended terms (i.e., meaning "including, but not limited to"), unless specifically noted otherwise. The recitation of ranges of values ​​herein is intended merely to serve as a shorthand method for individually referring to each individual value falling within the range, unless otherwise indicated herein, and each individual value is incorporated herein as if set forth individually herein. All methods described herein can be performed in any suitable order, unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples or exemplary language (e.g., "etc.") provided herein is intended merely to better clarify the subject matter disclosed herein and does not limit the scope of any invention(s) arising from this disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential.

[0048] It will be understood that the present disclosure can be incorporated in the form of various embodiments, only a few of which are described herein. Variations of these embodiments will become apparent to those skilled in the art upon reading the above description. Accordingly, the present disclosure and any inventions derived therefrom include all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, unless otherwise indicated herein or otherwise clearly contradicted by context, any combination of the above-described elements in all possible variations thereof is included in the present disclosure.

Claims

1. An image processing device, one or more memories for storing instructions; one or more processors, the one or more processors, when executing the instructions, Obtaining one or more video sources of video data; specifying a layout that identifies positions within a frame in which to place the acquired video sources; generating one or more pixel mappings to one or more output frame pixel locations, the one or more pixel mappings mapping respective sources of video data to the specified layout; Obtaining output frames from one or more stream buffers; dynamically rendering one or more of the output frame pixels from the one or more stream buffers via the one or more pixel mappings, respectively; The image processing device is configured to:

2. 2. The image processing apparatus of claim 1, wherein the one or more stream buffers include at least a first stream buffer and a second stream buffer such that a second stream buffer is provided with a subsequent frame of the output video stream while a previous frame of the output video stream is being dynamically rendered. Image processing device.

3. 2. The image processing apparatus of claim 1, wherein the one or more pixel mappings to one or more output frame pixel locations each include one or more pixel locations of one of the video sources of video data. Image processing device.

4. 4. An image processing apparatus as claimed in claim 3, wherein the one or more pixel locations of one of the video sources of video data include a predetermined number of neighbouring pixels in the source video. Image processing device.

5. 5. The image processing apparatus of claim 4, wherein the pixel mapping further comprises respective weighting information associated with each of the predetermined number of neighboring pixels in the source video. Image processing device.

6. 2. The image processing device according to claim 1, wherein the specified layout is: a source boundary definition that identifies a portion of a video frame to be included as each one of said video sources; a target boundary definition describing the location within the image buffer using relative pixel coordinates at which said respective ones of said video sources are stored; orientation information that specifies an orientation of the respective one of the video sources as viewed in the target frame; 23. An image processing device comprising:

7. 2. The image processing apparatus of claim 1, wherein execution of the instructions causes the one or more processors to: calculating a transformation from said source pixels to said target pixels using a two-dimensional transformation corresponding to a two-dimensional rotation and a two-dimensional translation from a coordinate space of source pixels to a coordinate space of target pixels; The image processing device is further configured to perform the steps of:

8. 1. An image processing method, comprising: Obtaining one or more video sources of video data; specifying a layout that identifies positions within a frame in which to place the acquired video sources; generating one or more pixel mappings to one or more output frame pixel locations, the one or more pixel mappings mapping respective sources of video data to the specified layout; Obtaining output frames from one or more stream buffers; dynamically rendering one or more of the output frame pixels from the one or more stream buffers via the one or more pixel mappings, respectively; An image processing method comprising:

9. 9. The image processing method of claim 8, wherein the one or more stream buffers include at least a first stream buffer and a second stream buffer such that a second stream buffer is provided with a subsequent frame of the output video stream while a previous frame of the output video stream is being dynamically rendered. Image processing methods.

10. 9. A method of image processing as claimed in claim 8, wherein the one or more pixel mappings to one or more output frame pixel locations each include one or more pixel locations of one of the video sources of video data. Image processing methods.

11. 11. A method of image processing as claimed in claim 10, wherein the one or more pixel locations of one of the video sources of video data include a predetermined number of neighbouring pixels in the source video. Image processing methods.

12. 12. The image processing method of claim 11, wherein the pixel mapping further comprises respective weighting information associated with each of the predetermined number of neighboring pixels in the source video. Image processing methods.

13. 9. The image processing method according to claim 8, wherein the specified layout is a source boundary definition that identifies a portion of a video frame to be included as each one of said video sources; a target boundary definition describing the location within the image buffer using relative pixel coordinates at which said respective ones of said video sources are stored; orientation information that specifies an orientation of the respective one of the video sources as viewed in the target frame; An image processing method comprising:

14. 9. The image processing apparatus of claim 8, wherein execution of the instructions causes the one or more processors to: calculating a transformation from said source pixels to said target pixels using a two-dimensional transformation corresponding to a two-dimensional rotation and a two-dimensional translation from a coordinate space of source pixels to a coordinate space of target pixels; The image processing device is further configured to perform the steps of:

Images