Method and system for controlling the appearance of an LED wall and creating digitally augmented camera images - Patent Application 20070122999

Abstract

The present invention provides a method, preferably a computer-implemented method, for controlling the appearance of an LED wall 11 depending on the angle of view of a camera 2 relative to the LED wall 11, the method comprising: h , α v and determining the appearance of the LED wall 11 based on the determined LED wall view angles α and α h , α v Furthermore, the present invention proposes a method for creating a digital augmented camera image based on an input image including image data of the LED wall 11 taken by a camera 2, and a control system for controlling the appearance of said LED wall 11 depending on the angle of view of the camera 2 relative to the LED wall 11.

Description

[Technical Field]

[0001] The present invention relates to a method for controlling the appearance of an LED wall depending on the camera's angle of view according to the subject matter of claim 1, a method for creating a digital augmented camera image based on an input image taken by a digital camera according to the subject matter of claim 2, and corresponding (processing) systems according to the subject matter of claims 11 and 13. [Background technology]

[0002] Extended reality is an umbrella term covering various technologies that combine real-world camera footage captured by a digital camera with virtual content image data, such as computer-generated virtual graphics. These technologies include virtual reality, augmented reality, and mixed reality. To create these effects, the real camera footage must be combined with rendered virtual graphics or virtual content image data. To ensure that the combination of the real camera footage and the virtual content image data appears congruent, the virtual content image data must be rendered with the correct perspective. To ensure accurate rendering of the virtual content image data, the camera position and / or orientation must be determined with high spatial and temporal precision.

[0003] In XR, the goal is often to synthesize real-world camera footage with virtual content image data so that the virtual elements blend seamlessly. This way, viewers can enjoy the added virtual elements while still feeling like everything they see is real. To achieve this goal, the virtual content image data must be rendered to match the real-world image data as closely as possible. For example, if a real-world camera uses optical lenses that distort the images it captures, the virtual content image data must also be rendered with distortion to mimic the effect of the real lens on the images it captures.

[0004] Virtual reality effects have been achieved using so-called green screens. These are areas uniformly covered in a particular shade of green placed in the background of a scene being filmed by a camera. While other colors can and have been used, green is the most common. In the filmed image, a particular shade or hue of the background color can be recognized and replaced with other video content. This allows virtual content image data to be inserted as a background to actors or objects filmed in the scene. This process involves selecting areas of the filmed image that are set transparent based on the frequency of a particular color or hue present in the background screen (e.g., a particular shade of green in the case of a green screen), thereby allowing virtual content image data to be inserted into the scene. This process is called chroma keying.

[0005] A problem that arises with the use of green screens and chroma keying relates to the color-based selection of image areas to be replaced with virtual content image data. In real-world scenarios, the green background in a camera image does not have a perfectly uniform tone or hue due to uneven lighting and color distortions caused by the camera itself. With this in mind, it is appropriate to allow a certain degree of latitude in the color selection for chroma keying. However, the degree of latitude must be chosen carefully. If the color or hue selection is too narrow, not all areas to be augmented by the virtual content image data will be selected, resulting in green artifacts appearing in the augmented image. Excessive latitude in the color or hue selection may result in image content being inadvertently removed.

[0006] An additional challenge in using green screens is adjusting the lighting so that the real footage matches the virtual graphics. This is especially difficult if the graphics are changing over time. For example, if the actual real-world light source remains stationary while the light source in the virtual graphics moves, the resulting footage combining real and virtual elements will look unnatural.

[0007] For these reasons, LED walls are preferred over green screens when producing high-quality virtual studios. An LED wall is a digital display made up of many individual light-emitting diodes (LEDs) arranged in a grid (i.e., an LED grid).

[0008] Extended reality technology using LED walls was recently popularized by Lucasfilm in the hit Star Wars series "The Mandalorian," and has since rapidly gained popularity, especially during the COVID-19 pandemic, which has forced the film industry to rethink how movies are made and adhere to safety restrictions in order to reduce the number of people on set. This new XR technology is a modern alternative to green screen studios.

[0009] Problems arising from the use of LED walls are related to their limited dimensions. LED walls of satisfactory quality are quite expensive. Therefore, budgetary constraints often dictate the use of smaller LED walls, which do not fill the entire background of the scene captured by the camera, especially when the camera is located at a certain distance from the LED wall. In such cases, it is necessary to augment the captured image with virtual content image data in areas outside the LED wall to artificially stretch the LED wall graphics into areas of the camera image where the LED wall is not yet visible. If the colors of the virtual content image data do not exactly match the colors on the LED wall, undesirable artifacts will be visible at the boundaries of the LED wall in the digitally augmented camera image.

[0010] A further problem associated with the use of LED walls is the fact that the brightness and / or color of the LEDs in an LED wall can vary depending on the angle from which the LED wall is viewed. As the angle of the camera relative to the LED wall surface changes during capture of a scene, the brightness and color of areas within the LED wall can change. This can further reduce the satisfactory alignment between the virtual content image data and the input image data, and can introduce artifacts into the digitally augmented camera image, particularly at the boundaries of the LED wall. Summary of the Invention

[0011] As described above, it is clear that there remains a need for improvements in XR technology using LED walls. Therefore, the present invention aims to provide a solution for improving the quality of camera images depicting LED walls. In particular, the present invention aims to reduce visual artifacts in camera images depicting LED walls and digitally augmented camera images that include depictions of LED walls.

[0012] The above mentioned object is solved by a method for controlling the appearance of an LED wall depending on the angle of view of a camera according to the subject matter of claim 1, a method for creating a digital augmented camera image based on an input image taken by a digital camera according to the subject matter of claim 2, a control system for controlling the appearance of an LED wall depending on the angle of view of a camera relative to the LED wall according to the subject matter of claim 11, and a digital image processing system for creating a digital augmented camera image based on an input image comprising image data of an LED wall according to the subject matter of claim 13. Preferred embodiments of the invention are defined by the subject matter of the dependent claims.

[0013] Specifically, the problem is solved by a method, preferably a computer-implemented method, for controlling the appearance of an LED wall in response to a camera's angle of view relative to said LED wall, comprising: a) determining an LED wall angle of view of a camera relative to the LED wall; b) adjusting the appearance of the LED wall based on the determined LED wall angle of view.

[0014] The subject of the present invention is a method for creating a digital augmented camera image based on an input image containing image data and / or an image representation of an LED wall, comprising: c) controlling the appearance of the LED wall in response to the angle of view of the camera relative to the LED wall as described above; d) capturing at least one input image with the camera, the input image comprising image data of the LED wall; e) determining at least one dilated area in the input image to be digitally dilated; f) providing virtual content image data for at least the at least one extended area; g) generating a digital augmented output image by augmenting said input image with said virtual content image data in at least said at least one augmented area.

[0015] The present invention is based on the surprising discovery that the quality of digitally enhanced camera images depicting one or more LED walls can be significantly improved by taking into account the effect of the camera's angle of view relative to the LED wall. The method of the present invention enables accurate correction of brightness and color artifacts that occur when capturing images of an LED wall at different angles, which can cause noticeable artifacts during digital enhancement of the camera input image. Thus, the method of the present invention can significantly improve the quality of digitally enhanced camera images that include image data and / or image representations of an LED wall.

[0016] It should be noted that the objective of the present invention is solved by both of the methods described above, both of which solve the objective of the present invention by reducing the visual artifacts associated with the use of an LED wall.

[0017] The method for controlling the appearance of an LED wall as a function of the camera's angle of view relative to the LED wall allows the appearance of the LED wall to be controlled such that the visual appearance of the LED wall in an input image captured by the camera does not change when the camera's angle of view changes, thereby significantly improving the realism of the background of a scene provided by the LED wall.

[0018] When an input image containing a depiction of an LED wall is digitally augmented, the image quality can be improved by the method for creating a digitally augmented camera image described above. One problem with using an LED wall is the difficulty of ensuring a seamless transition between the area of ​​the input image where the LED wall is visible and the virtual content image data. Because the visual appearance of the LED wall (i.e., the brightness and / or color and / or color balance of the LED wall) varies depending on the camera's angle of view relative to the LED wall, significant distortions in brightness and / or color and / or color balance can occur between the area of ​​the input image depicting the LED wall and the virtual content image data. When the input image is augmented with the virtual content image data, abrupt transitions in brightness and / or color and / or color balance can be seen at the boundary between the input image area depicting the LED wall and the augmented area where the virtual content image data overlaps.

[0019] The present invention provides a solution to this problem by adjusting the brightness and / or color and / or color balance of the LED wall according to the camera's LED wall field of view before the input image is captured. This allows for simple and effective adjustment of the LED wall appearance, i.e., its brightness and / or color and / or color balance, to match the virtual content image data. This ensures a seamless transition between the input image area where the LED wall is captured and the virtual content image data. By adjusting the LED wall appearance according to the camera's LED wall field of view, transition errors between the focus area in the input image (i.e., the image area that will not be replaced or augmented with the virtual content image data) and the image area digitally augmented with the virtual content image data can be significantly reduced.

[0020] The present invention is based on a detailed understanding of the nature of artifacts that occur in an LED wall as a function of the angle of view. Brightness and / or color artifacts when capturing an image of an LED wall are caused by the physical configuration of the LED wall. Typically, each pixel of an LED wall is composed of an individual color LED, e.g., red, green, and blue LEDs. These color LEDs have a direction-dependent light emission pattern. The intensity of light emitted by the individual color LEDs of an LED wall is typically highest in a direction perpendicular to the surface of the LED wall and decreases as the angle between the surface normal of the LED wall and the viewer's line of sight increases.

[0021] The intensity of brightness and / or color artifacts may further depend on the relative placement of individual color LEDs within an LED wall pixel. In some walls, the color LEDs of individual LED wall pixels are arranged in a vertical column. When the angle of view in front of such an LED wall changes in the horizontal plane (i.e., when a camera is panned in front of the LED wall so that only its horizontal orientation relative to the LED wall changes), the brightness of the individual LED wall pixels changes primarily due to the direction-dependent emission patterns of the color LEDs. When the angle of view changes in the vertical plane (i.e., when a camera is tilted in front of the LED wall so that only its vertical orientation relative to the LED wall changes), a change in the color of the individual LED wall pixels may also be perceived because the color LEDs of the individual LED wall pixels partially obscure each other, changing the color composition of the individual pixels.

[0022] The occurrence of these artifacts is accounted for by adjusting the brightness and / or color and / or color balance of the color LEDs that make up the LED wall, or more precisely, the LED wall pixels of the LED wall, based on the camera's LED wall angle of view determined for each input image. In some embodiments, the LED wall angle of view may be determined as a single value for the entire LED wall. This may be preferable when the distance between the camera and the LED wall is sufficient so that the angle of view at which the camera observes different regions of the LED wall is substantially the same for all LED wall pixels.

[0023] In cases where the camera is relatively close to the LED wall such that the LED wall extends across most of the camera's field of view, the LED wall angle of view is preferably determined individually for multiple LED wall positions, preferably for each LED wall pixel. This approach can significantly improve image quality, as the camera's LED wall angle of view can vary significantly throughout the input image frame.

[0024] In the context of the present invention, the camera is preferably configured as a digital camera with an image sensor for capturing images in a digital memory, which allows image enhancement in real time. However, the present invention can also be implemented with an analog camera, where the captured images are digitized. Therefore, in the context of the present invention, the term "camera" refers to any camera system suitable for providing a digital input image that can be enhanced with digital image content.

[0025] In the context of the present invention, the term “LED wall appearance” or “LED wall appearance” refers to the visual appearance of an LED wall, which is primarily defined by the brightness and / or color and / or color balance of the LED wall, or more precisely, the brightness and / or color and / or color balance of individual LED wall pixels. The brightness and / or color and / or color balance of individual LED wall pixels is determined by the brightness of the monochromatic LEDs of each individual LED wall pixel. Therefore, adjusting the LED wall appearance (based on the camera's LED wall angle of view) may correspond to adjusting the brightness and / or color and / or color balance of the LED wall based on the determined LED wall angle of view, and / or adjusting the brightness of the color LEDs of the LED wall pixels of the LED wall based on the determined LED wall angle of view. Specific preferred embodiments of this adjustment will be described in more detail below. For simplicity, the “camera's LED wall angle of view” may also be referred to as the “LED wall angle of view” or simply the “angle of view.”

[0026] An LED wall in which the present invention can be used includes a plurality of LED wall pixels arranged in a planar configuration (e.g., arranged in a rectangular or hexagonal grid pattern). Preferably, the LED wall pixels include a plurality of (single) color LEDs, e.g., each LED wall pixel includes a red LED, a blue LED, and a green LED. By controlling the brightness of the individual color LEDs within an LED wall pixel, the LED wall pixel can be tuned to any color and brightness. Control of the individual LED wall pixels and the color LEDs within the individual LED wall pixels may be performed by an LED wall controller communicatively connected to the LED wall and sending appropriate control signals to the LED wall to control the appearance of the individual LED wall pixels.

[0027] In the context of the present invention, the appearance of the LED wall can be adjusted by adjusting color information of the image data sent to the LED wall for display. The term "color information" is understood to include one or more parameters that allow modification of the LED wall appearance. Color information can include one or more of the following parameters: luminance, e.g., color, which can be parameterized by the luminance values ​​of the individual color channels (e.g., red, green, and blue) of the individual LED wall pixels; color balance; and / or hue.

[0028] According to one embodiment, adjusting the LED wall appearance may be performed by determining an appropriate adjustment control signal (based on the determined LED wall angle of view) to be sent to an LED wall controller. In this case, the LED wall controller receives image data to be displayed on the LED wall and generates corresponding control signals for individual LED wall pixels corresponding to the image data, thereby causing the LED wall to display an image corresponding to the image data received by the LED wall controller. The control signal may be adjusted based on the adjustment control signal determined based on the camera's LED wall angle of view. The adjustment may include adjusting color information of the image data based on the determined LED wall angle of view. The adjusted image data is then sent to the LED wall for display.

[0029] To reduce the amount of data exchanged with the LED wall controller, according to another preferred embodiment, the adjustment of the LED wall appearance is performed by modifying image data to be displayed on the LED wall based on the determined LED wall angle of view. The modified image data, which is the image data adjusted based on the determined LED wall angle of view, is sent to the LED wall controller. The modified image data is displayed on the LED wall. In this embodiment, the adjustment of the LED wall appearance based on the camera's LED wall angle of view can be performed without the involvement of the LED wall controller. This allows the use of a conventional LED wall controller configured to receive the (modified) image data and control individual LED wall pixels to display the (modified) image data on the LED wall.

[0030] An input image captured by a (digital) camera is typically provided as a digital image made up of pixels, each with a finite, discrete numerical representation for the intensity or gray level for each pixel. These quantities are typically output as spatial coordinates, denoted x and y on the x-axis and y-axis, respectively. In the context of this disclosure, the 2D area spanned by the spatial coordinates representing the input image is also referred to as the image frame.

[0031] In the context of the present invention, the term "input image" refers to an image captured by a camera. The input image may be used as input for digital augmentation. The input image, which includes image data and / or an image representation of the LED wall, represents a camera image in which the LED wall is visible within the camera's field of view.

[0032] At least one augmented area is determined in the input image (as may be captured by the camera after the LED wall appearance has been adjusted based on the camera's angle of view relative to the LED wall). The at least one augmented area is essentially an area of ​​the input image containing image data to be replaced with virtual content image data. In the context of the present invention, these areas are referred to as "at least one" augmented area, which refers to at least one contiguously connected area in the input image, or "at least one" augmented area, which refers to any multiple augmented areas in the input image, regardless of whether these areas are contiguously connected. The use of the plural augmented area does not imply that the input image must contain multiple disjoint or separate augmented areas. The augmented areas may be contiguously connected to form a contiguous augmented area.

[0033] In addition to the augmented areas, the input image also includes areas that are not replaced or augmented with virtual content image data. These areas of the input image are called regions of interest. In the context of the present invention, augmented areas and regions of interest can be said to be two basic types of image content distinguished in the input image. The input image may include areas depicting interesting objects or scenes, such as an LED wall, optionally in front of which there are people and / or objects (in this case, both the people and objects and the image depiction of the LED wall may belong to the areas of interest). These areas in the input image form the areas of interest in the input image. In addition to these areas of interest, the input image further includes augmented areas. These areas are replaced with image information not included in the input image. In the case of an LED wall scene, the augmented areas typically include areas outside the LED wall where neither the LED wall nor the interesting people or objects are visible in the input image.

[0034] The input image is modified by extending the input image with virtual content image data, at least in the extension area. The virtual content image data is not particularly limited. In the case of an input image depicting an LED wall, the virtual content image data may be image data that complements the image data displayed on the LED wall, thereby virtually extending the image displayed on the LED wall beyond the boundaries of the LED wall.

[0035] By combining the virtual content image data (at least in the augmented area) with the image data contained in the region of interest, a digitally augmented output image is obtained. Different methods for combining the virtual content image data with other image information in the input image can be used and are known in the art. For example, after the augmented area is determined, the corresponding pixels in the input image can be set to transparent. The resulting modified input image can then be overlaid on the image containing the virtual content image data. To smooth the transition between the virtual content image data in the augmented area and the input image information in the region of interest, the virtual content image data can be provided so as to partially overlap the region of interest, and the overlapping area can be overlaid with a transparency gradient.

[0036] The methods according to the present invention are preferably computer-implemented. They can be performed on a dedicated computer to which input images taken by a camera are provided. Alternatively, they can be implemented on a microcomputer or microcontroller included in the camera or another device.

[0037] It should be noted that these methods are not strictly limited to the order of steps described above, and the steps may be partially performed simultaneously. Obvious limitations on the order of steps include that steps a) and b) must be performed before step c), and step d) must be performed after step c).

[0038] Providing the virtual content image data can be achieved by receiving pre-compiled virtual content image data from a data source and / or by creating or rendering the virtual content image data in real time. One preferred application scenario of the present invention is extending the image content shown on an LED wall beyond the boundaries of the LED wall visible in the input image. In this case, the image data for display on the LED wall and the virtual content image data may be provided from the same data source.

[0039] The camera's LED wall angle of view (hereinafter also referred to as the camera's angle of view) is the angle between the camera's optical axis and the reference direction or vector of the LED wall. As the reference vector of the LED wall, for example, the surface normal vector of the LED wall (which can be uniquely determined in the case of a flat LED wall, and can be defined as the surface normal vector of the tangent plane of the LED wall in the case of a curved LED wall) may be used.

[0040] Different methods may be used to determine the LED wall angle of view. According to a preferred method, the LED wall angle of view is determined by: providing a 3D model of the LED wall; establishing the position and attitude of the camera, preferably including establishing the position and orientation of the optical axis of the camera; and determining an LED wall angle of view based on the 3D model and the established position and orientation of the camera, and / or the established position of the camera and the established orientation of the optical axis.

[0041] This method allows for reliable and accurate determination of the LED wall field angle.

[0042] The 3D model of the LED wall preferably includes the (Cartesian) 3D coordinates of at least some of the compartment features of the LED wall. If the LED wall consists of a rectangular screen, for example, the 3D model may include the (Cartesian) coordinates of at least the corners of the LED wall.

[0043] Preferably, the camera position and orientation are determined or predicted before or during the determination of the field of view of the LED wall and before or during the capture of the input image including the LED wall. This allows the camera's viewpoint relative to the 3D model of the LED wall to be determined. A camera tracking system can be employed to determine the camera position and orientation. Suitable camera tracking systems include systems based on optical tracking methods such as odometry, GPS, or SLAM. Optical tracking systems suitable for determining the camera position and orientation are described, for example, in EP 1 211 520 A2.

[0044] The camera position indicates the absolute 3D position of the camera in space. The camera pose indicates the combination of the camera position and the camera orientation, which is preferably described by a vector parallel to the camera's optical axis and pointing in the camera's viewing direction. This vector is called the optical axis in the context of the present invention. The optical axis may be represented by a vector parallel to the optical axis and pointing in the camera's viewing direction, as described above, or by a non-directional 3D line.

[0045] According to a further preferred embodiment, adjusting the appearance of the LED wall includes adjusting color information of the LED wall, preferably the brightness and / or color and / or color balance of the LED wall, preferably by adjusting the brightness of individual color channels of the LED wall based on the determined LED wall view angle. The color channels preferably correspond to the colors of the individual color LEDs that make up the LED pixel. For example, adjusting the appearance of the LED wall may be achieved by individually adjusting the brightness of the red, blue, and green channels based on the determined LED wall view angle. This embodiment allows for improved adjustment of the appearance of the LED wall, since the view angle dependence of the brightness of different color channels may vary for different colors.

[0046] In this context, adjusting the appearance of the LED wall preferably includes adjusting the brightness of color LEDs of LED wall pixels of the LED wall according to the LED wall angle of view, where the LED wall angle of view is preferably determined individually for a plurality of LED pixels, preferably for each LED wall pixel. This approach can significantly improve the quality of the LED wall appearance correction, especially when the camera is positioned such that the LED wall spans a large portion of the camera's field of view. In such cases, the LED wall angle of view varies significantly for each individual LED wall pixel.

[0047] In these cases, it is preferable to determine the LED wall view angle for a plurality of LED wall pixels, preferably for each LED wall pixel, individually, and in this case, adjustment of the appearance of the LED wall based on the determined LED wall view angle is performed for a plurality of LED wall pixels, preferably for each LED wall pixel, individually.

[0048] As described above, the appearance of the LED wall can be adjusted by generating appropriate adjustment control signals that are sent to the LED wall controller along with image data for display on the LED wall. In this case, the LED wall controller generates modified control signals for individual LED wall pixels based on the image data, which are adjusted based on the adjustment control signals. According to another preferred embodiment, the appearance of the LED wall is adjusted by modifying or adjusting the image data based on the determined LED wall angle of view of the camera. The adjusted image data is then sent (as display data) to the LED wall controller for display on the LED wall. Also, the LED wall angle of view can be determined individually for multiple LED wall pixels (or for each LED wall pixel), and the modification or adjustment of the image data is performed by taking into account the different LED wall angles of view determined for different LED wall pixels.

[0049] According to a preferred embodiment, determining the LED wall field angles comprises determining a plurality of LED wall field angles for different reference orientations. More preferably, the step of adjusting the appearance of the LED wall based on the determined LED wall field angles is performed separately for some or each of the plurality of LED wall field angles for the different reference orientations.

[0050] As an example, when the color LEDs of individual LED wall pixels are arranged in a vertical column as discussed above, determining the LED wall field angle preferably includes determining the horizontal LED wall field angle (i.e., the LED wall field angle measured in a horizontal plane) and determining the vertical LED wall field angle (i.e., the LED wall field angle measured in a vertical plane).

[0051] In adjusting the LED wall appearance, the horizontal LED wall view angle can be used to correct only the brightness, and the vertical LED wall view angle can be used to correct only the color balance. If the color LEDs of individual LED wall pixels are arranged in a triangle, the determination of the LED wall view angle may take into account three reference directions and perform corrections for each direction separately.

[0052] When determining a plurality of LED wall view angles for different reference orientations, adjusting the LED wall appearance based on the plurality of view angles for different reference orientations may also be performed by determining individual view angles for different reference orientations for different LED wall pixels. Preferably, the LED wall view angles for different reference orientations are determined individually for each LED wall pixel.

[0053] In a preferred embodiment, the method includes creating a view-angle dependent calibration function for adjusting the LED wall appearance. The view-angle dependent calibration function is then used to adjust the LED wall appearance by adjusting color information based on the view-angle dependent calibration function. Preferably, a view-angle dependent calibration function is created separately for at least one color channel, preferably all color channels, of the LED wall.

[0054] Preferably, the step of creating the field-of-view-dependent calibration function includes sending reference image data to the LED wall for display. The reference image data may be, for example, a specific color (e.g., a color corresponding to one of the LED wall color channels, e.g., red, blue, or green) for uniform display across the entire LED wall. Alternatively, the reference image data may be configured such that only a limited area of ​​the LED wall, such as a limited area at the center of the LED wall, displays the reference image data, while the remainder of the LED wall remains dark. As interpreted above, when a camera is positioned close to the LED wall such that the LED wall extends across most of the camera's field of view, the camera's LED wall field of view may vary significantly across an input image frame. In such cases, the reference image data is preferably configured such that only a limited area of ​​the LED wall (as seen by the camera at a nearly constant LED wall field of view) displays the reference image data, while the remainder of the LED wall remains dark, because displaying the reference image data in other areas of the LED wall seen at different field of view only introduces noise into the field-of-view-dependent calibration function creation process. Preferably, a limited central area of ​​the LED wall is used in the field-of-view-dependent calibration function creation process. The reference image data (eg, a uniform color with uniform brightness) is then displayed only in this limited central area of ​​the LED wall, while the rest of the LED wall remains dark.

[0055] The step of creating the angle-of-view-dependent calibration function is based on a comparison of color information of the reference image data with color information contained in at least one (calibration) input image, i.e., an image captured by a camera, including the LED wall. Preferably, after creating the angle-of-view-dependent calibration function for one color channel, the method steps are repeated for another color channel until an angle-of-view-dependent calibration function has been created for each color channel of the LED wall. In the following, the term "(angle-of-view-dependent) calibration function" may refer to a (angle-of-view-dependent) calibration function of one color channel, which may be expressed as an angle-dependent vector, or to a (angle-of-view-dependent) calibration function of multiple color channels.

[0056] According to one embodiment, the reference image data is kept constant during the process of creating the angle-of-view-dependent calibration function, so that the LED wall displays the same color information throughout the entire process. Preferably, creating the angle-of-view-dependent calibration function includes capturing a reference input image of the LED wall at a predetermined reference angle of view, such as a 0° angle of view or an angle of view corresponding to a perpendicular orientation of the LED wall relative to the camera's optical axis. More preferably, creating the angle-of-view-dependent calibration function includes capturing multiple calibration input images of the LED wall at different angles of view.

[0057] Preferably, the method further includes a step of determining calibration function values ​​for different angles of view based on a comparison, at different angles of view, of color information of a portion of the calibration input image depicting the LED wall and color information of a portion of the reference input image depicting the LED wall. Preferably, creating the angle-of-view-dependent calibration function further includes a step of interpolating calibration values ​​based on the angle-of-view-dependent calibration values ​​determined in the comparing step.

[0058] According to another preferred embodiment, the reference image data (and thus the appearance of the LED wall) is not kept constant during the creation of the view-angle-dependent calibration function. This embodiment also includes the steps of transmitting the reference image data to the LED wall for display, capturing a reference input image at a predetermined reference LED wall view angle, and capturing a calibration input image of the LED wall at a different view angle. In contrast to the above method, for a given LED wall view angle, a comparison of the color information of the captured calibration input and the color information of the reference input image is used to modify the appearance of the LED wall. If the color information does not match (i.e., the difference in color information is not below a predetermined threshold), an adjustment control signal is determined to match the appearance of the LED wall (contained in the captured calibration input image) with the reference input image. The appearance of the LED wall is modified based on the determined adjustment control signal.

[0059] Preferably, a further calibration input image of the LED wall (with its appearance modified based on the determined adjustment control signal) is captured, and its color information is compared with that of the reference input image. If a mismatch is determined (i.e., the difference in color information is not less than a predetermined threshold), the adjustment control signal is modified so that the appearance of the LED wall (contained in the captured calibration input image) more closely matches the reference input image. This loop can be repeated until an appropriate adjustment control signal is iteratively determined to match the appearance of the LED wall with the reference input image.

[0060] Once a match between the reference input image and the captured calibration input image is found, a calibration function value is determined for a given LED wall field angle based on the correspondingly determined adjustment control signal. This iterative method alters the appearance of the LED wall based on the adjustment control signal, such that at a given LED wall field angle, the LED wall appearance captured by the camera matches the reference input image. This approach enables accurate and straightforward determination of the adjustment control signal on which the calibration function value is based, and such determination enables field-angle-dependent modification of the LED wall appearance so that the input image captured by the camera at a given field angle matches the input image acquired at the reference LED wall field angle. Thus, visual artifacts in the camera image depicting the LED wall can be effectively reduced because the LED wall appearance can be continuously adjusted so that the camera "sees" the LED wall as it appears at the reference LED wall field angle.

[0061] Preferably, in both of the above-described embodiments, the comparison of the color information of the calibration input image with the color information of the reference input image includes a comparison of a limited central portion of the LED wall in the input image. As noted above, when the camera is positioned relatively close to the LED wall, the LED wall angle of view varies throughout the entire area of ​​the input image, or in other words, the LED wall angle of view varies significantly for individual LED wall pixels. To enable reliable determination of the angle-of-view-dependent calibration function, the determination of the calibration function value preferably considers only limited areas of the reference and calibration input images depicting the LED wall, corresponding to a substantially constant LED wall angle of view in each image. Preferably, the reference and calibration input images are captured such that the relative position of the LED wall in the input image frame is substantially constant, preferably centered. For the comparison of the reference and calibration input images, a portion of the input image depicting the central portion of the LED wall is also preferably used to mitigate the impact of the camera lens vignetting effect on the process of creating the angle-of-view-dependent calibration function. Furthermore, as described above, in this context, the reference image data is preferably configured such that only a limited area of ​​the LED wall, preferably the center of the LED wall, displays the reference image data, while the remainder of the LED wall remains dark. A comparison of the color information in the calibration input image and the reference input image is then performed for an area of ​​the input image that displays a limited area of ​​the LED wall that displays the reference image data.

[0062] The view-angle-dependent calibration function may further be used to perform a step of adjusting the LED wall appearance (i.e., adjusting color information such as the brightness and / or color and / or color balance of the LED wall) based on the determined LED wall view angle. The view-angle-dependent calibration function may be a scalar function indicating the view-angle-dependent luminance of the LED wall. For example, the value of the calibration function may indicate the ratio between the luminance of the LED wall at a given LED wall view angle and the luminance of the LED wall at a reference view angle. More preferably, the value of the calibration function indicates a multiplication ratio or gain factor that, when applied to image data at the corresponding LED wall view angle, results in an appearance of the LED wall at the corresponding LED wall view angle that corresponds to the appearance of the LED wall at the reference view angle. The view-angle-dependent calibration function may also be a vector function that separately indicates the view-angle-dependent luminance of each of the different colors of the color LEDs. For example, the view-angle-dependent calibration function may be a three-component vector including scalar view-angle-dependent calibration functions for each of the red, green, and blue channels. The angle-dependent intensity of each of the different colors (e.g., red, green, and blue) of a color LED can be expressed as the ratio of the intensity of each color at a given LED wall field angle to the intensity of the corresponding color at a reference field angle.

[0063] The step of creating the field-angle-dependent calibration function of the present invention automatically takes into account the fact that the representation of the LED wall in the input image can be affected by the camera. The appearance of the LED wall in the input image depends not only on the LED wall field angle but also on the characteristics of the camera. For example, the sensitivity of the camera's image sensor may not be perfectly linear for each color channel. Furthermore, the camera may have a specific selected camera LUT, i.e., a conduit / code that converts color (RGB) input values ​​from the camera sensor into different color (RGB) output values ​​to adjust the look or mood of the input image. The color information of the LED wall representation in the input image is affected not only by the LED wall field angle but also by the camera LUT. By determining the field-angle-dependent calibration function by comparing a reference input image (which can be kept constant or changed based on the comparison) with a calibration input image, the camera-dependent modifications of the LED wall appearance in the input image are effectively canceled out, so that the camera-dependent modifications of the LED wall appearance in the input image are automatically taken into account in the calibration process.

[0064] When a 3D model of the LED wall is provided, it can be used not only to determine the camera's viewing angle of the LED wall but also to determine the extension area. Preferably, the extension area is determined by: providing a 3D model of the LED wall captured in the input image; establishing a camera position and pose; calculating an image position of the LED wall in the input image based on the 3D model of the LED wall and the established position and pose of the camera; determining an expansion area based on an image position of the LED wall in the input image.

[0065] Given a 3D model of the LED wall, areas in the input image that do not depict the LED wall (and thus represent augmented areas) can be determined with high reliability, accuracy, and speed. The position and pose of the camera at the time the input image was captured is established, and the position of the LED wall in the input image can be calculated based on the 3D model of the LED wall and the position and pose of the camera.

[0066] As detailed above, the 3D model of the LED wall preferably includes 3D (Cartesian) coordinates of at least some local features of the LED wall, such as the (Cartesian) coordinates of the corners of the LED wall. The position and pose of the camera may be determined by any suitable tracking means, as described above.

[0067] According to a further preferred embodiment, the providing of the virtual content image data is based on or takes into account a camera vignette model. Preferably, the providing of the virtual content image data comprises providing a camera vignette model, wherein the providing of the virtual content image data and / or the generating of the digital augmented output image is based on the camera vignette model.

[0068] Vignetting refers to a decrease (or increase) in brightness or saturation of an image towards the periphery that occurs in all lenses and lens systems used in cameras. The vignetting effect varies depending on the type of lens used and may also vary with various camera settings such as aperture. The image effect caused by vignetting, or the vignette effect, is characterized by a decrease (or increase) in brightness in areas of the image away from the center of the image and a darker (or lighter) border around the image.

[0069] Taking into account the camera's vignetting effect further improves the obtainable quality and realism of the digitally augmented output image. In an application scenario of the present invention, the input image includes an image representation of an LED wall that typically does not fill the entire image. As described above, determining the augmented area in such cases is relatively straightforward; any area of ​​the input image that does not depict either the LED wall or a foreground object, such as a person, can be determined to be the augmented area. The image displayed on the LED wall can then be augmented by overlaying corresponding virtual content image data onto the augmented area. In this way, the image displayed on the LED wall and captured in the input image is virtually stretched or augmented by the virtual content image data.

[0070] However, because the image content on the LED wall captured in the input image is subject to camera photography aberrations and errors, modifying the virtual content image data based on the camera's vignette model (thereby virtually subjecting the data to the same photography errors as the data in the input image) can significantly improve the quality of the digitally augmented camera image. In this scenario, the vignette model is preferably used to apply a vignette effect to the virtual content image data, such that the virtual content image data is subject to the same vignette effect as the rest of the content of the input image that was captured by the camera and that is not replaced by the virtual content image data.

[0071] A vignette model may be provided to allow for vignetting effects in the creation of digitally enhanced images. The vignette model is a function that, for a given 2D point coordinate in an image, outputs a reduction in brightness or saturation relative to the center of the image at that particular 2D point coordinate. The vignette model may be configured as a radial function indicating a reduction in brightness or saturation depending on the (radial) distance from the center of the image. Alternatively, the vignette model may be provided as a look-up table that provides a reduction in brightness or saturation for each pixel of the image frame. The reduction in brightness or saturation may be represented, for example, by a number between 0 and 1, where 0 indicates a complete reduction in brightness or saturation and 1 indicates no reduction in brightness or saturation (e.g., relative to the center of the image frame). Depending on the selected camera LUT, the vignette model may appear to change not only the brightness but also the color of the image, although brightness is often significantly reduced.

[0072] The vignette model preferably depends on the aperture size of the camera: the smaller the aperture of the camera, the less noticeable the vignetting effect, and therefore the vignette model preferably includes a set of sub-models for different aperture sizes of the camera.

[0073] Providing the virtual content image data based on the vignette model may include applying a vignette effect to the virtual content image data based on or derived from the vignette model, which may further improve the quality of the digitally augmented output image because camera photography aberrations and errors are taken into account when providing the virtual content image data.

[0074] In a further embodiment, the vignette model is applying a devignetting effect based on a vignetting model to the input image, preferably at least to an area outside the extended area of ​​the input image; The virtual content image data is used in the step of creating the virtual content image data by combining the virtual content image data with a devignetted input image, preferably with an area of ​​the input image that is outside the extended area and to which the devignetting effect has been applied.

[0075] In addition, in this embodiment, by devignetting image data in the input image, the abrupt transition between the input image data (vignetted by capturing it with a camera) and the virtual content image data is mitigated or eliminated, thereby improving the image quality of the digitally enhanced camera image.

[0076] According to a further embodiment, the vignette model is used to adjust the appearance of the LED wall. In this embodiment, the camera's vignetting effect is compensated for by adjusting the LED wall appearance based on the vignette model, thereby canceling the camera's vignetting effect in the input image. Because the vignette model accounts for brightness or saturation variations due to the camera's vignetting effect, the LED wall appearance may be adjusted based on the vignette model by modifying color information of the image data displayed on the LED wall so that the vignetting effect in the camera's input image is suppressed.

[0077] As described above, the value of the vignette model, which indicates the reduction in brightness and / or saturation due to the camera's vignetting effect, may be provided as a numerical value on a predetermined scale. For example, a numerical value of zero in the vignette model may indicate that the corresponding pixel is completely covered by the vignetting effect and always has a brightness of 0, i.e., is completely black. The numerical value may also be normalized. For example, a value of one in the vignette model may indicate that the corresponding pixel has not experienced any reduction in brightness and / or saturation due to the vignetting effect, or that it is the least bright pixel relative to the other pixels in the input image frame.

[0078] Using values ​​between 0 and 1 for the vignette model facilitates applicability of the vignette model. The vignette model can be used to apply a vignette effect to image data, particularly virtual content image data, for example, by multiplying the luminance and / or saturation values ​​of that pixel by the corresponding values ​​in the vignette model. More complex vignette models can also take into account the camera LUT when applying the vignette effect to image data to account not only for luminance reduction, but also for potential color shifts due to the camera LUT.

[0079] Conversely, image data, particularly input image data, can be devignetted by dividing the luminance and / or chroma values ​​of that pixel by the corresponding values ​​of the vignette model. More complex vignette models can also take the camera LUT into account when applying a vignetting effect to the image data to implement not only a reduction in luminance, but also potential color shifts due to the camera LUT.

[0080] According to a preferred embodiment, the vignette model includes a function that maps a vignette value, indicative of the reduction in brightness and / or saturation due to the camera's vignetting effect, to each pixel location in the camera's image frame. The mapping may be provided in a look-up table, in which each pixel in the camera's image frame is assigned a value indicative of the reduction in brightness and / or saturation. This allows for the provision of a conceptually simple and computationally easily accessible vignette model.

[0081] According to a further embodiment, the vignette model includes a function that maps a vignette value, which indicates a reduction in brightness and / or saturation due to the camera's vignetting effect, to a distance relative to the center of the camera's image frame. This vignette model requires less storage space than a pixel-based vignette model, while still providing a good representation of the camera's vignetting effect. The vignette value is provided as a numerical value on a predetermined scale, as in the previous embodiment, and may be further normalized. Preferably, the vignette value uses a value between 0 and 1.

[0082] When image data is modified using this type of vignette model, the distance between the pixel coordinate to be modified and the center of the image is calculated to obtain the corresponding vignette value. The geometric center of the image frame may be used as the center of the image. If the image frame has a resolution of W × H pixels in the x and y directions, the center of the image may be determined as (W / 2; H / 2). Alternatively, to account for asymmetric or off-axis alignment of the camera's lens system, the center of the image may be offset relative to the geometric center of the image frame. In this case, the center of the camera's image frame preferably includes an offset value indicating the offset between the geometric center of the camera's image frame and the center of the vignette model where the minimum or maximum vignette value occurs. The offset value preferably comprises a 2D vector indicating the offset in the x and y directions in the image frame.

[0083] In a further preferred embodiment, the virtual content image data is subjected to a lens distortion effect corresponding to the lens distortion imparted by the camera before generating the digitally augmented output image, thereby further improving the quality of the digitally augmented output image.

[0084] The object of the present invention is to provide a control system for controlling the appearance of an LED wall depending on the angle of view of a camera relative to the LED wall, preferably by means of the above-mentioned method for controlling the appearance of an LED wall depending on the angle of view of a camera relative to the LED wall, comprising: Receives image data to display on the LED wall, Determine the camera's angle of view relative to the LED wall, a processing unit configured to generate a display signal used by the LED wall controller to adjust the appearance of the LED wall based on the image data and the determined LED wall field angle; The problem is further solved by a control system comprising: a communication unit communicatively coupling the digital image processing system to an LED wall controller for controlling the LED wall; and configured to send a display signal to the LED wall controller for adjusting the appearance of the LED wall.

[0085] The object of the present invention is further solved by a digital image processing system for creating a digital augmented camera image based on an input image comprising image data of an LED wall, preferably by the above-mentioned method for creating a digital augmented camera image based on an input image comprising image data and / or an image representation of an LED wall.

[0086] A digital image processing system includes the above control system, wherein the processing unit further includes: Receives an input image from a camera; determining an area in the input image to be digitally expanded; providing virtual content image data for at least the extended area; The input image is configured to generate a digitally augmented output image by augmenting the input image with virtual content image data in at least the augmented area.

[0087] The technical advantages achievable by the control system and digital image processing system of the present invention correspond to the technical advantages achievable by the method for controlling the appearance of an LED wall and the method for creating digital augmented camera images described above. Aspects, features, and advantages described in the context of the method of the present invention are also applicable to the system of the present invention, and vice versa.

[0088] In particular, any task that a processing or communication unit of a system of the invention is configured to perform may be performed as a method step in a method of the invention, and vice versa. Preferably, a method of the invention is implemented using a corresponding system of the invention as described above.

[0089] The processing unit and the communication unit may be implemented as separate physical entities and / or as software modules contained and executed on a common computing unit.

[0090] To determine the camera's LED wall field of view and / or receive input images from the camera, the control system and / or digital image processing system may be communicatively coupled to the camera by a wired or wireless connection or may be implemented as a digital or software component integrated into the camera.

[0091] The digital processing system according to the invention comprises a control system for controlling the appearance of the LED wall according to the invention, and therefore the features of said control system are also applicable to said digital processing system. In the following preferred improvements of the system are described which are applicable to both the control system and the digital processing system.

[0092] According to a preferred embodiment, the (control and / or digital processing) system comprises a memory unit for storing data. Preferably, the memory unit is configured to store a 3D model of the LED wall. The 3D model is provided from the memory unit to the processing unit for determining the LED wall field angle and / or determining the extended area in the input image, as described above with respect to the method of the present invention. The memory unit may further be configured to store a field angle-dependent calibration function, as described above with respect to the method of the present invention, which function is provided to the processing unit for adjusting the LED wall appearance by performing the method steps of the present invention as described above. According to a preferred embodiment, the control system is configured to create the field angle-dependent calibration function according to the method described above.

[0093] The adjustment of the LED wall appearance described in connection with the method of the present invention is implemented in the system of the present invention by generating a corresponding display signal. The display signal may include image data for display on the LED wall and / or an adjustment (control) signal for adjusting the appearance of the LED wall. As described above in the context of the method of the present invention, the control system is preferably configured to receive image data for display on the LED wall, generate adjustment control signals based on the determined LED wall field of view, and transmit the adjustment control signals together with the image data to the LED wall controller. In this case, the LED wall controller generates modified control signals for individual LED wall pixels based on the image data, and the modified control signals are adjusted based on the adjustment control signals. According to another preferred embodiment, the control system is configured to adjust the appearance of the LED wall by modifying or adjusting the image data based on the determined LED wall field of view of the camera. The adjusted image data is then transmitted to the LED wall controller as display data for display on the LED wall. According to a further preferred embodiment, the storage unit is configured to store a vignette model of the camera and provide the vignette model to the processing unit. This allows the vignette model to be used in providing virtual content image data and / or generating a digitally augmented output image, as described above in the context of the method of the present invention. The method steps using the vignette model may be performed by a correspondingly configured processing unit.

[0094] The object of the present invention is further solved by a computer-readable medium comprising instructions which, when executed by at least one processor, cause the at least one processor to perform a computer-implemented method including the steps according to the above-described inventive method. The technical advantages achieved by the inventive method correspond to the technical advantages achieved by the above-described computer-implemented method and the corresponding computer-readable medium. Aspects, features, and advantages described in the context of the inventive method are also applicable to the inventive computer-implemented method and computer-readable medium. [Brief explanation of the drawings]

[0095] The above and further features and advantages of the present invention will become more readily apparent from the following detailed description of preferred embodiments of the invention, taken in conjunction with the accompanying drawings. [Figure 1] 1 is a schematic diagram of a camera with a control system or digital image processing system for capturing a scene with an LED wall according to one embodiment of the present invention; [Figure 2] FIG. 2 is a schematic diagram of an input image including a region of interest and an augmented area, captured using the settings of FIG. 1. [Figure 3] FIG. 1 is a schematic diagram of a camera with a control system or digital image processing system used to adjust the appearance of an LED wall depending on the camera's angle of view of the LED wall, according to one embodiment of the present invention. [Figure 4a-b] FIG. 1 is a schematic diagram of an LED wall pixel having multiple color LEDs. [Figure 5a-c] FIG. 1 is a schematic diagram of the angle of view for an LED wall. [Figure 6] 4b shows exemplary calibration function profiles depending on the vertical LED wall field angle for the individual color channels of an LED wall having LED wall pixels configured according to FIG. 4a. [Figure 7] 4b shows exemplary calibration function profiles depending on horizontal LED wall field angle for color channels of an LED wall having LED wall pixels configured according to FIG. 4a. [Figure 8] FIG. 1 is a schematic illustration of a vignetting effect in an input image frame. [Figure 9] 1 illustrates the radial coordinates used in the vignette model according to a preferred embodiment of the present invention. [Figure 10] 1 shows a flow diagram illustrating the creation of an angle-of-view dependent calibration function according to a preferred embodiment of the present invention. [Figure 11] 1 shows a flow diagram illustrating the creation of a field-angle dependent calibration function according to a further preferred embodiment of the invention. DETAILED DESCRIPTION OF THE INVENTION

[0096] Figure 1 shows a schematic diagram of a scenario in which the method and system according to the invention can be used. Figure 1 shows a system 3 which can be configured as a control system and / or a digital image processing system according to an embodiment of the invention. For simplicity, when features are described below, the unit 3 will be referred to as system 3 applicable to the control system and digital image processing system of the invention.

[0097] The system 3 comprises a processing unit 31, which will be described in detail below, and a storage unit 32. The system 3 is connected to a camera 2, which may be constituted by a separate unit or may be included in the electronics of the camera 2. The camera 2 is provided to capture a scene including an LED wall 11. The LED wall 11 constitutes an object of interest that is included in the image captured by the camera 2. A further object of interest, such as a person 12, may be located in front of the LED wall 11.

[0098] The camera 2 may be a photo camera for capturing still images, or more preferably a cinema camera for capturing a cinema sequence consisting of a plurality of sequential images. The area captured by the camera 2 is indicated by a dotted line in Figure 1. The camera 2 comprises a tracking system 21 configured to establish the position and / or attitude of the camera 2.

[0099] The system 3 further comprises a communication unit 33. The communication unit 33 may consist of a terminal for connecting a communication line such as a network cable, or a wireless interface. The communication unit 33 is configured to establish a communication link from the system 3 to the LED wall controller 15, as indicated by the dashed line. The LED wall controller 15 is connected to the LED wall 11 and is configured to control the visual appearance of the LED wall 15 by controlling the brightness and color of individual LED wall pixels.

[0100] To control the (visual) appearance of the LED wall 11, the system 3 is configured to create display signals that are sent to the LED wall controller 15. The display signals include image data that may be provided as pixel-based color information to be displayed by individual LED wall pixels of the LED wall 11.

[0101] To control the visual appearance of the LED wall 11 based on the angle of view of the camera 2, the system 3 is configured to send an adjustment control signal together with the image data to the LED wall controller 15, as discussed below. The LED wall controller 15 then generates a control signal for controlling the appearance of the LED wall 11 based on the image data and the adjustment control signal. In this case, the display data sent to the LED wall controller 15 by the communication unit 33 includes the image data and the adjustment control signal. Alternatively, the system 3 (particularly the processing unit 31) is configured to modify or adjust the image data displayed on the LED wall 11, for example, based on the determined LED wall angle of view of the camera 2. In this case, the display data sent to the LED wall controller 15 by the communication unit 33 includes the adjusted image data. In this case, the LED wall controller 15 may control the appearance of the LED wall 11 using the adjusted image data without performing any additional corrections. This reduces the amount of data that needs to be sent to the LED wall controller 15.

[0102] The scenario shown in Figure 1 may occur when filming a movie scene in which a person 12 acts in front of an LED wall 11 that displays the background of the scene, or may occur in a television production studio in which a person 12 is an anchorman presenting the news or weather forecast in front of an LED wall 11 that displays additional information such as a weather map or a virtual studio background.

[0103] FIG. 2 is a schematic diagram of an image captured by camera 2 in the setup shown in FIG. 1. The area captured by camera 2, indicated by the dotted line in FIG. 1, is included in the image frame of camera 2 to obtain the input image shown schematically in FIG. 2. The input image is preferably provided as a pixel-based digital image. In the input image, an LED wall 11 and a person 12 in front of it are visible. Both the LED wall 11 and the person 12 constitute objects of interest. With possible digital processing of the input image, these objects of interest are maintained, at least to a large extent, in the input image. The entire input image area covered by the objects of interest may be referred to as region of interest 1, and is delimited by a dash-dotted line in FIG. 1.

[0104] 2 illustrates a scenario in which part of person 12 (his feet) is not included in the area of ​​the input image bounded by LED wall 11. This part may be omitted from region of interest 1 for simplicity, or may be automatically determined, for example by a suitable AI algorithm, and included in region of interest 1, as shown in FIG.

[0105] The input image of FIG. 2 further includes areas that do not depict objects of interest, such as the LED wall 11 or the actor 12, i.e., areas outside the region of interest 1. These cross-hatched areas in FIG. 2 may be designated as augmented areas 4. In the case of digital processing of the input image, the augmented areas 4 are augmented with virtual image content that is not visible in the input image of FIG. 2. For clarity, the entire augmented area 4 in the input image is referred to as augmented area 4.

[0106] In the case of a movie scene, the extended area 4 may be extended with scenery images that complement the scenery displayed on the LED wall 11. In the case of a television production studio, it may be desirable to display virtual studio content in the extended area 4.

[0107] The visual appearance of the LED wall 11 in the input image, shown schematically in FIG. 2, may change as the angle of view of the camera 2 relative to the LED wall 11 changes. When the camera 2 is oriented to look directly at the LED wall 11, the perceived brightness of the LED wall 11 may be higher than when the camera 2 views the LED wall at an oblique angle. As the angle of view of the camera 2 changes, the brightness, color, and / or color balance of the LED wall 11 may change. Because the LED wall 11 needs to create the impression of a realistic background whose appearance does not change with changing angle of view, it is disadvantageous if the LED wall appearance changes with the camera angle of view. The present invention provides a viable approach to address this problem.

[0108] FIG. 3 is a schematic diagram of a camera 2 connected to a system 3 for adjusting the appearance of an LED wall 11 depending on the camera's 2 LED wall angle of view.

[0109] Similar to the embodiment of Figure 1, camera 2 is provided with a tracking system 21 for establishing the position and pose of camera 2. Camera 2 is pointed towards LED wall 11 to capture input images including LED wall 11. The pose of camera 2, i.e., its orientation, defines its viewing direction. The camera's viewing direction lies on the camera's optical axis, shown by the dashed-dotted line and designated by reference symbol o. The orientation of LED wall 11 is characterized by a normal vector n that is perpendicular to the surface of LED wall 11.

[0110] The LED wall 11 includes a plurality of LED wall pixels 13. A typical configuration of an individual LED wall pixel 13 is shown schematically in FIGS. 4a and 4b. The LED wall pixel 13 of FIG. 4a includes three color LEDs 14 designated red (r), green (g), and blue (b). The LED wall pixel 13 of FIG. 4a is essentially rectangular and / or square, with the individual color LEDs 14 arranged in a vertical column. The LED wall pixel 13 of FIG. 4b also includes three color LEDs 14 labeled red (r), green (g), and blue (b). However, the LED wall pixel 13 of FIG. 4b is essentially circular, with the color LEDs 14 arranged in a triangular configuration.

[0111] The LED wall 11 of Figure 3 may be composed of LED wall pixels 13 according to the configuration of Figure 4a or 4b. When viewing the LED wall 11 at different angles of view, the (relative) intensities of the individual color LEDs 14 of the LED wall pixels 13 may change, causing angle-of-view-dependent variations in the visual appearance of the LED wall 11. The variations in appearance are caused by variations in the brightness and / or color and / or color balance of the LED wall 11 with changing angle of view, and are manifested as corresponding variations in brightness and / or color and / or color balance in the input image of camera 2 depicting the LED wall 11 as the angle of view of camera 2 relative to the LED wall 11 changes.

[0112] In order to suppress the above-mentioned variations in the visual appearance of the LED wall 11 with changing angle of view, the present invention proposes adjusting the appearance of the LED wall 11 based on the determined LED wall angle of view. To perform this adjustment, the system 3 is configured to generate a control signal based on the determined LED wall angle of view of the camera 2, and the control signal is sent to the LED wall controller 15. Based on the control signal, the LED wall controller 15 adjusts the appearance of the LED wall 11 by adjusting the brightness and / or color and / or color balance of the LED wall 11 based on the angle of view of the camera 2 relative to the LED wall 11.

[0113] In this embodiment, the angle of view of the camera 2 with respect to the LED wall 11 is defined as the angle between the optical axis vector o of the camera and the normal vector n of the LED wall 11, and is called the LED wall angle of view or angle of view α.

[0114] To enable the determination of the (LED wall) angle of view α for each input image, a tracking system 21 of the camera 2 is used. The tracking system 21 is configured to establish the position of the camera 2 and the orientation of the camera 2, which is defined by the orientation of the optical axis vector o. A 3D model of the LED wall 11 is stored in a storage unit 32 of the system 3. The processing unit 31 is configured to receive information about the position and orientation of the camera 2 (preferably determined by the tracking system 21) and the 3D model of the LED wall 11, and to determine the (LED wall) angle of view α based on the position and orientation of the camera 2 and the 3D model of the LED wall 11. Alternatively, the storage unit 32 stores a normal vector n of the LED wall 11. In this case, the processing unit 31 is configured to receive information about the position and orientation of the camera 2 and the normal vector n, and to determine the LED wall angle of view α based on the position and orientation of the camera 2 and the normal vector n.

[0115] It should be noted that the above disclosure is interpreted as determining one LED wall angle of view α based on the position and orientation of camera 2 and normal vector n. Determining one LED wall angle of view α for the entire LED wall 11 may be sufficient when camera 2 is located quite far away from LED wall 11 so that the LED wall 11 essentially appears at one angle of view α. When camera 2 is positioned relatively close to LED wall 11, the LED wall angle of view α varies significantly across the camera's field of view and across the corresponding input image frames. In these cases, it is preferable that the LED wall angle of view α be determined individually for multiple positions of the LED wall 11, preferably for each LED wall pixel 13. When referring to angle of view α for brevity and clarity in the following description, it is understood that multiple angles of view α can be determined at a given position and orientation of camera 2. Preferably, the angle of view α is determined individually for each LED wall pixel 13, and adjustment of the appearance of the LED wall 11 based on the LED wall angle of view α is performed individually for each LED wall pixel 13.

[0116] Based on the LED wall angle of view α, the processing unit 31 creates a display signal to be used by the LED wall controller 15. The display signal may be transmitted to the LED controller 15 via the communication unit 33 of the system 3. The display signal includes image data to be displayed on the LED wall 11 as well as an adjustment control signal, and the LED wall controller 15 adjusts the image data based on the adjustment control signal to adjust the brightness and / or color and / or color balance of the LED wall 11 according to the LED wall angle of view α of the camera 2. Alternatively, the processing unit 31 modifies the image data taking into account the LED wall angle of view α of the camera 2. The modified image data is then transmitted to the LED wall controller 15 for display on the LED wall 11.

[0117] The step of generating the adjusted control signal and / or modified image data may be based on a calibration function that depends on the field angle α. The calibration function preferably includes a numerical value that indicates the ratio between the brightness and / or intensity of the LED wall pixels 13 or color LEDs 14 observed at a reference field angle (e.g., 0°) and the field angle α. That is, the calibration function may be set to a value of 1 for a field angle value of 0° and a value different from 1 (typically <1) for field angle values ​​different from 0°. Alternatively, the calibration function may include a gain factor. For a field angle value of 0°, the gain factor may be set to 1, and for field angles different from 0°, a gain factor different from 1 (typically >1) may be used.

[0118] The calibration function may be a scalar function indicating a luminance ratio depending on the angle of view α. Alternatively or additionally, the calibration function may indicate a vector of intensity ratios for different color channels of the LED wall 11, i.e., for each type r, g, b of color LEDs 14. When the value of the calibration function is provided as a gain factor, the gain factor may be based on the inverse of the intensity ratio.

[0119] The calibration function may be provided as a look-up table containing pairs of values ​​between the angle of view α and the corresponding calibration function value, or alternatively, the calibration function may be provided as an analytical function, such as a polynomial function.

[0120] To determine the calibration function, reference image data is sent to the LED wall controller 15 for display on the LED wall 11. A reference input image is captured by a camera at a 0° angle of view. The brightness and / or color and / or color balance and / or intensity of individual color channels (e.g., r, g, and b) are determined for an area of ​​the reference input image depicting the LED wall 11. Preferably, the area of ​​the reference input image is selected to be small enough to accurately represent discrete values ​​of the LED wall view angle α. Further calibration input images are then captured at different angles of view, and the brightness and / or color and / or color balance and / or intensity of individual color channels (e.g., r, g, and b) are determined for an area of ​​the input image depicting the LED wall 11. Preferably, the area of ​​the calibration input image is also selected to be small enough to correspond to the area selected in the reference input image to accurately represent discrete values ​​of the LED wall view angle α. Based on the determined values, values ​​of the calibration function at different angles of view can be obtained. To achieve a calibration function with sufficient granularity, it is necessary to capture multiple input images at different angles of view or to interpolate calibration function values ​​obtained at different angles of view. During the determination of the calibration function, the appearance of the LED wall may be held constant or may be systematically adjusted to create the calibration function, both concepts being described in more detail below with reference to Figures 10 and 11.

[0121] The field angle α may be determined as the 3D angle between the camera orientation vector o and the normal vector n. The performance of the calibration function-based image correction method can be significantly improved if the field angle is determined along a reference plane that takes into account the actual configuration of the LED wall pixels 13. The calibration function may be set differently for the field angle at different reference planes.

[0122] For example, if the LED wall pixel 13 has an essentially rectangular configuration as shown in Figure 4a, then there are two main reference directions of interest: the vertical direction along which the color LEDs 14 of the individual LED wall pixels 13 are aligned, and the horizontal direction along which the individual LEDs extend. These reference directions are shown by dotted lines in Figure 4a. The corresponding reference directions for the circular LED wall pixel configuration of Figure 4b are correspondingly shown in Figure 4b.

[0123] When dealing with an LED wall including rectangular LED wall pixels 13, as shown in Figure 4a, the change in LED wall appearance as the angle of view varies is qualitatively different depending on whether the angle of view varies horizontally or vertically. Variations in the angle of view in the horizontal plane result in a substantially equal variation in brightness for all three color LEDs 14 in each LED wall pixel 13. Thus, when varying the angle of view in the horizontal plane, the brightness of the LED wall 11 will change, but there should be no noticeable color shift if the light distribution of each individual LED pixel 13 remains substantially the same.

[0124] Conversely, when changing the viewing angle in the vertical plane, the color LEDs 14 partially obscure each other, resulting in different intensity variations of the individual LED colors r, g, and b. Therefore, even if the light distribution of the individual LED wall pixels 13 is substantially the same in this case, their angle-dependent light distributions are expected to be different due to the partial shading of the individual pixels. Therefore, color shifts will also become more noticeable and / or the color balance will change much more than in the horizontal case.

[0125] This effect can be taken into account by providing a calibration function that includes multiple calibration functions for different color channels and also distinguishes between different reference directions or orientations. This is shown schematically in Figures 5a-c. In other words, the calibration function does not depend on the scalar value of the (3D) LED wall view angle α, but is preferably a function of multiple view angles at different reference orientations.

[0126] Figures 5a-c show a schematic representation of an LED wall 11 with a normal vector n. For clarity, the camera 2 is not shown in Figures 5a-c, and only the camera's optical axis vector o is shown, which defines the line of sight (orientation) of the camera 2. The LED wall 11 shown in Figures 5a-c includes a rectangular configuration of LED wall pixels 13, as shown in Figure 4a.

[0127] The angle of view may be determined as the three-dimensional angle α between the optical axis vector o of the camera and the normal vector n of the LED wall 11, as shown in FIG. 5a.

[0128] In order to better consider the effect of variations in the field of view on the LED wall appearance, it is preferable to set the calibration function not based on the (absolute or 3D) field of view α shown in Figure 5a, but rather by considering the vertical and horizontal reference directions corresponding to the configuration of the LED wall pixels 13.

[0129] As interpreted above with reference to Figure 4a, for LED wall pixels having a rectangular configuration, the vertical and horizontal directions are preferred reference directions for defining the calibration function, since the appearance variations of these LED wall pixels having variable field angles vary significantly depending on these reference directions. Therefore, it is preferable to set the calibration function to depend on the field angle in the corresponding reference plane.

[0130] This is shown in Figures 5b and 5c. In Figure 5b, the optical axis vector o is projected onto the horizontal plane shown by the dotted line. The angle between the normal vector n of the LED wall 11 and the projection of the optical axis vector o onto the horizontal plane is the horizontal angle of view α h The calibration function is determined as follows: h The different parameters of the calibration function are set to depend on the horizontal angle of view α h Preferably, the luminance of the individual color channels may be made dependent on the horizontal angle of view α h It is made to depend on.

[0131] In Fig. 5c, the optical axis vector o is projected onto the vertical plane indicated by the dotted line. The angle between the normal vector n of the LED wall 11 and the projection of the optical axis vector o onto the vertical plane is the vertical angle of view α v The calibration function is determined as follows: v The different parameters are set to depend on the vertical angle of view α v Preferably, the brightness of the individual color channels, i.e., the brightness of all color LEDs 14 of the same color, can be adjusted based on a calibration function depending on the vertical angle of view α v is adjusted accordingly.

[0132] Angle of view α h and α vis the horizontal angle of view α h The left and right gaze directions and the vertical angle of view α v Preferably, the angle is determined symbolically rather than as an absolute value to distinguish between up and down gaze directions. The calibration function is preferably asymmetric with respect to positive and negative values ​​of the field of view. This further improves the accuracy of the calibration function and the quality enhancement obtained by correcting the input image based on the field of view.

[0133] FIG. 6 shows the vertical LED wall view angle α for the individual color channels of an LED wall 11 having LED wall pixels 13 configured according to FIG. 4a. v 6 shows an example of a calibration function profile that depends on . For ease of reference, the main configuration of the LED wall pixel 13 is shown schematically in the upper left corner of FIG. 6, with the vertical reference axis indicated by a dotted line.

[0134] Vertical LED wall viewing angle α v The profile of the calibration function of varies with different color channels. Therefore, the vertical LED wall viewing angle α v The calibration function for can be expressed as a three-component vector, one component for each of the color channels r, g, and b.

[0135] In the graph of Figure 6, α v A positive value indicates the angle of view of the camera 6 looking down on the LED wall 11, and a negative value indicates the position and orientation of the camera 2 looking up on the LED wall 11.

[0136] The calibration functions for the individual color channels are labeled r, g, and b for red, green, and blue, respectively, and are shown by dotted lines for the r channel, dash-dotted lines for the g channel, and dashed lines for the b channel.

[0137] For the green channel, the calibration function is essentially symmetric because: vis changed to a positive value (i.e., when the LED wall 11 is viewed from a high position), the brightness of the green LED in each LED pixel 13 is reduced not only due to the angle-dependent emission pattern of the green LED, but also due to the fact that the red LED located above the green LED in each LED pixel 13 partially obscures the green LED and reduces their brightness.

[0138] Vertical LED wall viewing angle α v is changed to a negative value (i.e., when looking up at the LED wall 11 from a low position), the brightness of the green LED in each LED pixel 13 is also reduced not only due to the angle-dependent emission pattern of the green LED, but also due to the fact that the blue LED located below the green LED in each LED pixel 13 partially obscures the green LED and reduces their brightness.

[0139] The placement of the red and blue LEDs is vertically symmetric with respect to the position of the green LED, so that the partial shading of the green LED by other colored LEDs in the same LED pixel is vertically symmetric, resulting in a symmetric calibration function for the green channel.

[0140] The calibration function for the red channel is α v decreases faster for negative values ​​than for positive values. The reason is that the vertical LED wall viewing angle α v This is because when the vertical LED wall angle α is changed to a negative value, the red LED in each LED wall pixel 13 partially obscures the green LED of the same LED wall pixel 13 that is located close to the red LED. v When changing to a positive value, the red LED is partially obscured only by the blue LED of the upper LED wall pixel 13, which is positioned further away than the colored LEDs in the same LED wall pixel. v As the partial obscuration of the red LED becomes less pronounced as λ increases, the decrease in brightness of the red channel becomes less pronounced, and therefore the decrease in the calibration function for the red channel on the positive side of the calibration function becomes less pronounced.

[0141] The calibration function for the blue channel is essentially the inverse of the red channel. That is, the calibration function for the blue channel is α v decreases faster for positive values ​​than for negative values. Vertical LED wall viewing angle α v When the vertical LED wall view angle α is changed to a positive value, the blue LED in each LED wall pixel 13 is partially obscured by the green LED of the same LED wall pixel 13 that is located close to the blue LED. v When changing to a negative value, the blue LED is only partially obscured by the red LED of the lower LED wall pixel 13, which is positioned further away than the colored LEDs in the same LED wall pixel. Thus, a negative vertical LED wall view angle α v As λ increases, the partial obscuration of the blue LED becomes less pronounced, the decrease in brightness of the blue channel becomes less pronounced, and therefore the decrease in the calibration function for the blue channel on the negative side of the calibration function becomes less pronounced.

[0142] FIG. 7 shows the horizontal LED wall view angle α for the individual color channels of an LED wall 11 having LED wall pixels 13 configured according to FIG. 4a. h For ease of reference, the main configuration of the LED wall pixel 13 is shown schematically in the upper left corner of Figure 7, with the horizontal reference axis indicated by a dotted line.

[0143] As can be seen from the figure, the calibration function is essentially symmetric for all three color channels and essentially identical for all three color channels. Clearly, when the LED wall field angle is varied in the horizontal plane, the brightness variation of the individual color LEDs in each LED wall pixel is mainly affected by the angle-dependent emission patterns of the individual LEDs. The shading by adjacent LEDs is affected only by LEDs from horizontally adjacent LED wall pixels that are located at a distance greater than the distance between color LEDs in the same LED wall pixel. Therefore, as the horizontal LED wall field angle α h The decrease in the calibration function with increasing magnitude is moderate.

[0144] It should be noted here that the calibration functions shown in FIGS. 6 and 7 merely constitute examples of suitable calibration functions in the context of the present invention. The calibration function values ​​in FIGS. 6 and 7 essentially provide the quotient of the luminance of an individual color channel perceived at a given LED wall view angle and the luminance perceived at an LED wall view angle of 0°. Because luminance typically decreases with increasing deviation of the LED wall view angle from 0°, the calibration function values ​​in FIGS. 6 and 7 will range from 0 to 1. In another embodiment, the calibration function values ​​consist of gain values ​​that indicate what gain needs to be applied to each color channel at a given LED wall view angle to match the color at a reference wall view angle (e.g., 0°). In this embodiment, the calibration function values ​​effectively represent the inverse of the calibration function values ​​shown in FIGS. 6 and 7.

[0145] The calibration function values ​​for the individual color channels and / or individual reference orientations can be used by the processing unit 31 to create a display signal for transmission to the LED wall controller 15, as described above. By way of example, the processing unit 31 can calculate the LED wall field angle (i.e., the absolute 3D wall field angle α, or different LED wall field angles for different reference orientations, e.g., α h and α v), preferably individually for a plurality of different LED wall pixels 13, and more preferably for each LED wall pixel 13. The processing unit 31 is then configured to determine a corresponding value of the calibration function for each color channel. In some embodiments, the processing unit 31 is configured to calculate the inverse of the calibration function value for each color channel and use these inverse values ​​as gain coefficients for the individual color channels. In other embodiments, as described above, the calibration function values ​​comprise gain values, so that they can be used directly as gain values. The gain coefficients may then be transmitted as adjustment control signals along with the image data to the LED wall controller 15, thereby enabling the LED wall controller 15 to modify the image data based on the adjustment control signals to obtain modified image data for display on the LED wall 11. Alternatively, the processing unit 31 is configured to modify the image data based on the calibration function values, preferably by multiplying the luminance values ​​of the individual color channels of each LED wall pixel 13 in the image data by corresponding angle-of-view-dependent gain coefficients to obtain modified image data. This modified image data is then transmitted to the LED wall controller 15 as display data, so that the LED wall controller 15 does not need to further adjust the display data.

[0146] By adjusting the LED wall appearance according to the LED wall angle of view, changes in the appearance of the LED wall 11 in the input image of the camera 2 due to variations in the LED wall angle of view are minimized or eliminated. The system 3 may further be configured as an image processing system 3 configured to digitally augment an input image of the LED wall 11 by determining an augmentation area 4 in the input image, providing virtual content image data, and generating a digitally augmented output image by augmenting the input image in at least the augmentation area 4 with the virtual content image data.

[0147] 1 and 2, the storage unit 32 contains a 3D model of the LED wall 11. When capturing an input image, the corresponding position and orientation of the camera 2 is established by the tracking system 21. This allows the image position of the LED wall 11 in the input image shown in FIG. 2 to be calculated based on the 3D model of the LED wall 11 and the established position and orientation of the camera 2. The extension area 4 can then be determined based on the calculated image position of the LED wall 11 in the input image.

[0148] If the system 3 is configured as the image processing system 3 described above, it may be preferable to further improve the quality of the digitally augmented image by taking into account the vignetting effect of the camera 2 when providing the virtual content image data and / or generating the digitally augmented output image. To achieve this, a vignette model of the camera 2 may be stored in the storage unit 32 for providing to the processing unit 31 when digitally augmenting the camera input image.

[0149] Therefore, the vignetting effect of camera 2 is taken into account in the context of the present invention by proposing a vignette model that models the vignetting effect of camera 2. This allows correcting the vignetting effect in the (input) image captured by camera 2 to obtain a de-vignetted camera image and / or applying the vignetting effect to the virtual content image data. Alternatively, the vignette model may be used to modify the display data sent to LED wall controller 15 such that the vignetting effect of camera 2 in the input image is suppressed.

[0150] The vignette model is stored in the storage unit 32. It quantifies the reduction in brightness and / or saturation for each position within the image frame. The vignette model for the camera 2 is configured to quantify the reduction in brightness and / or saturation at each position within the image frame of the camera 2, which is affected by the limited aperture of the camera 2, particularly its lens system. FIG. 8 schematically illustrates a typical vignetting effect that occurs in a camera 2 with a lens system. As indicated by the increasing hatching density in FIG. 8, areas up to a certain radial distance around the center of the image frame may not exhibit significant vignetting, while areas around the image frame gradually decrease in brightness and / or saturation. Note that the camera's lens system may also cause a vignetting effect, with areas with a stronger vignetting effect located at the center of the image frame and decreasing the vignetting effect as the image moves away from the center.

[0151] The vignette model may be configured by a lookup table that contains, for each pixel in the image frame, a vignette value that quantifies the brightness and / or desaturation. To determine the vignette value for a location in the image frame of camera 2, the corresponding pixel coordinate is identified in the lookup table and the corresponding vignette value is looked up.

[0152] Alternatively, the vignette model can provide a (scalar) function that indicates the vignette value as a function of the (radial) distance d from the image center c of the image frame. Figure 9 shows a schematic representation of such a vignette model. The image center c is depicted along with three exemplary image (frame) positions p1, p2, and p3. To determine the vignette values ​​for image frame positions p1, p2, and p3, their respective distances d1, d2, and d3 to the image frame center c are determined, and the corresponding vignette values ​​are determined from the scalar function included in the vignette model. Image position p1 has the smallest distance d1 to center c, and therefore exhibits the lowest vignetting effect. The greatest vignetting effect is achieved at image position p3, which is located in the lower-left corner of the image frame and has the greatest distance d3 to the image center c.

[0153] The vignette model may consist of a linear function, an nth order polynomial function, or any other function suitable for modeling the vignetting effect of camera 2. The asymmetric alignment of the lens system of camera 2 with respect to the image sensor can be taken into account by an offset of the frame center c that can be provided in the vignette model.

[0154] Preferably, the vignette value is a number in the interval [0;1], where 0 indicates that the corresponding image location is completely obscured and 1 indicates no reduction in brightness (or minimal reduction in brightness relative to other image locations).

[0155] The vignette model preferably takes into account different aperture sizes of the camera 2. Thus, the vignette model preferably includes multiple vignette models for different aperture sizes of the camera 2.

[0156] The vignette model stored in the modeling unit 32 can be used in a variety of ways to improve the quality of the digitally enhanced output image.

[0157] 1 and 2, determining the augmented area 4 is relatively straightforward. In this scenario, it is preferable to modify the virtual content image data using a vignette model to reduce visual artifacts in the digitally augmented output image.

[0158] To provide a high-quality digitally augmented output image, it is necessary to ensure that the virtual content image data matches in brightness and / or saturation (and, if possible, the camera LUT) with the image data in the region of interest 1 of the input image. If the brightness and / or saturation of the virtual content image data differs from the image data in the region of interest 1 in the input image, visible seams will be obtained in the boundary area between the region of interest 1 and the augmented area 4.

[0159] Due to the vignetting effect of camera 2, significant discontinuities in brightness and / or saturation may occur if the input image of FIG. 2 is extended with virtual content image data in extended area 4 without properly considering the vignetting effect of camera 2. The image data included in the input image will obviously be subject to the vignetting effect of camera 2 because the input image is captured by camera 2.

[0160] To seamlessly complement the input image with the virtual content image data, the processing unit 31 is configured to receive a vignette model from the modeling unit 32. The processing unit 31 is configured to create virtual content image data for enhancing the input image by applying a vignette effect based on the vignette model to the virtual content image data. Here, a designated position of each pixel of the virtual content image data in the image frame can be determined, and the brightness and / or saturation value of each pixel of the virtual content image data can be modified based on the vignette model. When a vignette model with a vignette value of 0 to 1 is used, as described above in detail, the modification based on the vignette model can include multiplying the brightness and / or saturation value of each pixel of the virtual content image data by the corresponding vignette value.

[0161] By applying a vignetting effect to the virtual content image data based on the vignette model of camera 2, the virtual content image data is modified to appear as if it had been captured by camera 2. Subsequent augmentation of the input image with the modified virtual content image data results in a digitally augmented output image of improved quality, as visual artifacts at the boundary between the region of interest 1 and the augmented area 4 are suppressed.

[0162] FIG. 10 shows a flow diagram illustrating the creation of a view-angle-dependent calibration function according to a preferred embodiment of the present invention. In step A, reference image data is provided to the system 3. The reference image data is then sent to the LED wall controller 15 for display on the LED wall 11 (step B). The reference image data is displayed constantly for further processing to ensure a consistent appearance of the LED wall 11 (however, the appearance still changes as the LED wall view angle α varies). It is particularly preferred that the reference image data be configured so that the same color is displayed on the entire LED wall 11 with uniform brightness. Alternatively, the reference image data is configured so that the reference image data is displayed only in a limited area of ​​the LED wall 11, while the rest of the LED wall 11 remains dark. The following process for creating the view-angle-dependent calibration function may be performed separately for each color channel of the LED wall 11. In this case, the reference image data is configured so that the color corresponding to that color channel is displayed on the LED wall 11 with uniform brightness.

[0163] Next, a reference input image of the LED wall 11 is captured by the camera 2 at a predetermined reference LED wall view angle α0, preferably a 0° view angle at the center of the reference input image (Step C). The view angle α of the camera 2 is then corrected to the calibration LED wall view angle for which the value of the calibration function is determined. A calibration input image of the LED wall at the selected calibration LED wall view angle is captured (Step D). To determine the calibration function value for the selected calibration LED wall view angle, color information in a portion of the reference input image showing an area of ​​the LED wall 11 displaying the reference image data is compared with color information in a portion of the calibration input image. Preferably, the portions of the reference input image and the calibration input image to be compared are selected to be small enough to accurately represent the discrete LED wall view angles. More preferably, these portions are selected to depict corresponding areas of the LED wall 11. Based on the comparison, the value of the calibration function for the selected calibration LED wall view angle is determined. For example, if the calibration LED wall field angle is selected such that the perceived brightness of one or more color channels of the LED wall pixels 13 is reduced, then the comparison of color information in a (central) portion of the reference input image with a (corresponding) portion of the calibration input image may be performed by dividing the brightness values ​​of the individual color channels in the compared portions of the reference and calibration input images, thereby obtaining brightness and / or intensity ratios of the compared color channels or gain factors for the compared color channels, as discussed above.

[0164] Steps D and E are then repeated for multiple different calibration LED wall view angles (step F). A calibration function is obtained with multiple calibration function values ​​for different selected calibration LED wall view angles (which may be further supplemented by interpolating intermediate values ​​for LED wall view angles not selected as calibration LED wall view angles).

[0165] After creating the view-angle dependent calibration function for a particular color channel, the method steps shown in FIG. 10 may be repeated for another color channel of the LED wall 11 to obtain a view-angle dependent calibration function for the individual color channel, such as those exemplarily shown in FIGS. 6 and 7.

[0166] 11 shows a flow diagram of an alternative method for deriving the calibration function. Instead of deriving the calibration function value from a comparison of input images taken of the LED wall 11 with the same appearance, the appearance of the LED wall 11 when the camera is pointed at the calibration LED wall angle of view is varied until it matches the appearance of the LED wall 11 in the reference input image. The calibration function value is then based on the corresponding adjustment control signal to match the appearance of the LED wall 11.

[0167] Steps A'-C' correspond to steps A-C in the flow diagram of FIG. 10 , and therefore will not be described again. After capturing the reference input image, the camera 2 is reoriented to a selected calibration LED wall field angle, and a calibration function value is determined for this selected calibration LED wall field angle (step D'). Subsequently, a calibration input image of the LED wall 11 at the selected calibration LED wall field angle is captured (step E'). In step F', the color information of the reference input image and the calibration input image is compared similarly to step E of FIG. 10 , i.e., the color information of a portion of the reference input image is compared with the color information of a (corresponding) portion of the calibration input image. However, in contrast to the process shown in FIG. 10 , a calibration function value is not determined from this comparison. Rather, the comparison is performed to determine whether the color information of a portion of the reference input image matches the color information of a (corresponding) portion of the calibration input image. A match is determined if the difference in color information, e.g., the difference in luminance of individual color channels, is below a predetermined threshold. Due to variations in the LED wall field angle, this is typically not the case.

[0168] If the color information does not match, the system 3 is configured to determine an adjustment control signal based on the determined difference in color information. The adjustment control signal is determined to match the appearance of the LED wall 11 at a selected calibration LED wall view angle to the appearance of the LED wall 11 at a reference LED wall view angle. For example, if the calibration LED wall view angle is selected such that the perceived luminance of one or more color channels of the LED wall pixels 13 is lower than the reference input image, the adjustment control signal is determined to compensate for the luminance difference.

[0169] The determined adjustment control signal is used to modify the reference image data by either sending the adjustment control signal to the LED controller 15 which adjusts the reference image data and sends the modified image data to the LED wall 11, or by adjusting the reference image data in the processing unit 31 and sending the modified reference image data to the LED wall controller 15 for display on the LED wall 11 (step H').

[0170] A further calibration input image of the LED wall 11 depicting the modified reference input image data is captured, and the color information in the new calibration input image is compared with the color information in the reference input image. If the difference in color information exceeds a predetermined threshold, step H' is repeated. If the difference in color information falls below the predetermined threshold, a calibration function value based on the adjustment control signal is determined and stored (step G'). Method steps D'-H' are then repeated until a sufficient number of calibration function values ​​for different LED wall view angles have been determined. Further calibration function values ​​may be determined by interpolating additional calibration function values ​​for LED wall view angles not selected as calibration LED wall view angles.

[0171] Similar to the method shown in FIG. 10, the method of FIG. 11 may be performed separately for each individual color channel of the LED wall 11. [Explanation of symbols]

[0172] 1. Areas of Interest 11 LED Wall 12 people 13 LED wall pixels 14 color LED 15 LED Wall Controller 2 Cameras 21 Tracking System 3 (Control / Image Processing) System 31 Processing section 32 Storage section 33 Communications Department 4. Expansion Area p x Image (frame) position d x Distance from the center of the image c Image (frame) center o Optical axis vector n LED wall surface normal α LED Wall View Angle α h Horizontal LED wall viewing angle α v Vertical LED wall viewing angle

Claims

1. 1. A method, preferably a computer-implemented method, for controlling the appearance of an LED wall (11) depending on the angle of view of a camera (2) relative to said LED wall (11), comprising: a) The LED wall view angle (α, α) of the camera (2) relative to the LED wall (11) ｈ , α ｖ ) b) The determined LED wall angle of view (α, α ｈ , α ｖ and adjusting the appearance of the LED wall (11) based on the

2. 1. A method, preferably a computer-implemented method, for creating a digital augmented camera image based on an input image including image data of an LED wall (11) captured by a camera (2), comprising: c) controlling the appearance of the LED wall (11) according to the angle of view of the camera (2) relative to the LED wall (11) according to claim 1; d) taking, by said camera (2), at least one input image containing image data of said LED wall (11); e) determining at least one dilated area (4) in the input image to be digitally dilated; f) providing virtual content image data for at least said at least one extended area (4); g) generating a digital augmented output image by augmenting said input image with said virtual content image data at least in said at least one augmented area (4).

3. The LED wall view angle (α, α ｈ , α ｖ ) is determined by: providing a 3D model of said LED wall (11); establishing the position and orientation of the camera (2), preferably including establishing the position and orientation of the optical axis vector (o) of the camera (2); The LED wall view angle (α, α) is calculated based on the 3D model and the established position and orientation of the camera (2), and / or the established position of the camera (2) and the orientation of the optical axis vector (o). ｈ , α ｖ 3. The method of claim 1, further comprising the step of:

4. The step of adjusting the appearance of the LED wall (11) comprises adjusting the color information of the LED wall (11), preferably adjusting the brightness of the individual color channels of the LED wall (11) in accordance with the determined LED wall view angles (α, α ｈ , α ｖ 4. The method of claim 1, further comprising adjusting the value of the saturation level based on the saturation level.

5. The step of adjusting the appearance of the LED wall (11) includes adjusting the brightness of the color LEDs (14) of the LED wall pixels (13) of the LED wall (11), and preferably adjusts the LED wall viewing angle (α, α ｈ , α ｖ 5. The method according to claim 1, wherein the luminance θ is determined individually for a plurality of LED wall pixels (13), preferably for each LED wall pixel (13).

6. The LED wall angle of view (α ｈ , α ｖ The step of determining the LED wall view angles (α) for different reference orientations may be performed by ｈ , α ｖ 6. The method of claim 1, further comprising determining:

7. The method according to any one of claims 1 to 6, wherein the step of adjusting the appearance of the LED wall (11) is based on a field-angle dependent calibration function.

8. A method according to any one of claims 1 to 7, in particular claim 7, comprising a step of creating a field-of-view dependent calibration function for adjusting the appearance of the LED wall (11), preferably the field-of-view dependent calibration function being created separately for at least one, preferably each color channel of the LED wall (11).

9. The step of creating the angle-of-view dependent calibration function includes: taking a reference input image of the LED wall (11) at a predetermined reference angle of view; taking a plurality of calibration input images of the LED wall (11) at different angles of view; A method according to any one of claims 1 to 8, in particular claim 8, comprising a step of determining calibration function values ​​for the different angles of view based on a comparison of color information of a portion of the calibration input image depicting the LED wall with color information of a portion of the reference input image depicting the LED wall.

10. The step of determining the expansion area includes: providing a 3D model of said LED wall (11); establishing the position and attitude of the camera (2); calculating an image position of the LED wall (11) in the input image based on the 3D model of the LED wall (11) and the established position and orientation of the camera (2); and determining the extended area (4) based on the image position of the LED wall in the input image.

11. 10. A control system for controlling the appearance of an LED wall (11) as a function of the angle of view of a camera (2) relative to said LED wall (11), preferably by a method according to any one of claims 1 or 3 to 9, comprising: receiving image data to be displayed on the LED wall (11); The LED wall angle of view (α, α) of the camera (2) relative to the LED wall (11) ｈ , α ｖ ) is determined, The image data and the determined LED wall view angle (α, α ｈ , α ｖ a processing unit (31) configured to generate a display signal used by an LED wall controller (15) to adjust the appearance of the LED wall (11) based on the a communication unit (33) configured to communicatively couple a digital image processing system (3) to the LED wall controller (15) for controlling the LED wall (11), and to transmit the display signal to the LED wall controller (15) for adjusting the appearance of the LED wall (11).

12. 12. The control system according to claim 11, further comprising a memory unit (32) for storing data, preferably for storing a 3D model of the LED wall (11) provided to the processing unit (31), and further preferably for storing a view angle dependent calibration function provided to the processing unit (31) for creating the display signal.

13. A digital image processing system (3) for creating a digital augmented camera image based on an input image comprising image data of an LED wall (11), preferably by a method according to any one of claims 2 to 10, comprising: a control system for controlling the appearance of the LED wall (11) according to the angle of view of a camera (2) relative to the LED wall (11) according to claim 11 or 12; The processing unit (31) further comprises: receiving an input image from a camera (2); determining a dilated area (4) in the input image to be digitally dilated; providing virtual content image data for at least the expansion area (4); A digital image processing system (3) configured to generate a digital augmented output image by augmenting the input image with the virtual content image data in at least the augmented area (4).

14. The processing section (31) providing a 3D model of the LED wall (11) captured in the input image; Establishing the position and orientation of the camera (2); Calculating an image position of the LED wall (11) in the input image based on the 3D model of the LED wall (11) and the established position and orientation of the camera (2); 14. The digital image processing system (3) of claim 13, configured to determine the extended area (4) based on the image position of the LED wall (11) in the input image.

15. 15. The digital image processing system (3) according to claim 13 or 14, comprising a memory unit (32) for storing data, preferably for storing a 3D model of the LED wall (11) provided to the processing unit (31), and further preferably for storing a field-angle dependent calibration function provided to the processing unit (31) for creating the display signal.

Images