Selecting image analysis regions based on comparison of dynamics levels

Abstract

A system (1) is configured to determine a first dynamism level in a video frame, determine a second dynamism level in each of a plurality of analysis regions (52, 73) in the video frame, compare each of the second dynamism levels to the first dynamism level, select a subset of the analysis regions (73) based on the comparison, determine image characteristics from the subset of analysis regions in the video content, determine one or more light effects based on the image characteristics, and control one or more lighting devices to present the one or more light effects and / or store a light script specifying the one or more light effects to be presented when presenting the video content on a display device (9).

Description

Technical Field

[0001] The present invention relates to a system for determining image characteristics from an analysis region in video content, said image characteristics being used to determine one or more light effects to be presented on one or more lighting devices when said video content is presented on a display device.

[0002] The invention further relates to a method of determining image characteristics from an analysis area in video content, said image characteristics being used to determine one or more light effects to be presented on one or more lighting devices when said video content is presented on a display device.

[0003] The invention also relates to a computer program product enabling a computer system to perform such a method. Background Art

[0004] EP 1994801 Al discloses controlling an ambient lighting element comprising: receiving a content signal, analyzing the content signal to determine a motion vector of an object depicted in the content signal, presenting the content signal on a display device, and adjusting an ambient lighting effect provided by the ambient lighting element as determined by the motion vector.

[0005] Philips' Hue Entertainment and Hue Sync are growing in popularity. Philips Hue Sync enables light effects based on the content being played on a computer, such as video games. Dynamic lighting systems can dramatically impact the experience and impression of audiovisual material, especially when the colors emitted by the lights match the colors seen in the combined environment surrounding the screen. This new use of light can bring the atmosphere of video game lighting into the room with the player. Gamers can immerse themselves in the atmosphere of the game environment, enjoy the flash of weapon fire or the glow of magic spells, and sit in the glow of force fields as if they were real.

[0006] Hue Sync works by observing an analysis area of ​​video content and calculating light output parameters presented by Hue lights around the screen. A similar technique is described in US 2009 / 175536 A1. US 2009 / 175536 A1 discloses extracting and processing video content encoded in a rendered color space to simulate ambient light sources, including extracting color information from the video signal and transforming the color information through an unrendered color space using a three-primary color matrix to form a second rendered color space to drive the ambient light sources. Decoding the video signal into frames can allow for the extraction of average or other color information from selected screen areas to reduce bitstream load, and negative gamma correction helps prevent garish or inappropriate chroma and brightness.

[0007] Currently, in Hue Sync, fixed, predefined analysis areas are used for color extraction, i.e., for determining light effects. A disadvantage of this approach is that, for video games, the determined light effects are often considered to not match the video content of the video game. Summary of the Invention

[0008] A first object of the present invention is to provide a system capable of determining image characteristics from video content in a manner suitable for gaming.

[0009] A second object of the present invention is to provide a method capable of determining image characteristics from video content in a manner suitable for gaming.

[0010] In a first aspect of the invention, a system for determining image characteristics from an analysis area in video content, the image characteristics being used to determine one or more light effects to be presented on one or more lighting devices when the video content is presented on a display device, the system comprising at least one output interface and at least one processor configured to obtain a video frame, determine a first dynamism level in the video frame, and determine a second dynamism level in each of a plurality of analysis areas in the video frame.

[0011] The at least one processor is further configured to compare each of the second dynamism levels with the first dynamism level, select a subset of the analysis areas based on the comparison, determine image characteristics from the subset of analysis areas in the video content, determine one or more light effects based on the image characteristics, and control the one or more lighting devices using the at least one output interface to present the one or more light effects and / or store a light script specifying the one or more light effects.

[0012] The dynamics level in video frames indicates how dynamic those video frames are, i.e., how much the video frames change from one frame to the next. The dynamics level in an analysis region indicates how dynamic that analysis region is, i.e., how much the analysis region changes from one frame to the next. As will be described later in this specification, there are a variety of ways to determine the dynamics level.

[0013] Games can vary widely in how the "action" is distributed across the screen. In some games, it's the entire screen with some user interface (UI) elements. For other games, the UI might take up most of the screen; and for a third type of game, all the action might take place in the very center of the screen, with little or no action on the sides. The system allows for the selection of one or more analysis regions that have a dynamics level that is most similar to the overall dynamics level of the video frame. For example, dynamics levels can be considered similar if they represent similar amounts of change in luminance and / or chrominance.

[0014] While analysis regions with higher dynamics levels can often be preferred over those with lower dynamics levels, there are situations where higher dynamics levels are less preferred, for example because slower dynamics are more optimal for ambient light. Furthermore, analysis regions where changes primarily occur in brightness, while the overall video content has more colorful dynamics, are also less optimal. By selecting analysis regions based on a comparison of their dynamics levels with the dynamics levels of the video frame, the dynamics of the light effects can be aligned with the dynamics of the video content. This helps to match the light effects to the video content (e.g., of a video game).

[0015] The first dynamics level may be determined per pixel or per pixel region and then eg averaged.For example, the video frame and the video content may belong to the same game, the same type of game, or the same collection of games.

[0016] The at least one processor may be configured to select the subset of analysis areas by selecting one or more second dynamism levels that are similar or identical to the first dynamism level from the determined second dynamism levels. If second dynamism levels are determined for a sufficient number of analysis areas, then it is likely that one of these analysis areas has a second dynamism level that is identical or similar to the first dynamism level, resulting in a light effect that closely matches the video content. Alternatively, one or more second dynamism levels that are most similar to the first dynamism level may be selected.

[0017] One or more of the subset of analysis regions can have a different size and / or location than the default analysis region while remaining within the boundaries specified for the default analysis region. Typically, lighting devices are associated with a certain default analysis region based on their positioning, e.g., a lighting device on the left side of a TV is associated with a partition on the left side of the video frame. By maintaining the boundaries specified for the default analysis region, this association can be ensured to remain correct.

[0018] The video frame may be a portion of the video content. This allows, for example, the analysis region to be changed while a game is being played. Alternatively, for example, the analysis region may only be changed the next time the same game, type of game, or collection of games is played.

[0019] The at least one processor may be configured to determine the first dynamism level and / or the second dynamism level in one or more of the following ways (for optimal performance, multiple of these ways may be implemented and then compared):

[0020] A: The at least one processor may be configured to determine the first level of motion and / or the second level of motion by comparing consecutive ones of the video frames. A1: For example, the at least one processor may be configured to determine chrominance and / or luminance differences between the consecutive ones of the video frames. A2: For example, the at least one processor may be configured to detect edges in each of the consecutive ones of the video frames and determine a change in the detected edges between the consecutive ones of the video frames.

[0021] B: The at least one processor may be configured to determine the first dynamic level and / or the second dynamic level by determining a color histogram over the video frame. For example, the at least one processor may be configured to determine how many colors have appeared in the color histogram more than a predetermined number of times.

[0022] In a second aspect of the present invention, a method for determining image characteristics from an analysis area in video content, wherein the image characteristics are used to determine one or more light effects to be presented on one or more lighting devices when the video content is presented on a display device, the method comprising obtaining a video frame, determining a first dynamism level in the video frame, and determining a second dynamism level in each of a plurality of analysis areas in the video frame.

[0023] The method further includes comparing each of the second dynamism levels with the first dynamism level, selecting a subset of the analysis areas based on the comparison, determining image characteristics from the subset of analysis areas in the video content, determining one or more light effects based on the image characteristics, and controlling the one or more lighting devices to render the one or more light effects and / or storing a light script specifying the one or more light effects. The method may be performed by software running on a programmable device. The software may be provided as a computer program product.

[0024] Furthermore, a computer program for carrying out the methods described herein and a non-transitory computer-readable storage medium storing the computer program are provided.The computer program can, for example, be downloaded from or uploaded to an existing device, or stored when these systems are manufactured.

[0025] A non-transitory computer-readable storage medium stores software code portions that, when executed or processed by a computer, are configured to perform executable operations for determining image characteristics from an analysis region in video content, the image characteristics being used to determine one or more light effects to be presented on one or more lighting devices when the video content is presented on a display device.

[0026] The executable operations include obtaining a video frame, determining a first dynamism level in the video frame, determining a second dynamism level in each of a plurality of analysis areas in the video frame, comparing each of the second dynamism levels to the first dynamism level, selecting a subset of the analysis areas based on the comparison, determining image characteristics from the subset of analysis areas in the video content, determining one or more light effects based on the image characteristics, and controlling the one or more lighting devices to present the one or more light effects and / or storing a light script specifying the one or more light effects.

[0027] As will be appreciated by those skilled in the art, aspects of the present invention may be embodied as devices, methods, or computer program products. Thus, aspects of the present invention may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, which may all be generally referred to herein as "circuits," "modules," or "systems." The functions described in this disclosure may be implemented as algorithms executed by a processor / microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied (e.g., stored) thereon.

[0028] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of the present invention, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0029] A computer-readable signal medium may include a propagated data signal having computer-readable program code embodied therein (e.g., in baseband or as part of a carrier wave). Such a propagated signal may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0030] Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, cable, RF, or any suitable combination thereof. The computer program code for implementing the operations of aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java™, Smalltalk, or C++), conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages), and functional programming languages ​​(such as Scala or Haskell). The program code may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0031] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, particularly a microprocessor or central processing unit (CPU), to produce a machine such that instructions executed by the processor of the computer, other programmable data processing device, or other device create a device for implementing the functions / actions specified in the flowchart and / or one or more block diagram blocks.

[0032] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus, or other device to operate in a particular manner so that the instructions stored in the computer-readable medium produce an article of manufacture that includes instructions for implementing the functions / actions specified in the flowchart and / or one or more block diagram blocks.

[0033] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in the flowchart and / or block diagram blocks.

[0034] The flowcharts and block diagrams in the various figures illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions described in the blocks may not appear in the order described in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the opposite order depending on the functions involved. It will also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified function or action, or a combination of dedicated hardware and computer instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] These and other aspects of the invention are apparent from and will be further elucidated, by way of example, with reference to the accompanying drawings, in which:

[0036] Figure 1 is a block diagram of an embodiment of the system;

[0037] Figure 2 is a flow chart of a first embodiment of the method;

[0038] Figure 3 An example of video content being displayed is shown;

[0039] Figure 4 Shown for analysis Figure 3 Examples of analysis areas of video content;

[0040] Figure 5 An example of selecting an analysis region with off-center dynamics is shown;

[0041] Figure 6 An example of selecting an analysis area with colorful dynamics is shown;

[0042] Figure 7 An embodiment is shown where the central analysis area is restricted to certain boundaries;

[0043] Figure 8 An embodiment of determining the level of dynamics for five central analysis areas is shown;

[0044] Figure 9 is a flow chart of a second embodiment of the method;

[0045] Figure 10 is a flowchart of a third embodiment of the method;

[0046] Figure 11 is a flowchart of a fourth embodiment of the method;

[0047] Figure 12 An example of selecting an analysis area of ​​an LED strip is shown; and

[0048] Figure 13 is a block diagram of an exemplary data processing system for executing the method of the present invention.

[0049] Corresponding elements in the drawings are denoted by the same reference numerals. DETAILED DESCRIPTION

[0050] Figure 1 An embodiment of a system for determining image characteristics from an analysis region in video content is shown: a mobile device 1. When the video content is presented on a display device, the image characteristics are used to determine one or more light effects to be presented on one or more lighting devices. For example, mobile device 1 may be a mobile phone or tablet.

[0051] Mobile device 1 is connected to wireless LAN access point 17. Bridge 11 is also connected to wireless LAN access point 17, for example, via Ethernet. Lighting devices 13, 14, and 15 wirelessly communicate with bridge 11, for example, using the Zigbee protocol, and can be controlled, for example, by mobile device 1 via bridge 11. For example, bridge 11 may be a Philips Hue bridge, and lighting devices 13-15 may be Philips Hue lamps. Wireless LAN access point 17 is connected to the internet 18. An internet server 19 is also connected to the internet 18.

[0052] The mobile device 1 includes a processor 5, a receiver 3, a transmitter 4, a memory 7, and a display 9. The processor 5 is configured to obtain a video frame (e.g., by using screen capture software), determine a first dynamism level in the video frame, determine a second dynamism level in each of a plurality of analysis regions in the video frame, and compare each of the second dynamism levels to the first dynamism level.

[0053] The processor 5 is further configured to select a subset of analysis areas based on the comparison, determine image characteristics from the subset of analysis areas in the video content, determine one or more light effects based on the image characteristics, and control the lighting devices 13-15 to present the one or more light effects using the transmitter 4 and / or store a light script specifying the one or more light effects, for example, in the memory 7 or on an internet server 19. For example, the video content may be presented on the display 9 and / or the TV 21. The TV 21 is also connected to the wireless LAN access point 17.

[0054] For example, the video frames and video content may belong to the same game, the same type of game, or the same collection of games. For example, the game(s) may be obtained from an internet server 19 using receiver 3. Information identifying the determined analysis regions may be transmitted to internet server 19 to allow others to use them for the same game, the same type of game, or the same collection of games. For example, when other users start playing the same game, the same type of game, or the same collection of games, the determined analysis regions may be recommended to them.

[0055] The selection of a new analysis region is also referred to herein as "analysis region adjustment." The amount of analysis region adjustment that can be made can be limited by the mobile device 1. For example, the analysis region can only be moved or resized within predefined boundaries. For example, regions surrounding the current analysis region within the predefined boundaries (of the same or different sizes) can be analyzed, and if more suitable analysis regions are identified, they can be suggested to the user.

[0056] Predefined boundaries can be associated with zones (e.g., left, right, center). For example, if a zone is mapped to a lighting fixture in the center, the analysis area for that zone should not be significantly offset from the center, and the same applies to the left and right zones. Zones typically have a default analysis area.

[0057] The video frame can be part of the video content. Therefore, analysis region adjustments can be performed in real time (typically without user intervention). This can be implemented, for example, by using two analysis regions. The first analysis region is the current analysis region from which the light effects are determined. This is typically the default analysis region at the start of video content (e.g., a game). The second analysis region follows the action, for example, within the boundaries of the corresponding partition, based on object / blob detection in a dynamics heat map.

[0058] The fact that the second analysis area includes spots does not necessarily mean that the second analysis area's dynamics level is more similar to the overall dynamics level than that of the first analysis area. However, if the comparison indicates this is the case, then the second analysis area is selected to optimize the dynamics of the generated light effect. For example, in a shooter game, some action may drift off-center during brief combat. This will be detected, and the light presentation will be targeted to that area within the necessary timeframe.

[0059] exist Figure 1 In the embodiment of mobile device 1 shown in FIG, mobile device 1 includes a processor 5. In alternative embodiments, mobile device 1 includes multiple processors. Processor 5 of mobile device 1 can be a general-purpose processor (e.g., from Qualcomm or ARM-based) or a dedicated processor. For example, processor 5 of mobile device 1 can run an Android or iOS operating system. Memory 7 can include one or more memory units. For example, memory 7 can include solid-state memory. For example, memory 7 can be used to store the operating system, applications, and application data.

[0060] For example, the receiver 3 and transmitter 4 may communicate with the wireless LAN access point 17 using one or more wireless communication technologies, such as Wi-Fi (IEEE 802.11). In alternative embodiments, multiple receivers and / or multiple transmitters are used instead of a single receiver and a single transmitter. Figure 1 In the embodiment shown in , a separate receiver and a separate transmitter are used. In an alternative embodiment, the receiver 3 and the transmitter 4 are combined into a transceiver. For example, the display 9 may include an LCD or OLED panel. The display 9 may be a touch screen. The mobile device 1 may include other components typical for mobile devices, such as a battery and a power connector. The present invention may be implemented using a computer program running on one or more processors.

[0061] exist Figure 1 In the embodiment of the present invention, the system of the present invention is a mobile device. In alternative embodiments, the system of the present invention is a different device, such as a PC, a laptop, a TV (such as TV 21), an HDMI module, or an Internet server (such as Internet server 19).

[0062] The HDMI module can capture content sent by the PC to the TV 21 by connecting between the PC and the TV 21 and then sending light commands to the bridge 11 or directly to the lighting devices 13-15. This approach can be used if the content is provided to the TV 21 via an HDMI cable.

[0063] Figure 2 A first embodiment of a method for determining image characteristics from an analysis region in video content is shown in FIG. When the video content is presented on a display device, the image characteristics are used to determine one or more light effects to be presented on one or more lighting devices. Step 101 includes obtaining a video frame. Step 103 includes determining a first dynamics level in the video frame. Step 105 includes determining a second dynamics level in each of a plurality of analysis regions in the video frame. Step 107 includes comparing each of the second dynamics levels to the first dynamics level.

[0064] Step 109 includes selecting a subset of analysis areas based on the comparison. Step 111 includes determining image characteristics from the subset of analysis areas in the video content. Step 113 includes determining one or more light effects based on the image characteristics. Step 115 includes controlling one or more lighting devices to present the one or more light effects and / or storing a light script specifying the one or more light effects.

[0065] Thus, while the user is playing a game or watching other video content, the dynamic level is monitored across the entire screen (e.g., at each pixel or each small area), and these measured dynamics are then compared to the dynamics of each analysis area (which corresponds to the dynamics of the light effect determined from that analysis area), and the analysis area(s) with the most similar dynamic level to the entire screen can be used as the new analysis area(s), e.g., just before starting the game next time. For example, the selected analysis area(s) can have a higher dynamic level, more colorful dynamics (i.e., changes occur more in color than just brightness), or slower dynamics (e.g., more optimal for ambient light) than the unselected analysis area(s).

[0066] With the help of Figures 3 to 8 A first embodiment of the method is shown. Figure 3 An example of video content being displayed on the display 9 of the mobile device 1 is shown. Typically, multiple regions of the screen are mapped to different lighting devices, and each region of the analysis area is analyzed separately. Figure 4Three analysis areas 51, 52 and 53 are shown which are respectively mapped to lighting devices 13, 14 and 15. In a basic implementation of step 113, the color of each analysis area is extracted from the pixels in the analysis area and used for the light effect of the corresponding lighting device.

[0067] If combined Figures 9 to 11 As described above, dynamics levels can be determined using different algorithms. Dynamics heatmaps can give a good picture of both overall content dynamics and the dynamics of each pixel or small region. When creating such a map, the dynamics level of each analyzed region can be determined and compared to the overall dynamics.

[0068] Figure 5 and Figure 6 An example is shown in which an analysis region different from the default analysis region is more optimal than the default analysis region. Figure 5 An example of selecting an analysis region with off-center dynamics is shown. Figure 5 An alternative analysis region 61 is shown, along with the default analysis region 52 for the video content center.

[0069] In some games, such as League of Legends, there is often a lot of off-center action, and the dynamics level of the default center analysis area is often lower than the overall dynamics level, resulting in lighting effects that are less dynamic than the displayed video content. Figure 5 The dynamics heatmap is shown with the highest level of dynamics in spot 63. An alternative analysis region 61 has been identified to surround this spot 63, and it has been determined that selecting the alternative analysis region 61 rather than the default analysis region 52 is beneficial because it will result in a light effect with a higher level of dynamics, more similar to the light effect of video content. When the game is started again, the user may be advised to move the center partition to achieve better alignment between the level of dynamics of the screen and the level of dynamics of the light effect.

[0070] If real-time analysis is desired, it may be beneficial to determine the dynamics level of only a few analysis regions for each different subarea of ​​the screen (e.g., left, center, right). This process can be performed iteratively, including at least one new analysis region each time. For example, one or more new analysis regions can be included next to the current analysis region to see if the dynamics level of the center analysis region increases if the center analysis region moves slightly.

[0071] Figure 6 An example of selecting an analysis area with rich and dynamic features is shown. Figure 6 The default analysis area 52 and the alternative analysis area 73 for the video content center are shown. Some games, such as Call of Duty, have visualizations of the weapons carried by the characters. Figure 6, figure 71 is such a visualization of a weapon. As the character runs or walks, the weapon moves up and down—often entering and leaving the default analysis area—and creates very dynamic light effects that don't match the level of dynamism on screen.

[0072] Furthermore, the default analysis area 52 will have a very low level of color dynamics: although there can be a lot of movement, the same set of colors is used. On the other hand, the alternative analysis area 73 (the area slightly above the center of the screen and just above the weapon) has more colorful dynamics. Therefore, it is beneficial to select the alternative analysis area 73 instead of the default analysis area 52, as this will result in an overall less dynamic, but more colorful light effect. The next time the user starts the game, they will receive a suggestion for moving the center area to avoid the influence of gun movement.

[0073] exist Figure 5 and Figure 6 In the example shown, the central analysis area is moved, but its size remains unchanged. However, it's also possible to change the size of the analysis area in addition to or instead of it. For example, the sides of video content are often used to create atmosphere. If a very dynamic UI element (e.g., chat in an online game) is used to one side of the video content, this will create a very dynamic light effect that doesn't fit the overall game dynamics. In this case, the system can recommend a larger analysis area to reduce the impact of the dynamic UI element, i.e., to create a slower motion.

[0074] While it might be better to simply remove these dynamic parts from the analysis entirely, in many cases these UI elements are transparent (such as overlay chat), so removing them could affect the overall color of the light effect. Additionally, because it's not always clear what the boundaries of these areas are, it's simpler to simply increase the size of the analysis area to reduce their impact on these UI elements.

[0075] Figure 7 An example is shown in which the alternative analysis area 85 has a smaller size than the default analysis area 52. Figure 7 The reason for the smaller size is that Figure 7 In the embodiment of , the central analysis region is constrained to certain boundaries: the boundaries of rectangle 81. Thus, the alternative analysis region 85 has a different size and position than the default analysis region 52 while remaining within the boundaries specified for the default analysis region 52. Figure 5 As in the example of , the alternative analysis area 85 is selected instead of the default analysis area 52, as this will result in a light effect with a higher level of dynamics.

[0076] exist Figure 7In the embodiment of , object / blob detection is repeatedly applied on the dynamic heat map and, if necessary, a new central analysis region is selected. For example, if blob 83 has moved to the left, another analysis region is selected that includes blob 83. Figure 7 In the embodiment of FIG. 8 , the new analysis area may need to be (even) smaller in size to remain within the boundaries of rectangle 81 , for example if spot 83 has also moved upwards.

[0077] exist Figures 5 to 7 In the example above, the overall dynamism level is compared to the dynamism levels of two analysis areas—the default analysis area and the alternative analysis area. Figure 8 An embodiment of determining the level of dynamism for five central analysis regions, a default central analysis region 52 and four alternative analysis regions 91-94, is shown. In this embodiment, the central analysis regions are restricted to certain boundaries, namely the boundaries of a rectangle 81, similar to Figure 7 In the embodiment of Figure 8 In the examples above, the analysis areas are all the same size.

[0078] exist Figures 5 to 8 In the example of , analysis regions are selected only for the center partition. Alternatively or additionally, analysis regions may be selected for other partitions (eg, for the left partition and the right partition).

[0079] Figure 9 A second embodiment of a method for determining image characteristics from an analysis area in video content is shown in FIG. Figure 9 In the embodiment, Figure 3 Step 103 includes sub-step 131, Figure 3 Step 105 includes sub-step 133, and Figure 3 Step 109 comprises sub-step 135. The first dynamics level determined in step 131 represents the overall dynamics of the video frame and is determined per pixel or per pixel region.

[0080] In steps 131 and 133, a first dynamics level is determined in the video frame and a second dynamics level is determined in each of a plurality of analysis regions in the video frame, respectively. In steps 131 and 133, the first dynamics level and / or the second dynamics level are determined by comparing consecutive video frames in the video frame.

[0081] Steps 131 and 133 involve determining the chrominance and / or luminance differences between these consecutive video frames, specifically the cumulative frame differences. In these steps, the differences between each consecutive frame are accumulated over a set time (e.g., a playback session). Additional thresholds or threshold functions can be applied to reduce the impact of small variations. For example, the cumulative differences can be calculated in grayscale, where all colors are separated and measured by RGB color combinations, or by luminance and color components (e.g., in XYZ or Lab color space).

[0082] Step 107 comprises comparing each of the second dynamism levels with the first dynamism level. Step 135 comprises selecting a subset of the analysis areas by selecting one or more second dynamism levels that are (most) similar or identical to the first dynamism level. Next, perform Figure 3 Steps 111-115.

[0083] Figure 10 A third embodiment of a method for determining image characteristics from an analysis area in video content is shown in FIG. Figure 10 In the embodiment, Figure 3 Step 103 includes sub-step 141, and Figure 3 Step 105 comprises sub-step 143. The first dynamics level determined in step 141 represents the overall dynamics of the video frame and is determined per pixel or per pixel region.

[0084] In steps 141 and 143, a first dynamics level is determined in the video frame and a second dynamics level is determined in each of a plurality of analysis regions in the video frame, respectively. In steps 141 and 143, the first dynamics level and / or the second dynamics level are determined by comparing consecutive video frames in the video frame.

[0085] Steps 141 and 143 include detecting an edge in each of the consecutive ones of the video frames and determining a change in the detected edge between the consecutive ones of the video frames. Steps 141 and 143 are similar to Figure 9 The same steps 131 and 133 are used in the previous example, but they use the cumulative difference of edges instead of the cumulative difference of colors. As a result, areas with a lot of object movement are highlighted compared to stable areas of the screen (such as the edges of the UI).

[0086] Step 107 comprises comparing each of the second dynamics levels with the first dynamics level. Figure 9 , step 109 comprises a sub-step 135 which comprises selecting a subset of the analysis areas by selecting one or more second dynamic levels that are (most) similar or identical to the first dynamic level. Next, perform Figure 3 Steps 111-115.

[0087] Figure 11 A fourth embodiment of a method for determining image characteristics from an analysis area in video content is shown in FIG. Figure 11 In the embodiment, Figure 3 Step 103 includes sub-step 151, and Figure 3 Step 105 comprises sub-step 153. The first dynamics level determined in step 151 represents the overall dynamics of the video frame and is determined per pixel or per pixel region.

[0088] In steps 151 and 153, a first level of dynamism is determined in the video frame and a second level of dynamism is determined in each of a plurality of analysis regions in the video frame, respectively. In steps 151 and 153, the first level of dynamism and the second level of dynamism are determined by determining a color histogram over the video frame. This is also referred to herein as entropy estimation.

[0089] Instead of calculating the difference between each consecutive frame, the change in chrominance and / or luminance of each pixel / region over the measurement time is used to create a chrominance / luminance histogram of the pixel or region, and this color histogram is then used to measure entropy, which will indicate how much the pixel has changed over time. Steps 151 and 153 also include determining how many colors have appeared in the color histogram more than a predetermined number of times. In an alternative embodiment, the level of dynamics is determined from the color histogram in another manner.

[0090] Step 107 comprises comparing each of the second dynamics levels with the first dynamics level. Figure 9 In the embodiment, step 109 includes a sub-step 135, which includes selecting a subset of the analysis areas by selecting one or more second dynamic levels that are similar or identical to the first dynamic level. Figure 3 Steps 111-115.

[0091] exist Figures 9 to 11 In some embodiments, different methods are used to determine the level of dynamics. In alternative embodiments, multiple of these methods are combined in the same embodiment. For example, the level of dynamics determined using different methods can then be compared, making it possible to distinguish between overall dynamics and color dynamics. This results in better performance.

[0092] Figure 12An example of selecting an analysis area for a pixelated LED strip 200 is shown. Because each LED of a pixelated LED strip is typically mapped to a relatively small analysis area, even slightly shifting the analysis area can result in undesirable lighting effects that appear to be mismatched with the content. This can be mitigated or prevented by increasing the analysis area without changing the center of the area, but different solutions are available for pixelated LED strips.

[0093] In this example, LED strip 200 has five LEDs 201-205. LEDs 201-205 are associated with default analysis regions 211-215, respectively. For each LED, a dynamic level for the default analysis region and a dynamic level for the corresponding alternative analysis region are determined. Alternative analysis regions 221-225 are larger than the corresponding default analysis regions 211-215, but have the same center as the corresponding default analysis regions 211-215.

[0094] If, for example, the dynamism level of the alternative analysis region 223 is more similar to the overall dynamism level than the dynamism level of the default analysis region 213, and the alternative analysis region is sufficiently similar to the overall dynamism level, then the alternative analysis region 213 is selected. If the dynamism levels of neither the default analysis region 213 nor the alternative analysis region 223 are sufficiently similar to the overall dynamism level, then one of the following solutions may be used:

[0095] A. Blending the light effects determined for LED 203 from adjacent analysis zones (i.e., analysis zones 212 and 214) creates a uniform light effect and prevents mismatches between the dynamics of LED 203 and the overall dynamics. Thus, adjacent zones 212 and 214 can form a subset of the analysis zones selected for LED 203. Light effects determined from adjacent analysis zones can even be blended with light effects determined from one of the analysis zones associated with LED 203 (e.g., default analysis zone 213 or alternative analysis zone 223). For example, some parameters of the light effects determined from analysis zones 212 and 214 (e.g., parameters representing dynamics) can be blended, while one or more other parameters (e.g., parameters representing the color palette) are taken from analysis zones 213 or 223.

[0096] B. Instead of mixing light effects, the same result can be achieved by determining the light effects directly from the combined analysis areas 212 and 214, rather than first determining the light effects from the analysis areas 212 and 214 separately and then mixing them. The advantage of solution B over solution A is that it maintains the overall architecture, i.e., mapping the area(s) on the screen to the light sources (LEDs). The advantage of solution A over solution B is that the computational requirements are lower.

[0097] C. Instead of changing the size of the analysis area or moving the analysis area, the weight of each pixel's contribution to the generated light effect can be varied. In this way, analysis area 213 or 223 can still be selected for LED 203, but the impact of static or overly dynamic pixels can be reduced compared to other pixels within the same analysis area.

[0098] Figure 13 Depicted instructions can be executed as reference Figure 2 and Figures 9 to 11 A block diagram of an exemplary data processing system for the described methods.

[0099] like Figure 13 As shown in , data processing system 300 may include at least one processor 302 coupled to memory element 304 via system bus 306. In this way, the data processing system may store program code in memory element 304. Further, processor 302 may execute program code accessed from memory element 304 via system bus 306. In one aspect, the data processing system may be implemented as a computer suitable for storing and / or executing program code. However, it should be appreciated that data processing system 300 may be implemented in the form of any system including a processor and memory capable of performing the functions described in this specification.

[0100] Memory element 304 may include one or more physical memory devices, such as, for example, local memory 308 and one or more mass storage devices 310. Local memory may refer to random access memory or other non-persistent storage device(s) typically used during the actual execution of program code. Mass storage devices may be implemented as hard drives or other persistent data storage devices. Processing system 300 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code in order to reduce the number of times program code must be retrieved from mass storage device 310 during execution. For example, if processing system 300 is part of a cloud computing platform, processing system 300 may also be able to use memory elements of another processing system.

[0101] Optionally, input / output (I / O) devices, depicted as input device 312 and output device 314, may be coupled to the data processing system. Examples of input devices may include, but are not limited to, a keyboard, a pointing device such as a mouse, or a microphone (e.g., for voice and / or speech recognition). Examples of output devices may include, but are not limited to, a monitor or display, or speakers. Input and / or output devices may be coupled to the data processing system directly or through intervening I / O controllers.

[0102] In an embodiment, the input and output devices may be implemented as a combined input / output device (in Figure 13 314). An example of such a combined device is a touch-sensitive display, sometimes also referred to as a "touch screen display" or simply a "touch screen." In such an embodiment, input to the device can be provided by movement of a physical object (such as, for example, a user's finger or a stylus) on or near the touch screen display.

[0103] Network adapter 316 may also be coupled to data processing system 300 to enable it to couple to other systems, computer systems, remote network devices, and / or remote storage devices through intervening private or public networks. Network adapters may include data receivers for receiving data transmitted to data processing system 300 by the systems, devices, and / or networks, as well as data transmitters for transmitting data from data processing system 300 to the systems, devices, and / or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapters that may be used with data processing system 300.

[0104] like Figure 13 As shown in FIG, memory element 304 can store application programs 318. In various embodiments, application programs 318 can be stored in local memory 308, one or more mass storage devices 310, or separate from local memory and mass storage devices. It should be appreciated that data processing system 300 can further execute an operating system (OS) that can facilitate the execution of application programs 318. Figure 13 ). Application 318, implemented in the form of executable program code, may be executed by data processing system 300 (eg, by processor 302). In response to executing the application, data processing system 300 may be configured to perform one or more operations or method steps described herein.

[0105] Various embodiments of the present invention may be implemented as a program product for use with a computer system, wherein the program(s) of the program product define functionality of the embodiments (including the methods described herein). In one embodiment, the program(s) may be embodied on various non-transitory computer-readable storage media, where, as used herein, the term "non-transitory computer-readable storage medium" includes all computer-readable media with the sole exception of transitory propagated signals. In another embodiment, the program(s) may be embodied on various transitory computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media on which information is permanently stored (e.g., read-only memory devices within a computer, such as CD-ROM disks readable by a CD-ROM drive, ROM chips, or any type of solid-state non-volatile semiconductor memory); and (ii) writable storage media on which information is stored that can be altered (e.g., flash memory, a floppy disk within a floppy disk drive or hard drive, or any type of solid-state random-access semiconductor memory). The computer program(s) may be executed on the processor 302 described herein.

[0106] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the present invention. As used herein, the singular forms "a" or "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that when used in this specification, the terms "comprise" and / or "comprising" specify the presence of stated features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0107] The corresponding structures, materials, actions, and equivalents of all means or step plus function elements in the following claims are intended to include any structure, material, or action for performing a function in combination with other claimed elements as specifically claimed. The description of the embodiments of the present invention has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the embodiments in the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiments are selected and described in order to best explain the principles of the invention and some practical applications, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications suitable for the particular use under consideration.

Claims

1. A system (1) for determining image characteristics from an analysis area in video content, the image characteristics being used to determine one or more light effects to be presented on one or more lighting devices (13-15) when the video content is presented on a display device, the system (1) comprising: at least one output interface (4); and At least one processor (5) configured to: - Get continuous video frames, - determining a first dynamics level in the video frames, wherein the first dynamics level in the video frames represents how much the video frames vary between consecutive ones of the video frames, - determining a second dynamism level in each of a plurality of analysis areas (52, 73, 61) in the video frame, wherein the second dynamism level in the plurality of analysis areas in the video frame represents how much the respective analysis area changes in consecutive ones of the video frames, - comparing each of said second dynamism levels with said first dynamism level, - selecting a subset of said analysis areas based on said comparison (73, 61), - determining image characteristics from a subset of said analysis areas in said video content, - determining one or more light effects based on said image characteristics, and - using the at least one output interface (4) to control the one or more lighting devices (13-15) to render the one or more light effects and / or to store a light script specifying the one or more light effects.

2. The system (1) according to claim 1, wherein the first dynamics level represents the overall dynamics of the video frame.

3. The system (1) according to claim 2, wherein the at least one processor (5) is configured to select the subset (73, 61) of the analysis areas by selecting one or more second dynamism levels from the second dynamism levels that are similar to or the same as the first dynamism level.

4. System (1) according to claim 1 or 2, wherein the first dynamics level is determined per pixel or per pixel region.

5. The system (1) according to claim 1 or 2, wherein at least one of the subset of analysis regions has a different size and / or position than a default analysis region while remaining within boundaries specified for the default analysis region.

6. The system (1) according to claim 1 or 2, wherein the video frame is a part of the video content.

7. The system (1) according to claim 1 or 2, wherein the video frame and the video content belong to the same game, the same type of games, or the same collection of games.

8. The system (1) according to claim 1, wherein the at least one processor (5) is configured to determine the first dynamism level and / or the second dynamism level by comparing consecutive ones of the video frames.

9. The system (1) according to claim 8, wherein the at least one processor (5) is configured to determine chrominance and / or luminance differences in the consecutive ones of the video frames.

10. The system (1) of claim 8, wherein the at least one processor (5) is configured to detect edges in each of the consecutive ones of the video frames and determine a change in the detected edges between the consecutive ones of the video frames.

11. The system (1) according to claim 1, wherein the at least one processor (5) is configured to determine the first dynamism level and / or the second dynamism level by determining a color histogram over the video frame.

12. The system (1) according to claim 11, wherein the at least one processor (5) is configured to determine how many colors have appeared in the color histogram more than a predetermined number of times.

13. A method of determining image characteristics from an analysis region in video content, the image characteristics being used to determine one or more light effects to be presented on one or more lighting devices when the video content is presented on a display device, the method comprising: - obtaining (101) consecutive video frames; - determining (103) a first level of dynamics in the video frames, wherein the first level of dynamics in the video frames represents how much the video frames vary between consecutive ones of the video frames; - determining (105) a second dynamism level in each of a plurality of analysis regions in the video frame, wherein the second dynamism level in the plurality of analysis regions in the video frame represents how much the respective analysis region changes in consecutive ones of the video frames; - comparing each of said second dynamism levels with said first dynamism level (107); - selecting (109) a subset of said analysis areas based on said comparison; - determining (111) image characteristics from a subset of said analysis areas in said video content; - determining (113) one or more light effects based on the image characteristics; as well as - controlling (115) said one or more lighting devices to render said one or more light effects and / or storing a light script specifying said one or more light effects.

14. A computer program product comprising at least one software code portion configured to enable the method of claim 13 to be performed when the software code portion is run on a computer system.

15. A non-transitory computer-readable storage medium storing software code portions, wherein the software code portions, when executed or processed by a computer, are configured to perform the method of claim 13.

Images