Coordinating dynamic HDR camera capture

Abstract

In order to meet the future requirements of high quality but economically viable high dynamic range video program creation, the inventors propose a method of setting in a video camera (201) a video camera capture mode specifying output pixel brightness in one or more hierarchical versions of a sequence of output video images, the method comprises:-moving an operator of the video camera to at least two positions (Pos1, Pos2) in the scene with different illumination compared to each other, and capturing at least one high dynamic range image (oImHDR) for each of the at least two positions of the scene; -a color composition instructs analysis of the captured at least one high dynamic range image for each of the at least two positions to determine a respective maximum luminance region and to determine, from the maximum luminance, at least one of: an aperture setting, a shutter time and an analog gain setting of the camera; for each of the at least two positions (Pos1, Pos2), capturing a respective main capture (RW) using the determined at least one of the aperture setting, the shutter time setting and the analog gain setting of the camera, and keeping the determined aperture setting, the shutter time setting and the analog gain setting constant for subsequent capture at least in the corresponding position; -determining at least a respective first hierarchical image (ODR) of said respective primary capture, comprising mapping the digits of said primary capture to Nit values of said first hierarchical image (ODR) by such a luminance distribution function (FLM) by means of the shape of a respective luminance distribution function directed by a color composition to establish; and-storing the respective brightness assignment functions (Fs1, Fs2) for the at least two positions or parameters uniquely defining these functions in the respective memory positions (221, 222) of the camera.

Description

Technical Field

[0001] The present invention relates to methods and apparatus for coordinating the capture of images by one or more cameras in various lighting regions of a scene, in particular for generating an image signal comprising a main high dynamic range image and a luminance mapping function for calculating a secondary graded image having a generally lower dynamic range different from that of the main HDR image based on the pixel colors of the main HDR image. Background Art

[0002] Optimal camera exposure is a difficult problem in non-uniform lighting environments (especially when there is no Single view However, when freely moving through various regions of different illumination and object brightness). Historically, many video productions were performed under controlled lighting (e.g., studio captures for dramas or news), where, for example, lights were spread across the ceiling to create a uniform base lighting. Nowadays, there is a desire to shoot on location, and also for cost reasons, sometimes with small teams (possibly one presenter, one person in charge of the camera, and one person in charge of audio). In addition, the boundary between professional and "non-professional" producers is becoming less clear, as we can see from internet video bloggers (vloggers) (who may know a little about lighting techniques, but always not enough to avoid challenging lighting conditions), or even amateur captures using mobile phones may be the most interesting news coverage of a local event. Other technical aids in terms of obtaining (good) YcbCr pixel color codes would always be useful. Especially for small production teams, if technical components can take on some of the burden of creating a good-looking video, the creators can focus more on other aspects of video creation, such as creativity, story, etc. (while not necessarily wanting an automatic system where they have no say in the color and brightness of various image objects).

[0003] It has been prepared in SDR (Standard Dynamic Range, also known as Low Dynamic Range LDR). Although ideally, people work under well-lit conditions (e.g., studio 1000 lux), one may have capture situations below ~300 lux (daytime; indoors), below ~10 lux (low light), or even below 1 lux (dim) [the ~ symbol indicates "about"]. When using an "amateur" camera, automatic exposure can occur quite automatically (in more professional systems, usually one will control some aspects, such as the maximum amount of noise, although in some cases, having some noise and at least seeing something may be better than seeing very little). The real-world environment can not only produce very different AverageBrightness, or the light level or illumination of the various regions in the scene to be captured, and (especially when there are luminous objects in the scene) there may be a significant spread or ratio between the brightness of the object regions projected onto different sensor pixels. For example, just before sunrise, the local environment where the cameraman is located may not have received any direct sunlight yet, but the distant sky may already be illuminated by the sun. Therefore, the sky may have a brightness of a few nits (which is the name of an engineer and a better pronunciation of the physical unit Cd / m^2), while the objects around you may have only a few nits or less brightness. At night, the average street brightness can be a few nits, yet when viewed in the direction of a light source, tens of thousands of nits can be seen. During the day in sunlight, an object may reflect thousands of nits, but again it all depends on whether we have a white diffused (or even specularly reflected) object in sunlight or a black object in some shaded area. And indoor objects will fall near the 100 nit level, again depending on whether the object is, for example, placed in a sunbeam near a window or in an adjacent unlit room, and when the door to the unlit room is opened, the unlit room can also be captured from the same shooting position. In fact, this is exactly why engineers want to move to an HDR imaging chain (another factor is the visual impact on the viewer). SDR cameras that do not utilize so many optoelectronics and have a small dynamic range between the darkest (noiseless) capture of dark object brightness and full pixel goodness will typically correctly expose to an actor or speaker indoors, thus clipping the objects outside the window to white (or a soft color). Sometimes, if the presenter is, for example, positioned facing a bookshelf deep into the room, illuminated with quadratic attenuation away from the window, and half of the bookshelf facing the window disappears (becomes clipped to maximum white), this will produce very strange results. In fact, in the case of such a limited dynamic range in at least one of the camera, codec, and display, lighting and exposure need to be very carefully considered, and usually at least something will be clipped to white or black. In the worst-case scenario, but those are usually only shot by amateurs (unless for a specific artistic reason), even half of the face will be clipped to white.

[0004] However, there are not only camera problems (regarding its maximum capture ability and the best controlled use of these), but also problems with the way of encoding the captured images in a standard way. Even if the camera can accurately capture very deep blacks, i.e., having a background noise of, say, less than 1 / 10,000 of a full pixel, the typical representation in any output image is to place the main person or object in the scene at about 25% of white (the white to be displayed usually occurs by driving the monitor to its maximum luminance, e.g., backlight fully on and LCD pixels fully open), as a percentage of the peak luminance (which typically corresponds to a luminance of 128 out of 255 in an approximate square root luminance encoding). This means that under these production standards, only 2 stops of luminance (i.e., multiplication factor x2) remain above the facial color, ensuring that anything brighter than the local diffused white in the scene will be clipped (possibly even in the raw camera capture of the sensor if the sensor is exposed according to the same technical standards). One might want to tune this by using the camera's controls, but this would be the setting from Current position of Current view in the current scene. Different scene shots may (if they exist at all) be slightly fine-tuned in post-production, but only used and blended together. Even if automatic camera exposure control can be applied multiple times to different shots at different locations, there is no relationship between these settings.

[0005] The human eye adapts to all of these in an almost perfect way, because it can change the chemistry of the signal path in the cones, resulting in different sensitivities, and, as desired, locally weight the signals of neurons for brighter and darker objects in the field of view (for example, staring at a bright red square for some time will thereafter make you see a negative red-cyan square in your field of view roughly where the original square was imaged, but this will soon be corrected again). In the brain, we are most interested in what type of object we see, such as a sufficiently ripe yellow banana, and not so much in the exact degree to which the banana is illuminated by which sunbeam. The brain wants to arrive at a representation that is ultimately general, allowing a tiger to be seen hiding in the bushes, whether during the day or at night. If we want to represent an original scene with original conditions (such as the original brightness in the field of view) by means of a (reasonably truthful or rough) simulation (such as a display version of the scene under dim living room lighting at night), ideally, this simulation contains more or less the relevant information regarding human vision, such that the brain can attempt to form the illusion of more or less seeing the exact scene. However, a camera counts photons, converting each group of N incident photons into measured photoelectrons, and is in this regard both a simple device and a dumb device for advanced applications. If some precise measurement is needed, such as whether a part of a building is sufficiently illuminated, this counting will be good, but this property is not very suitable for a video display chain. Even with one camera, a difference can be seen if, for example, a camera drives into a tunnel and then out of the tunnel back into a sunny environment. The lighting inside the tunnel is designed to be optimized for human vision and not necessarily for any camera. In some cases of latency, when driving out of the tunnel, the image of the external environment is first seen as almost completely white (in the capture), and then an automatic exposure algorithm based on the average brightness adjusts the external environment image back to a technically well-exposed image. As humans, we see the normal impression of a darker impression inside the tunnel and a bright but usually still well-visible outside environment. That is, human vision seems to be fully adapted to most of the lighting conditions in our world, even if many of them are artificial. Only under the worst conditions may there be visibility problems, such as when driving into a shaded area on a sunny day, mainly because sunlight is reflected on the dust on the front window.

[0006] In many scenarios, more than one camera may be involved in video production and more than one mobile cameraman may be involved, and it may be desirable to coordinate the luminance styles of those cameras. In several productions, this can be done in a separate venue or device, such as in an outside broadcast (OB) van (or even in a grading suite in the case of non-live broadcasts). Usually, everything related to the production of high-quality video must happen in real time and is thus carried out by expert teams focusing on different matters. For example, the director of a sports broadcast is too preoccupied and busy with everything related to the capture, except which cameraman should roughly capture what, and thus which main camera feed he can choose to ultimately be in the final broadcast signal at that time. In fact, although a movie is in perfect artistic synchronization among the plot, geometric capture composition, mood, music, etc., and has been thoroughly planned before the movie (storyboards, etc.), to a certain extent during shooting, and in post-production (where the applicable look is graded on what has been captured, even to the extent of shooting night scenes during the day), real-time producers must keep all technologies running. A live broadcast producer can only bring his talent and experience. For example, having watched and produced many football games before, he knows when the audience prefers to see the scoreboard rather than someone's boring commentary. But technically, the main camera feeds should be "just right", so that the director can rely on them when making a choice. Cameras can have some capture settings, such as knee points or black controls for sliders, and these are usually set to the same standard values so that, for example, the same-looking black is obtained. This can become apparent when cameras have different black performances, as you get milky-doped black. Thus, a standard video test signal (dyed with gray bars) can be used and adjusted slightly if needed. At his fast pace, if the colorimetry is truly wrong, the director can simply discard the feed, just like a cameraman who is still struggling to get the correct framing of a fast-moving action, for example, with a zoom. But he may also consider the feed important, and then you just get what you get, including any colorimetric artifacts. Therefore, things should preferably be standardized and simple, and technically, the main camera feeds should at least meet a minimum uniformity requirement (for example, if there is the same set of cameras, set all their controllable parameters to the same value for all cameras). Thus, any system that meets this scenario should be pragmatically simple and feasible enough.

[0007] For the increased complexity of free high-dynamic range production, compared to standard SDR production, one might expect a greater reliance on application-specific technical solutions. However, the complexity can also enable a unified approach (although typically customizable) for several methods. For example, any future improved system for consumer capture can have a similar need for relative simplicity of operation while strongly covering many different capture scenarios. High dynamic range creates new opportunities but also poses challenges. In fact, if one were to have (still not available) a camera with an infinite dynamic range, one could shoot in any setting and correct it all afterwards as one would expect. However, that complete freedom is not desired in all cases, as there may be situations where one might want to limit post-production or where important aspects have already been resolved during shooting.

[0008] If producing a movie consisting of several shots (e.g., day and night), the coordination of these shots can be done after capture (on the post), for example, a human color grader can change the final brightness in the typical master HDR image of the video (e.g., a maximum 1000 nits image) such that a dark scene looks dark enough after a previous day scene in the movie or, conversely, not too dark compared to an upcoming explosion scene, etc. We consider but are not limited to the selected maximum brightness of any (relevant) pixel in the image being the maximum brightness of that particular grading (also called the graded image) (1000 nits master grading) if one were to represent the scene using, for example, only one graded video. In principle, the grader can optimize the master image brightness of each pixel and any pixel in his color grading software (e.g., he can define the YCbCr color code of the pixels of an explosion such that the brightest pixel in the fireball is not higher than 600 nits even in a 1000 nits video maximum luminance (ML_V) master HDR video; the relationship between luminance and color code (such as a luminance-encoded luminance code) can be done by selecting the master EOTF (such as PQ), but in cases where the actual encoding details are possible variables and do not need to clarify the present invention, we will talk about pixel luminance). But this does not account for the Real worldThe relationship between the brightness of the fireballs therein. These brightnesses will depend not only on the heat generated in the explosion (which is usually not accurately achieved by the pyrotechnician), but importantly, the camera is not a luminance meter (absolutely; it is only a photon counter), so in addition to sensor physics, the amount of photoelectrons accumulated in the pixels corresponding to the brightest points of the fireball also depends on the aperture opening setting and the exposure time of the shutter (and possibly a neutral density filter in the optical path). This is to bridge the gap between the world of the scene (and the camera) and the world of the final display. What is captured is not the most important aspect, but what is to be seen when displayed (which is one of the reasons why some HDR encoding forms prefer to work with a display reference, even possibly inside the camera). In film production, the color grader can be the entity that spans the divide, i.e., selects the appropriate specifications for what should be precisely displayed for the captured image. It can be said that the display of the ultimately seen image is important, rather than the numbers of the pixel colors. However, opponents of the display reference method may argue that, on the one hand, it is uncertain how people will see the result, and the original image should be adhered to to a certain extent (e.g., this may have perfect future-proof quality). In addition, the applicant has particularly shown that for some ideally envisioned target displays, the requirements can be met by defining (at least one) graded image. It should also be noted that compared with the original camera, there is no real quality loss in at least the high maximum brightness master grading, and for most intents and purposes, it can form the stored original (since the original camera data is not very important for human consumption). We have found that even though the low-bit (8-bit) LDR corresponding grading is largely reversible, in any case, the display reference part is satisfied by the absolute nits HDR codec method of the secondary grading, such as the reference LDR grading corresponding to the primary HDR grading.

[0009] Grading (or color grading, although the main aspect of fine-tuning is the luminance or brightness of the color, the latter being a specific formula for the luminance channel of the video) involves, as needed, some human or automatic specification of the pixel luminance (or semi-autonomous combination) of various objects in the captured image along a selected luminance range. Some display scenario is required. For example, as Figure 6 shown, it may be desired to produce a 2000 nits white point luminance, i.e., the maximum brightness graded primary HDR video, as the output video (unique, or one of them) of one or more cameras recording the video program in the said scene. The original capture by the camera sensor is not grading because it is not optimized for human consumption on a display (e.g., an object in the shadow area may be darker than desired, whether in HDR grading or LDR grading). Although devices or users further downstream in the video communication chain can use all kinds of secondary grading (see Figure 10Brief Description), but the camera can output a (primary) graded version of the captured content (which can call the HDR master), but it may also be advantageous if it outputs a second (different) graded version of the captured video, e.g., usefully outputs an LDR video (which can call the LDR master). Some consumers of the content can use the first version, other consumers can use the second version, and other consumers can use both. Grading will assign various objects captured as digital numbers (digital numbers) from the camera's analog-to-digital converter (ADC206) to selected (visually pleasing) luminance values, where these objects are digital captures of the well-filled state of the corresponding sensor pixels by photoelectrons. The named numbers point to the fact that the number, as a relative indication of how bright an object is relative in the captured scene, is e.g., the 16-bit number 01101101111110000. For example, the ~5000 nits flame of an open fireplace in the scene can end up as the number 50000 (or normalized 50000 / 65536 = 0.76), but this will depend on the capture settings of the selected camera, e.g., aperture open, and the grader selects that their video looks good in a 2000 nits primary grading at, e.g., 500 nits (average pixel luminance of the image area of the burning object). Thus, if the end consumer purchases a TV with a display maximum luminance (ML_D) equal to 2000 nits (ML_M) value, then he will typically display all the luminances formulated in the video signal (i.e., in the primary HDR output image). That is, he will see the image almost exactly as the creator intended. If the TV only has the ability, e.g., ML_D = 700 nits, then it will be impossible to display the movie perfectly as expected, but the receiving-side device (such as a TV display) will degrade the 2000 nits primary HDR image by applying a display optimization mapping as described in previous patents, but this is not directly related to the current camera capture and production techniques described. The display optimization can be done roughly or attempt to maintain as much of the original expected appearance as possible within the limited luminance dynamic range capabilities. In this patent, we will refer to the image of the original numbers of the measured scene as "captured", to distinguish it from "image" (e.g., output image, such as from the camera to the OB truck, or the image to be broadcast, which can be defined differently, but can also be concisely defined similarly if designed to shift some of the color mathematics to the camera), and according to some reference (e.g., a 2000 nits target display), we typically consider grading it (i.e., having the optimal luminance) value, usually the luminance or actually the luminance value encoded for it. This common selection of reference makes it easier to coordinate the cameras, although still not trivial, but at least usually referable.It will be clear from the sentence that while "grading" refers respectively to the method of changing brightness, when it refers to a certain graded image (regardless of whether the grading is caused by mapping the original camera numbers during capture or by re-grading or further grading caused by an earlier generated grading), in fact, most of the time we will be talking about the resulting graded image (rather than the process).

[0010] In addition, even if one has the freedom to rely on perfect grading in post-production (i.e., any "errors" in pixel brightness or color can at least theoretically be corrected by a complete redefinition of these pixel color values during post-production grading), which is a freedom that real-time production (such as capturing a journey in France on a motorcycle) does not have, there are still challenges or opportunities where one would like to have more freedom in the original capture (exposure) of a single scene (this includes, for example, following someone through a journey in a factory; or a bike ride through a tunnel). In fact, one may want to combine freedom with consistency to obtain a certain consistent freedom. This may become important in the future, where not only different types of video production will emerge or continue to become more popular, but also new technologies (e.g., related to the capture of different views or aspects of a scene) will emerge or continue to become more popular.

[0011] In addition, in some cases, one may directly expect to produce several videos for different dynamic ranges of capture (e.g., an HDR version and a corresponding LDR version to be offloaded from the camera to a certain distribution network or memory). For example, the LDR version can be broadcast directly to customers (e.g., via a cable TV or satellite distribution system), and the HDR version can be stored in the cloud for subsequent use (e.g., future pay-per-view), which can be a replay or video clip reuse ten years later.

[0012] Several types of video production can benefit from the following new insights and embodiments. Classic movie or TV show production can shoot the same scene from different angles continuously, again and again (e.g., for a combat scene, once from behind the attacker, so we look down at the victim on the ground, and once laterally from a direction close to the ground), and for such applications, the innovations presented below can provide benefits for increasing production speed, simplifying capture, more simply implementing more complex capture situations, or increasing or relaxing post-processing possibilities, and in some cases, one may want to capture an entire plot through a complexly illuminated scene in one or several coordinated shots. We do see such an expectation that not only professional programs will no longer be shot only from a few static TV cameras placed around the scene, but even semi-professional company communication videos (e.g., private company network captures of reports of employee access from another company or hospital to a business unit) may want to move from a static presenter to someone who moves around, leaves the presentation room and walks into the corridor, and may even get into his car and continue presenting while driving. Especially when technology can help take over some complex work, "amateurs" can also start producing more professional videos. When making any capture "as is", i.e., with fixed exposure settings or relying on any automatic exposure done by the camera (suffering any consequences, such as the usual clipping, and incorrect colors in areas, e.g., too dark and severely invisible parts, or just worse than the best), this is not difficult, but it is quite challenging for high-quality HDR production, especially considering that the technology is relatively new and many are still on the basis of optimization. In fact, technology can help avoid it from sometimes getting a bit messy.

[0013] Figure 1 A shows an illustrative example of a typical non-trivial dynamic capture, with indoor and outdoor captures. It is known that even with a classic first-level exposure (i.e., involving some overall measurement of the illumination level of the current scene, and for filling the pixel wells up to a certain level, corresponding values for the camera settings, aperture, etc. are required, i.e., resulting in setting such values for further capture), for example, when near a window where the light level drops with the square of the distance, the exposure of the speaker's face may overexpose the objects near the window. This is of course a common problem in low dynamic range LDR (also called standard DR) capture, and of course when there are sunbeams, some pixels may clip to the maximum capturable level (white, e.g., brightness code 255).

[0014] We want the photographer to be able to freely walk from the outdoor area (101) to the indoor area (103), possibly through a corridor (102), in which there may be some lights (109) to produce a gradual change in local illumination (illuminance). In the sense of a "good" capture of all information, it is not difficult to have such a large capture dynamic range in a high-dynamic-range camera. That is, a camera with, for example, 14 stops or more or a 16,000:1 contrast (e.g., 100,000 photoelectrons full well and 5 electrons noise); with today's sensor designs, one can speak of an "equivalent full well", and it seems feasible to capture a dynamic range higher than 2 million to 1 for mainstream cameras in the not-too-distant future. Since it can be much darker than a bad camera (with good capture accuracy), one can, for example, only focus on fully capturing the brightest objects without color channel clipping or desaturation. If the brightest image white is (directly) mapped to the maximum displayable white of, for example, a 450-nit display, this may result in an unsatisfactory picture. However, at least in principle, by grading (since the information is still in the darker pixel colors with not much degradation, especially if captured without noise), it can always be brightened later. Note that for very challenging HDR scenes / environments, 14-stop capture may still require care (e.g., if one does not want excessive exposure noise in the dark corners, one should not expose the filament of a bulb that may exceed one million nits).

[0015] Even though the exposure of a high-dynamic-range camera ( / video usage chain) may not be too much of a problem since one can record objects as a relatively dark percentage of white and then brighten these objects according to the expectations of post-capture image processing, the issue of correct exposure arises again when a (usually) secondary image with a lower dynamic range is desired in the case of HDR capture. Then we need to compress the HDR scene within the low available dynamic range, and this requires more reflection (which can be partly regarded as virtual re-exposure). However, digitally there are more possibilities than simple control (with, for example, aperture opening). Nevertheless, creating a good LDR image for a challenging HDR scene or more specifically its main HDR grading is not necessarily easy.

[0016] Typically, the aperture or exposure control can be done manually or automatically. For example, as shown in EP0152698, the analysis of the captured image can include determining the maximum values captured for the red, green, and blue components and setting the aperture and / or shutter (which can also cooperate with neutral density filter selection and electronic gain) such that this maximum value is not clipped (or not clipped too much).

[0017] Popular control methods use the average of the scene brightness (or relative photon collection), at least in SDR capture scenarios (or more precisely, some intelligent averaging algorithms, such as giving less weight to bright sky pixels in the summation) as a very reasonable metric. This also particularly involves how to directly display the captured SDR brightness in the SDR display scenario: the brightest brightness in the image that is used as the maximum control range value of the display drive signal (e.g., 255), which usually drives the display so that it shows its maximum producible output (e.g., for a fixed-backlight LCD, the LCD pixels are driven to maximum transparency), and the displayed color visually appears white. A reasonable assumption of the "display as painting" method is that under a single illumination (i.e., fairly uniform, e.g., by using controlled base illumination, such as a matrix of lights on the ceiling in television studio production), a diffuse object in the physical world will reflect approximately 1% to 95% of the current illumination level. Specular reflections can be considered just white. This will form a histogram of brightness distributed around the approximately 25% level (or brightness distributed around half of the brightness code), and for the human eye, when displayed on a 200 nits ML_D display, i.e., an image where white pixels have a display brightness of 200 nits, it will look almost the same as on a 100 nits display, just like a painting that is mainly of interest due to the color of the object being illuminated by stronger or weaker lights, because the eye compensates for the brightness difference, and the brain only cares about seeing the relative differences of the object points separately painted on the LDR display. Then, it also makes sense to relatively characterize the scene brightness range by mapping its average value to what value the camera sensor pixels are filled to ~25%, and even for an LDR sensor, this will be a good sampling of the available information in such a scene. However, even in the SDR imaging chain, i.e., in the case where an SDR camera captures and produces an SDR image as output and is displayed on a traditional SDR display, the average is not such a good metric for, for example, bimodal scenarios (e.g., indoors and the outside world through a window) or very narrow peak scenarios (e.g., a coal mine that only contains a few black shadows). However, in the case of a little fine-tuning and / or grading by the camera operator, SDR seems to have worked impressively well in, for example, a century of practice, showing home viewers everything from the coronation of the queen to the depths of the ocean. There are only some urgent needs to promote the move to HDR, such as capturing and presenting objects brighter than white, and ideally there is also better handling of black, and a more professional control image description framework building layer starting from a common base, such as the target display associated with the HDR image, which has a maximum brightness, e.g., 2000 nits (ML_V of the video image).

[0018] However, this way of average-guided exposure is a widely deployed (in fact, ad-hoc / de-facto standardized) video creation practice from the SDR era, which is an example of many technical methods that warrant a complete rethinking and redefinition of methods in the HDR ecosystem. In the chicken-and-egg problem, most of the earliest HDR technologies or at least the standardized HDR technologies additionally have the technical ability to focus on the definition and encoding of HDR images for brighter displays (e.g., via Hybrid Log-Gamma or SL-HDR). There is no need to develop how the creation of images can be combined with this new encoding and display ability. One way to look at things could be to simply regard the codec as something that must be able to record everything that is produced, just work as a "translator", and keep producing as usual. Another way to look at things is that people might want to make good use of the enhanced ability, for example, just like a 3D movie with something thrown towards the viewer's head, now well-coordinated HDR effects such as powerful bright color explosions, etc. A third way to look at things is that people might want some additional creation techniques to control the huge new dynamic range ability that people have so that things don't look out of place. This seems to be the basic principle behind the 1000 nits bridging point in content creation, which attempts to combine the relative HDR paradigm of HLG with the absolute paradigms like Dolby Vision or SL-HDR.

[0019] US2017 / 0180759 is an example of a general system for obtaining a master HDR video to a receiver (e.g., end consumer) via some transmission method. It enables representing a raw master graded video with a target display maximum brightness of 5000 nits (also called white point brightness) as a proxy video for communication, which has a target display maximum brightness of, for example, only 2000 nits (i.e., pixel brightness with a brightness of at most 2000 nits). On the receiving side, the video of TDML can then be made lower than 2000 nits, e.g., 400 nits, but the original master HDR video (called intermediate dynamic range video) can also be reconstructed from the received proxy, or the output video can be made even brighter.

[0020] For the reader to quickly recapture the various parts of the video communication or usage chain, which should not be confused, we have outlined the general case (to represent various cases, consumer products of course have some components that are at most just residually present) for illustration in Figure 10 below.

[0021] Exactly what happens can depend on whether we have live production, such as for example a sports event or news coverage, or whether we have a movie that is shot over multiple days, then edited together and then distributed, but generally some components will be present, at least in relation to the present invention. In the shooting location 3000, there may be a first actor 3001 under controlled lighting 3003 (e.g., suspended from a crane), and a second actor 3002 in a dark area of the scene (this can be a constructed scene, or a natural scene found in place). There can be one or more cameras. In this example, the first camera 3004 is mounted on a person, for example as a Steadicam, and the second camera 3005 is mounted at an appropriate position on a tripod at least temporarily statically. As video compression gets better and the internet becomes more widely available, even the raw footage (e.g., rushes) can be transmitted to the production studio 3010, which no longer needs to be near the footage (e.g., an Internet Protocol Video Communication connection 3009 can be employed). We assume that some "director" (either in real-time or offline) is viewing the available raw camera shots. In the case of classic LDR shooting mainly regarding the timing, logical placement, etc. of the shoot, now synthetic decisions can also be made regarding the grading of various objects in the shoot of one or more scenes. In currently relatively simple and relatively new HDR products, for example, to enhance LDR using a fixed LUT associated with HDR, this may just come directly from the camera, perhaps with very little adjustment style. For this discussion, we assume that there will be someone (or an automaton) playing the role of a "grading machine", and that role will have a say in the brightness and color of the objects in the image (leaving the pace of the story to others, such as the director and / or editor). For example, generally different graded versions of the footage (or a part of the movie or a program, etc.) with different TDML can be viewed simultaneously on a first reference display 3011 and a second reference display 3012. The grading changes can be implemented via a color console 3013. It can also be mixed with graphic elements (e.g., robots or monsters) via a graphics unit 3015, which generally can have a connection 3016 to a graphics vendor (e.g., again via the internet) (these techniques are known and do not require a more in-depth elaboration in this patent application). The final product (the cut) is, for example, a 5000-nit future-proof master HDR grading M_5000. It can be stored on a memory 3018 for later use and sent to a certain content broadcaster 3020 (e.g., the British Broadcasting Corporation) via a certain distribution medium 3019 (e.g., again the internet). The broadcaster can distribute the video to end customers via various communication channels.For example, (and we omit various possible intermediate units, such as a wired headend) via a satellite, a first intermediate version (proxy) IM_2000 for transmission (with a TDML of 2000 nits) can be made by a 5000-nit master device, such as a device of a broadcaster, and transmitted via a television satellite 3021 to a satellite antenna 3050 connected to a satellite TV set-top box 3051. We assume that the STB is responsible for calculating a display-adapted version (IMDA_550) of the received video, which is coordinated with the end-user display 3052 and transmitted, for example, via an HDMI cable. It is typical today that the broadcaster also makes at least some of its content available via a web portal. For example, a secondary proxy video (IM_1000, with only 1000 nits TDML) can be transmitted via, for example, a content delivery network 3022, and the final consumer can access this version, for example, on his mobile phone 3055. Clearly, many different setups can cooperate with and benefit from the following innovative embodiments. In fact, video delivery is becoming more hybrid ("standard"), for example, IP packets can be delivered via classical communication systems such as wire or satellite, but also via telecommunication standards such as 5G, so the distinction between live "broadcasts", VOD, user-generated content, etc. is disappearing to some extent. Ergo, the video production examples are also only intended for conceptual illustration and are not intended to be limiting in any way.

[0022] Note that the video product and communication can involve various HDR images and LDR images. High dynamic range images are generally understood to have more dynamic range compared to the current state (well understood by those skilled in the art of video and television technology), low dynamic range (also known as, standard dynamic range). These images will have the ability to have a luminance range sufficient to go from deep black to Lambertian-reflecting white under uniform illumination (e.g., a white piece of paper). The darkness of black depends on various technical attributes of capture (e.g., camera noise) or display (e.g., ambient light reflected on the display screen). There are no significantly brighter pixels than white. If luminance values in nits are associated with pixel luminance (as in the more advanced HDR systems of the present invention), then LDR images will be characterized as having a typical white luminance of 100 nits (i.e., an associated TDML of 100 nits). A good minimum black for LDR will be taken as 0.1 nits. Since several HDR encodings will not desire deeper blacks, HDR images or videos can be defined according to the SDR state of the art as having the possibility of encoding brighter pixel luminance, typically at least twice as bright, i.e., a TDML of 200 nits or higher. The camera will have means to accurately capture those scene luminances above Lambertian-white (e.g., multiple pixels for different exposures, or multiple exposures of different lengths, or LOFIC and dual-gain conversion pixels, etc.), and the display will have means to display the extra bright pixels (e.g., a 2D LED backlight matrix with individually intensity controllable, where the normal luminance part of the display will receive local LED illumination such that a luminance of 100 nits or less is displayed for those pixels, and the LEDs for self-luminous objects in the image, for example, will be driven, say, 10x brighter such that those pixels are displayed as 1000 nits on the display screen).

[0023] US2017 / 0180759 relates to creating various secondary HDR videos starting from a master HDR video that has already been produced (useful especially for the various technical constraints of various communication systems), but it does not teach details about how to produce various grades (at least the master HDR graded video) from camera capture, and of course does not teach how to coordinate various shots at different locations with different local illuminations of the said environment in one or more cameras to easily achieve a specific grade as output (whether directly for, e.g., a consumer display, or for further processing, e.g., further re-grading, storage, mixing with other content, etc.).

[0024] Given the complexity of HDR video technology (and the fact that it is only a recently emerging technical field), there is a need for a practical and fast, mostly automated process for well-exposed main grading, and typically also a secondary grading for in-running dynamic lighting environment capture for one or more cameras, such that the graded output video sequence automatically exits the camera, enabling the camera operator and possibly the director to focus on artistic aspects (such as geometric composition) or story line aspects (such as action), such as from which angle to film an actor. SUMMARY OF THE INVENTION

[0025] The above need is met by a method for controlling a video camera (201), the method comprising setting a video camera capture mode in one or more graded output video sequences of an image, the video camera capture mode specifying the output pixel luminance for a corresponding position in the scene to be captured, the graded output video sequences being characterized as having different maximum pixel luminances, the method comprising:

[0026] - The operator of the video camera moves to at least two positions (Pos1, Pos2) in the scene having different illuminations relative to each other, and captures at least one high dynamic range image (o_ImHDR) for each of the at least two positions of the scene;

[0027] - A color composition guide analyzes the at least one high dynamic range image captured for each of the at least two positions to determine a corresponding maximum luminance region, and determines at least one of an aperture setting, a shutter time setting, and an analog gain setting of the camera based on the maximum luminance;

[0028] - For each of the at least two positions (Pos1, Pos2), a corresponding main capture (RW) is captured using at least one of the determined aperture setting, shutter time setting, and analog gain setting of the camera, and the determined aperture setting, shutter time setting, and analog gain setting are stored for subsequent image capture when the video camera resides at the corresponding position;

[0029] - Determining at least a corresponding first graded image (ODR) for the corresponding main capture, which includes determining an adjustable luminance allocation function (FL_M) for the corresponding position, which maps the numbers of the main capture to nit values of the first graded image (ODR); and

[0030] - Storing the corresponding luminance allocation functions (Fs1, Fs2) for the at least two positions or the parameters uniquely defining these functions in corresponding memory locations (221, 222) of the camera.

[0031] The performance mode of a camera is the way it captures video and, more precisely in this specification, how it outputs the pixels of the output image corresponding to the points imaged by the camera lens onto the image sensor in a scene with a specific brightness. The camera can output various versions of this video, i.e., variously graded videos (i.e., different gradings) with different maximum luminances (ML_V) (e.g., 3000 nits and 150 nits). The mode (and its characteristics) is set by: (usually guided by color composition, either manually or automatically) determining at least the luminance assignment function for each position to map the captured numbers to the luminance graded according to the expectation for that position, and generally coordinating the mappings of the individual positions to a common range that ends with the selected common maximum luminance (ML_V). That is, after at least a set of capture settings (aperture, shutter speed, etc.) has been determined or is suitable for good capture of the individual positions in the environment where the shooting will take place (except for potential clipping of very high scene luminances above a certain selectable maximum, which can be done iteratively by looking at which clipping occurs in the output images of each position). The maximum luminance captured as it is usually can be less than the final maximum luminance in the scene, but generally only a few pixels will be allowed to be clipped, or at least which clipping is less important, e.g., for a subsequent program or story. The region can be as small as a single pixel, e.g., selected by clicking on it guided by color composition (in any captured image during the initial environment discovery phase until appropriate values of the basic capture parameters and functions have been established). For example, another function that makes the darkest scene element in at least one output graded video darker than the first test function corresponds to another mode (of the camera's colorimetric performance, i.e., specifying the output pixel luminance). Thus, the performance mode is determined by functions at various positions at least. The color composition guidance knows how the maximum luminance will change in the maximum luminance region. For example, when the aperture is closed by one stop, it will linearly decrease by a factor of 2 in numbers and decrease by a certain amount (the exact value is not critical as only positioning at or near the absolute maximum of the output range is required) in the output grading depending on the initially set mapping function (e.g., the default mapping function or the mapping function loaded from a previous operation in the processor). If the aperture is opened, the maximum value will be clipped even higher, which can be seen because the adjacent slightly lower luminances will also start to be clipped. Then the value of the aperture (and possibly other parameters, i.e., all determining the exposure of the sensor pixels) can be selected as the value that satisfies just enough clipping of the imaging pixels, as selected for that shot given the capabilities of the camera (e.g., the noise performance for black may also be seen), but no more.Main capture is the digit output by the ADC when capturing using capture parameters (which has just been established and is now set to be fixed for any later capture at least for that corresponding position (and could be the second position if they also work well by faithfully capturing substantially all scene objects there)). It will serve as a stable starting point, which is very important for optimizing the correct grading functions for various positions thereafter. These functions are shape-adjustable (i.e., they can, for example, brighten the darkest pixels of the scene and darken the brightest pixels), and if the shape to be used produces at least one graded output video with an acceptable appearance according to the color composition guidance, then that shape is sufficient. If a sub-range of luminance is, for example, too dark, the color composition indicator can locally increase the function in that sub-range compared to the current function shape (such that it produces a higher output).

[0032] The video camera has roughly two internal technical processes. The first (which is actually the optimal sampling of the optical signal representing the physical world) is the correct recording and usually linear quantization of the color of a small part of the scene imaged by the lens onto typically four sub-pixels (e.g., based on red, green / / green-blue Bayer or cyan-magenta-yellow sampling) on the sensor (204). The controllable opening area of the aperture (202) and the shutter (203) (and possibly neutral density filters) determine how many photons flow into the wells of each pixel (linear multiplication of the local scene object luminance), such that one can control these settings so that, for example, the darkest pixel in the scene lies above the noise floor, e.g., a measurement result of 20 photo-electrons above 10 photon noise (and noise of ±X photons on the value 20), and the brightest object in the scene fills the pixels to, for example, 95% (i.e., 95% of what the pixel well can measure and the ADC will represent as the so-called digital DN). Using analog gain (205), one can imagine more photons entering by increasing the voltage representing, for example, 25% pixel fill to the level of 50% (similarly in the digital domain, it can be amplified, but we will consider under all image improvement processes in a more general class, not necessarily multiplicative linear brightening) as if the scene were brighter, as one might also want to choose the aperture and shutter for other visual attributes of the captured image such as depth of field and motion blur). The analog-to-digital converter (206) represents the spatial signal (e.g., the image captured by the red pixels) as a matrix of numbers. Assuming we have a sensor of good quality with a good ADC, for example, a 16-bit digital representation will give values between 0 and 65535. For various reasons, these numbers are not fully usable, especially in systems that require typical video, such as Rec.709 SDR video. Thus, the second internal process, the image processing circuit (207) can perform all the various required transformations in the digital domain. For example, it can transform the numbers by applying an approximately square-root-shaped OETF (Opto-Electronic Transfer Function) to finally obtain the Y’CbCr color coding of the pixels in the output image. Y’ is the luminance representing the pixel luminance, and Cb and Cr are the chrominance, also known as the chroma representing color, i.e., hue and saturation (for HDR, these can be, for example, non-linear components defined by the OETF version of the perceptual quantizer function standardized in SMPTE ST.2084).

[0033] The image processing circuit (207) (e.g., including a color pixel processing pipeline with configurable processing of incoming pixel color triples) will operate in a novel manner in the technical insights, aspects, and embodiments we describe below, as it can apply, for example, a configurable function to the luminance or lightness component of the pixel color to obtain a primary graded image (ImHDR) and / or a secondary graded image (ImRDR), such as a standard dynamic range (SDR) image (and thus output at least one graded image having reasonable image object pixel luminance or luminance percentage as output). Typically, these can be recorded in the camera internal memory 208. The camera may also have a communication circuit (218) that can output the image, for example, via a cable (SDI, USB, etc.), wifi, 5G, etc. (possibly by connecting the camera to an additional device). For the communication system (209) that the camera can use, for example, we will envision an Internet Protocol communication system for future-oriented professional cameras. Additionally, lay consumers taking pictures with a camera embodied as a mobile phone can directly upload to the cloud using IP over 5G, but of course there are many other possible communication systems, and this is not the core of our current technical contribution.

[0034] A simple configuration of a simple camera or a more general camera can, for example, provide a single grading as the output image (e.g., a 1000 nits ML_V HDR image with appropriately assigned brightness for any specific scene, such as a dark room with a small window to the outside, or a dim open-air market with sunlight falling on an object through a crack in the roof). That is, they produce an output video sequence of temporally continuous images, but usually with different brightness assignments for scene regions of various different illuminations, corresponding to the basic capture as at least one cameraman walks through the scene while capturing an action or other scene content. Thus, on the one hand, most of the information of the scene should somehow be present in the encoding of the video (i.e., the brightness of various scene objects in the represented output video; this is the "technical representation standard"), where this may involve, for example, a certain degree of clipping of the brightest sun-reflecting clouds adjacent to the sun, and on the other hand, preferably, not only encoding the brightness of these objects in a way that gives an apparently incorrect appearance brightness (i.e., when displayed), but also having a brightness with the correct appearance (even if one may still want to further adjust this first assignment of pixel brightness, for example, according to artistic preferences to give the final look of a movie scene). An example where the initial (e.g., automatic) grading and the artistic final grading may deviate is, for example, a high-key look with clipping. Creators usually do not intentionally do this in camera captures, but can clip some colors in the brighter half of the face illumination and boost the remaining colors to the upper region of the RGB color gamut, resulting in a bright, low-saturation color result. The display pixel brightness prepared for a high TDML grading (e.g., with ML_V = 5000 nits) should generally not be approximately 5000 nits, as this would make a person's face look like a light source. For this final master HDR grading (i.e., for a movie for release), one may want the clipped colors of the face to be at a level of, for example, 1000 nits. All of these can also be directly implemented from the camera by defining an appropriate mapping function, as Figure 6 shown.

[0035] The idea is to establish this main grading based on a well-configured capture from the sensor, i.e., most of the objects (and the brighter objects) in the scene are well represented with an accurate spread of pixel colors (e.g., different bright gray values of sunlit clouds).

[0036] Thus, the basic configuration is the quick or precise determination of the camera's basic capture settings (aperture setting, shutter time setting, analog gain setting that may be greater than 1.0). Then, any further graded determination can be stably established on these digital captures. In fact, the camera doesn't even need to output the raw capture, but only outputs a primarily graded 1000 nits (main) HDR image, for example. As mentioned above, the capture itself is only a technical image and not very useful for humans (psychologically all wrong luminances), so it doesn't need to be determined anywhere, but rather whether the main luminance assignment function is reversible and jointly stored, and in this case jointly output.

[0037] The optimal setting of the basic capture settings is done by a human (e.g., a color composition director, or the camera operator who can take on the role of the color composition director during this initialization phase if he is, for example, a consumer, or the only technical person in a 2-person off-site production team, with the others being the host), although this can also be determined automatically (i.e., by some firmware, for example).

[0038] Figure 6Shows an example of a user interface representation, which can be shown on some displays (depending on the application and / or embodiment, the display can reside in an OB van, or "at home", e.g., in the studio of a broadcaster or producer in a production using REMI (Remote Integration Model), or e.g., a computer at a location on a set of light overlays interacting with a camera, and a single production team (i.e., the camera operator) can perform his visual inspection to better control the (one or more) cameras, or it can be attached to the camera itself, e.g., a viewer). In principle, only the camera operator needs to be on-site (e.g., for producing a semi-professional high school program). Also shown is a graphical view for determining a mapping function (between various representations of pixel luminance), but in one of the views, the image view (610) shows the captured image (possibly mapped with some function to create the appearance for the display being used, thus creating a basic impression of various scene objects and at least good visibility, as accurate colorimetry is not always available in any application required to determine all the various settings). Assuming it is a touch screen (a person skilled in the art can understand from our basic example how to make an equivalent version by using, e.g., a mouse-like sensor), the human controller (i.e., the role of the color composition guidance named in our claims) can use his finger 611 to click on an object of interest (OOI). For example, a double click indicates (quickly) that the user wants these (bright) colors to be all well captured in the sensor, i.e., the pixel wells of at least one of the three color components overflow. This would mean that the flame is always well represented at least substantially according to the technical representation standard, and there would be no uncured color errors. Then, an image analysis software program interacting with the user interface software (e.g., running on a computer in an OB (Outside Broadcast) van) can determine which colors are in the area where the flame of an open fireplace is located. That is, the camera operator (collaborating with the color composition guidance) has captured a first raw captured or exported graded HDR image (or any exported grading of the display whose settings are determined by the color composition guidance). In the wording of this teaching, he has just determined the (at least one) original high dynamic range image (o_imHDR). Note that the original HDR image does not need to be, e.g., digital from an ADC (or actually a linear rescaling of those), and usually is not, because if some function within the range of some represented values is used to map those DNs, e.g., mapping the highest possible DN to the highest luminance code (not necessarily a power(2;N)-1, where N is the amount of bits representing luminance, e.g., if some luminance codes are reserved for management purposes, such as a timing code), then it can also be seen which objects are well captured from this secondary image representation).For example, all patches of the flame have a pixel value (narrow range maximum luminance) of 940 with no variation, which may indicate that the aperture must be closed to let less light in, reducing the values of all numbers in o_imHDR, making all spatial pattern details in the flame visible. Depending on whether the color composition guidance wants to see those flame details, the corresponding aperture setting will become the final setting (e.g., sometimes one may want to open the aperture more to not have too much capture noise issue at the lowest end of the basic capture, and the resulting o_imHDR). When the color composition indicator is clicked or tapped at at least one location, the software can check whether the colors have been well represented in that capture. For example, all pixels of this flame, whose red and green components fall between luminance 990 and 1020, and the blue component is low for a well-captured bright yellow. This has been used as a (graded or coded) representation of a well-captured image (e.g., the software in an OB van can also receive the capture and check whether all numbers are below the power (2; number of ADC bits)). If there is clearly no higher, even if roughly close to the maximum value, this may be a good capture, unless the maximum pixel luminance in o_imHDR is far from the absolute maximum and cannot be exceeded, in which case, it can be checked again by, for example, opening the aperture by 1 / 3 stop. On the one hand, as expected by humans, the pixels of this flame are not clipped, and on the other hand, the capture looks good because the flame is a bright scene object, so we do not capture too few photons of other objects in the scene, which may leave darker objects with too much noise in the scene capture / image. Note that the sun and some sunlit clouds may be brighter than the flame, but the human interaction UI specifically allows choosing those as non-maximum values to be captured, i.e., they may be severely captured and clipped (see. Figure 3the RW range). If the first original HDR image and the settings for the first original HDR image are not yet accurate, at least one additional original HDR image can be taken. This can be done either automatically by the software or advantageously under the guidance of a person. For example, if the software knows that the tap indicates selecting the image near the top of the image color gamut and if its color components (at least the maximum color component) are below the value corresponding to half-pixel filling, the software can, for example, choose to double the shutter time, also known as one stop (assuming it is still possible given the required image repetition rate of the given camera). If there is clipping, a secondary image can be taken using, for example, 0.75% of the previous exposure. Finally, if the original image is captured where the (selected) brightest object is indeed captured as having at least one color sub-pixel close to overflow, i.e., any corresponding image pixel that is close but not yet clipped (whether in DN, lightness, or luminance when the full luminance range has been associated with the corresponding lightness range of DN), the HDR capture situation is considered optimal, and the optimal values of the basic capture settings are loaded into the camera (or the main camera responsible for initializing the system color composition in the case of a multi-camera system). This is for that location in the scene. In principle, these settings may only be valid for that location in the scene. There may be scenarios where a set of capture values (i.e., one or more of aperture, shutter time, etc.) can be selected for all locations where the shot is to be taken (if there is not much dynamic range in the scene compared to the camera capabilities, or when using a camera with a very high sensor dynamic range), but in other cases, it may be desirable to coordinate the various capture settings, preferably using different optimal capture settings for various locations rather than sub-optimizing for the darkest or brightest scene colors. For example, if a criminal hidden in the darkest shadows needs to be captured well, some of the flames are clipped (remembering that the values he captures may need to be brightened through subsequent processing). However, if feasible, it may be useful if at least these basic capture settings are the same for the entire shot, i.e., all locations along the scene that are significantly differently illuminated (this is usually not true for the luminance assignment / remapping function, at least for those of the determined secondary lower dynamic range gradings). Thus, there may be a stage for merging the basic capture settings. For example, if for the first part of the scene, 1 / 100s can be a good setting for the shutter, and for another area with brighter objects, 1 / 200s is determined, the camera can load 1 / 200s for the entire capture at each location in the scene (if there will not be too much objectionable noise, this situation is optimized for the brightest desired object in the overall of all the shot locations).This is because people then determine the brightest object in the overall shooting environment that is over-expected: a shorter shutter time will reduce all the captured numbers, and depending on the distribution of brightness and / or lightness, if it is in different possible non-linear ways, these are also those values, but due to the dynamic range faithfully captured by a high-quality HDR camera, this is not a problem (compared to having the best possible capture for each individual position, ease of operation may be a more preferred attribute, and for many uses in many cases, the capture may be too high anyway). The optimization of the graded version of this capture will lie in the optimization of the mapping function.

[0039] We will (as explained, without wishing to be restrictive) assume in the exposition that after initialization (so that we can focus on the preset of the coordinated mapping function), that is, after the camera operator walks to at least various challenging lighting positions and applies the principles of the technology to discover the scene, a single set of optimal values for the basic capture settings is determined, but the skilled person can also understand how there can be several sets of basic capture settings for the corresponding various positions, and how these sets can be loaded when the camera operator walks and starts shooting at the corresponding positions during the actual video capture of, for example, a movie, just as the function for mapping brightness to obtain various (at least one) output gradings will be loaded to be calculated according to its corresponding memory location. Thus, when walking to (or starting to shoot in) a slightly darker room at the overall shooting position, the aperture will quickly be reset to its determined value for that position, etc.

[0040] Different illuminations include the following. It generally starts with the degree to which at least one light source illuminates the scene objects and gives them some brightness values. For example, for outdoor shooting, the contribution of the sun is greater, while the contribution of the sky is smaller, and these can give the illumination levels of all the diffused objects under the sun (whose brightness then depends on the reflectivity of the object, such as black or white). In an indoor room, the location-dependent illumination will depend on how many lights there are and where they are located, the orientation (lighting devices), etc. But for HDR capture, the local illumination or rather, the determination of the lighting situation should also include "outliers". As explained, for example, when determining the basic capture settings, all or most of the pixels of a bright flame or other bright object of thousands of nits should be considered, but especially for large bright areas, such as outdoor sunlight illumination seen through a window, it may be desirable to be captured at least after clipping, and may even be captured at a lower sensor pixel filling percentage. Small specular reflection spots (e.g., a light bulb reflected on metal) can be clipped in the capture. For simplicity, we illustrate as if the location is important, but the skilled person understands that in more specialized embodiments, one can also consider the orientation of the camera. As Figure 4As shown, if the first camera 401 is pointed at an angle towards the interior of the room (in this example, the brightest object is the flame 420, but it could also be an object that is much darker than outdoor objects), the first camera 401 in the first position can see a different color / brightness composition, whereas if it points forward, it will see the outdoor world through the window (410). The light bulb 411 is typically a small object, which may also be trimmed in any image (and sensor capture). This would also be the case even for the large oval light 421, although for such a large light, it would be good if it still had some varying gray-scale values from the outside to the middle, at least for the highest dynamic range graded image output of the camera (so that we don't have ugly "holes" or "spots" in the image). An acceptable decision would be not to trim all the pixels in the oval, but for example the brightest 10% in the center, so that the rest of the light still shows some grading (even though the viewer typically doesn't focus on the light, but on the action in the shot, having this information in at least one captured graded image could be good, for example for later image processing). If the camera (system) has been set up according to the described process, normal objects with "medium" lighting, such as the kitchen 412 or the portrait 422 or the plant 423, will be automatically acceptable in the basic capture (those objects, such as the portrait, may become more critical in the primary and secondary grading as the output of the (one or more) cameras). This means that they may be acceptable in the technical representation sense, because a reasonable subset of the code represents the various object colors (since the camera captures linearly, i.e., it doesn't perform numerically worse for any higher brightness range than for the darkest color range), but not necessarily in the visual sense (for the viewer's visual impression), because any "simple" assignment may not produce the desired in-object contrast in all the derived graded images. A more critical object to be checked by a human operator (or automatically) is the black poker 424 in the shadow area of the room (where the light from the oval light is blocked by the fireplace and there is also direct illumination from the flame). This capture may be too noisy, in which case one might decide to open the aperture and shutter more, and may lose the gradient in the oval light, but at least have a better quality capture of the poker (which would require brightening processing, for example when an SDR output video is desired).

[0041] The main capture (RW) will be the image with the correct basic capture settings (the last one in at least one high dynamic range image (o-ImHDR) has led to such settings). Based on this image of the current scene position and / or orientation, at least one grading will be determined, for example, to produce an HDR video image as output, rather than just a scaled (by linear multiplication) copy of the capture, but usually with better luminance positions (values) along the luminance range of at least one scene area (e.g., slightly dimming the brightest object, or placing important objects at a fixed level, such as 200 nits, or brightening the darkest captured numbers with pure scaling (such as mapping the maximum number to the maximum image luminance), and the numbers slightly correspond to the values they will have, and linearly brightening each lower value). Usually, one will decide the best representative (e.g., middle sub-range) values of multiple objects or regions (at least important objects or regions (portraits, fireplaces, kitchen sinks, etc.)), so that one obtains the shape of the function due to mapping a sub-range of input values to a sub-range of output values (e.g., luminance to luminance, or correspondingly lightness to lightness).

[0042] The graded image usually does not have an object luminance pixel ratio equal to its corresponding digital ratio (i.e., for each selected set of two pixels from the image L1 / L2 = DN1 / DN2, even after subtracting some black offset to compensate for different starting points), but there are different schools of thought or application expectations that the present technical solution must be able to meet.

[0043] The first application creates a main HDR grading that only slightly remaps the luminance positions of the numbers obtained by simple maximum-to-maximum scaling (i.e., mapping the ADC maximum or any first range maximum to the second range maximum, such as 1000 nits). For this redistribution / remapping from pure scaling, a simple shape function, such as a power function, can be used for the color composition guidance, and he may still want to adjust the power value. For some scenes, this is considered sufficient (at least for one of the possible graded image outputs).

[0044] However, we will clarify typical simple grading control examples (non-limitingly) to quickly establish the luminance mapping function of the main HDR grading: Fs1 and Fs2 are for indoor and outdoor positions respectively, i.e., two typical exemplary positions (assuming the basic capture settings are determined to be the same for all positions, such as when the elliptical lamp just starts to trim to the sensor and the ADC maximum). For a really good grading, when receiving the HDR output image defined by this 2000 nits ML_V ( Figure 7 (non-limitingly) to quickly establish the luminance mapping function of the main HDR grading: Fs1 and Fs2 are for indoor and outdoor positions respectively, i.e., two typical exemplary positions (assuming the basic capture settings are determined to be the same for all positions, such as when the elliptical lamp just starts to trim to the sensor and the ADC maximum). For a really good grading, when receiving the HDR output image defined by this 2000 nits ML_V Figure 2ImHDR; In the first part of the scene discovery phase, the image may already have been used as the image o_imHDR to first establish the basis for the correct camera capture settings, but now one or more HDR images are being generated for final grading, where for each position, given the camera shot under the determined capture settings, there is an optimal mapping function), the color composition guidance will relatively accurately want to see all the brightnesses displayed on a 2000 nits display in a typical viewing environment. Note that if only the primary SDR grading is output, especially one in a higher word length, such as 10-bit or other three-color component representations for lightness and chroma, the same principle can generally be applied, but then various sub-ranges for various scene image regions will have been mapped to different relative positions (e.g., brighter dark pixels), but ideally, at least in the future, one may always want to generate an HDR grading as the first grading for an HDR camera.

[0045] The interior is a relatively complex environment due to the presence of several different light sources (outdoor light through the window in the kitchen, oval lights, additional light from the flame, shadowy corners, etc.). Contrary to SDR shooting, where one typically only remembers the basic average light level (and adds some fill light for the darkest areas given a generally based lighting capture), in high-quality HDR shooting, one should remember the various sub-parts of the different lighting in the scene, such as objects on a table under a strong spotlight, objects seen through an open door in an adjacent unlit room, self-luminous objects, etc., and all these objects should give the final viewer the correct visual impression, or at least a reasonable impression (e.g., such that objects that are less important to the story do not draw all the attention) in the final (one or more) grading. Especially for making a beautiful HDR movie, one can pay more attention to the lighting composition of the scene (rather than mainly making the lighting "conducive to capture"), and thus also pay attention to the primary and further grading. However, for sufficient accuracy but still relatively fast (since there may be time before actual shooting, but there may not be much time, or lay consumers may not care much about too many operations), 3 control points are used in the illustrative example to control the indoor position function shown in the top view. Advantageously, the director can first establish some good bottom values. The guiding principle used here is not to map the brightest object in the scene at 2000 nits, i.e., some number close to 65000, and then see all other brightnesses "messily" terminating below this (linearly). The idea is to give the darker objects in the scene a brightness in the 2000 nit ML_V grading that is approximately their brightness in the 100 nit SDR grading, and perhaps slightly brighter (e.g., a multiplication factor of 1.2), and perhaps the brighter objects in a subset of the darker objects (determined by color composition guidance) terminate at several times 100 nits (e.g., 200 nits).

[0046] In fact, in this merely illustrative example, the director has decided for his grading to select his first control point CP1 to determine the luminance value plotted on the HDR luminance axis (shown vertically). If the portrait is not strongly illuminated by the oval light (which is the intensity he wants to be apparent to his viewers but not excessive in this 2000 nits HDR video, or the portrait might distract from the actions of the actor or presenter), then a value of 200 nits is reasonable. Assuming that under normal lighting levels (average indoor illumination for this configuration of lights present in the scene), the portrait pixels would be given a luminance of approximately 50 nits, he can now decide to map the average color of the portrait (or the clicked pixel or pixel set) to, say, 200 nits for this HDR grading to give the impression of additional luminance. A second control point CP2 can be used to determine the dark (fire poker). A good black value could be 5 nits. These two points have determined the first part of the first luminance mapping function Fs1 for this indoor location, i.e., the brightening segment F_Bri. A multi-linear mapping like this can be determined, or a smoother mapping that surrounds or substantially follows a multi-linear curve (e.g., a tangent), such as a smoothly curved parabola joined by line segments. As mentioned above, the flames in the fireplace are also objects of interest. In the previous sub-process, we have ensured good capture quality. Now, in the grading stage (the image processing circuit 207 will continuously perform grading on the fly during actual shooting for each sequentially captured video image, but now establish the best reference grading for it, i.e., the function used), the criteria for color composition guidance also ensure that the flames have a good HDR impact but are not too bright. The inter-object contrast will depend on the selection of other areas of the scene. Thus, an additional third control point CP3 can be introduced (e.g., by clicking on a display view that shows the function and some (representative) luminance on the vertical / output axis and numbers on the horizontal / input axis), and the director can move it to set the desired flame luminance in the main HDR grading to be output (i.e., ImHDR) to, say, 600 nits. This establishes the second segment (F_diboos), using which a second selectable sub-range of pixel luminance / color can be dimmed or enhanced (usually although color processing is typically 3D, the hue and saturation, i.e., the ratio of color components, can be largely maintained between input and output, only the luminance is changed, i.e., a common magnitude factor for the 3 color components). To keep the explanation simple, the rest of the function can be determined automatically. For example, the darkest color F_zer segment can be established by connecting the first control point to (0, 0). For the top segment, one can, for example, choose from two options.This segment can continue with the slope of the F_diboos segment, resulting in the F_cont segment, or it can apply additional relative enhancement with F_boos to the very brightest colors by connecting the ADC output maximum (65535) to the HDR image maximum (in this example, the color composition guide (the camera operator in this case) considers that a 2000-nit ML_V HDR image is a good representation of the captured scene). The brightest segment of the mapping function can be selected based on which object brightnesses are found in other positions of the total position of the capture to better coordinate the various objects throughout a movie or program (e.g., the outdoor lighting pixel brightness contrast compared to the selected flame brightness, especially the elliptical lamp brightness). The decision can also depend on which brightnesses are present or absent in various environments (or may be temporarily present if a person, for example, walks into the right part of the room with a mirror reflecting the outdoor environment), such as the choice of trim points. For example, although not absolutely necessary, in this example, it may be a good idea to have very bright objects in various positions (i.e., the brightest parts of the lights, even if some of those are street lights in a night scene), equal for all positions, e.g., 2000 nits in this example (note that typically as a graded version, e.g., an HDR output movie, will have a fixed ML_V for the entire movie or program). However, the description of the function also allows the opposite expectation: if for some reason the street lights in an outdoor night scene are desired to be only 1000 nits (e.g., less glare for the viewer on the darkest areas), it can be done equally well by specifying, storing in memory, and loading the function Fs3 for night outdoor shooting, which function Fs3 terminates in a small vertical segment that maps the brightest numbers to 1000-nit HDR output brightness (on the 2000-nit ML_V first graded version output video). It is up to the creator of the video to decide how "perfect" his HDR look should be and thus how much effort he should spend pre-configuring all the cameras and how good one or more grades coming out of the cameras should already be (e.g., ready to display), or how much post-work is to be done to obtain an even better-looking graded version according to the creator's expectations. The basic technology presented allows such simple scenarios with just two position-dependent functions having relatively simple (i.e., roughly graded) shapes, up to many different functions for many different positions and capture scenes. But to illustrate the principle, two examples are sufficient, as a person skilled in the art can make the necessary changes to understand other variations.

[0047] When moving to the next outdoor position (in this example, again with the same basic capture settings), now at Figure 7The daytime outdoor shot is described in which the best or desired second luminance assignment function Fs2 for such a simpler mostly shaded environment can be determined more quickly and easily, where there are only two control points (the third is implicit, i.e., no user interaction is required and is at the maximum of both axes). Now, as Figure 7As shown in the bottom figure, the house has, for example, a slightly lower scene brightness figure because the cloud has moved in front of the sun. In principle, the present technology could also determine a separate (additional) brightness assignment function (and generally a secondary grading function if needed) for these illumination situations (although in the existing position), but the idea is that this is not necessary in this approach because when configured well, the lower values will scale well, showing the actual dimming that occurs in the scene as a reasonable corresponding dimming in the output image and its final display (e.g., after a normalized display adaptation algorithm). That is, when a good mapping function for a good smaller sub-range of the image brightness to be displayed is selected, the sun coming out will result in the appearance of brighter objects, i.e., becoming twice as bright (2x) in the HDR grading (depending on the slope of the mapping function for this sub-range of DN, which in turn will generally depend on the selected ML_V of the output graded video), and vice versa, in at least one output grading, the returned cloud will dim the shadowed house well at this time. In this scenario, the color composition guidance can focus on two aspects of the grading. First, we want good values for the house in the shadow (even though it may not be possible to actually measure these two situations, the camera operator and the director know that there is a fixed relationship between the illuminated and shadowed house because usually the thickest clouds may not be reached in the same sunny day shoot, so for example a 1 / 5 slope can change a 10x scene brightness change to a 2x graded image brightness change). Since it is an outdoor object, those houses can be selected to be brighter in the 2000 nits main HDR grading output than the indoor strongly illuminated portrait, so for example at 300 nits (average). So when illuminated and when viewed from the indoor environment, they look about 200 nits darker than the house. Of course, in principle the director can decide to emphasize the brightness of the portrait more and can even grade this part of the scene with a locally different function. However, although not excluded per se, this kind of complex grading will be atypical, at least for the main grading output (for a fast and easy (single or multiple) camera capture optimization system). Although the main device (e.g., 2000 nits HDR output video) has been graded because it places all object brightnesses in a reasonable position for HDR home viewing, there is usually still a relatively simple relationship (although not as simple as a fixed function) with the relative brightnesses captured digitally by the camera. So if a 1000 DV scene object point measurement is below 25000 DV, the graded output nit value will also be lower no matter which function is selected because the function will increase strictly. The secondary grading can in principle include an image position-dependent mapping, although in practice it has been shown that for a single image source material (without a mixture of different images from different contributors), a single function for all pixels in the image, regardless of their position, generally satisfies several usage scenarios.

[0048] Note that the director can choose a function for the second position itself (i.e., independently), but it may be advantageous if the image color composition analysis circuit 250 has circuitry for presenting several at least two position-based grading situations for display. For example, it can send a split view to the display 253, where a rectangle half the width of the image can be positioned over the main capture RW captured at the previous position to compare the brightness of various objects in the grading at the current position. For example, the selection of the portion of the image containing the oval lamp and the portrait can be dragged to move it near, for example, a street lamp in another grading at another position (which can also be moved) to compare these objects side by side, making it easier to see how their internal brightnesses coordinate. For example, the director can first select Figure 4 the right side of the indoor scene of Figure 4 to compare the brightness appearance of the indoor objects such as the fireplace, portrait, plant, and wall with the ground, house, and sky captured at the second outdoor position to judge, for example, whether the viewer will not be startled when quickly switching from the first position shot to the second position shot, in case the video is re-edited later. Sometimes, due to the fast pace of modern production, you have a constant back-and-forth switching between two positions (e.g., a speaker), and it should not look like a disco strobe. Then, he can shift and swap so that he can compare the brightness of the outdoor house seen through the window in the indoor scene with the outdoor house and the objects in the outdoor shot. Note that if he also wants to compare the indoor object brightness levels in the indoor and outdoor shots, he should ideally ensure that he also has some indoor objects in the field of view when only one (rather than several) main capture RW is used for each position (or he can mix several objects of different outdoor captures in half the screen to compare the outdoor objects with the indoor objects in the other half). In this example, we can have, for example, a stool somewhere in the corridor. It will acquire brightness depending on what lighting is present in the corridor and what time of year it is outside (e.g., summer may have stronger sunlight). So, during the discovery phase, one can choose to look only at the indoor objects and consider the outdoor objects as the second position, or some of those in the first capture at the first position, and then determine the function accordingly. The main position for coordination can be, for example, where most of the shots occur, such as in an auditorium for a commercial or educational movie, with only a few outdoor scenes cut in.

[0049] Suppose the director wants the indoor objects to appear dark when viewed from the outside, but still be visible enough, which can be achieved by positioning them at, for example, 15 nits. Here, we again see the importance of the grading function (compared to simple physical measurements). It can be said that the same numbers can be assigned to indoor objects using a single set of pre-established capture settings, whether they are in the field of view of an indoor or outdoor shot. But for the viewer, there will be problems whether most of the pixels are indoor pixels (establishing the basic brightness appearance for the shot), or only a few of these objects are seen through an open window and the rest of the image pixels image the illuminated outdoor pixels. This fixes the selected F_diboos segment of the second luminance mapping function Fs2. The other two segments, F_zer2 and F_boos2, can be automatically obtained by connecting the corresponding control points to the corresponding extremes of the range. If the function works well enough for the director, i.e., creates a well-looking graded image, then he doesn't need to further fine-tune it (e.g., by adding a third control point), and can send it to the camera to store it in the function memory in the memory section of the second function. Although when using these examples, one won't obtain the most perfect graded output imaginable, the point is that one can obtain better results than when using some simple fixed assignment function for all shots, but still in a simple enough way to make it practical for many shooting scenarios (unless one doesn't know what environment one will encounter, such as unpredictable competitions of various people in various places around the world, but even then one can make pre-discovery settings for environments that might look similar enough to the environment one will encounter to obtain one or more graded outputs that look good enough and are optimized (e.g., 2000 nits ML_V HDR and 100 nits SDR)).

[0050] Re - grading using two or possibly three control points (both the main grading / master grading and, where applicable, the secondary grading, i.e., selected as the parallel output video of, for example, a camera) will generally be sufficient, although the method or system may allow the director to choose to adjust the function as he desires by continuing to add additional control points to change the shape of the (one or more) functions. However, this will also depend on the type of shot, e.g., how many cameras will have a static - angle appearance at more or less complex lighting positions, etc. Also, usually just having a few, even just two positions is sufficient. For example, a "rough" grading case can be created that is good enough for all brighter lighting environments (e.g., for all outdoor positions, whether in fully - lit open areas or in shaded areas between, for example, tall buildings), and a secondary grading for all substantially darker positions, e.g., all indoor positions (which can be around 100x darker in a real - world scenario, but still require some adjusted grading that generally conveys a certain degree of relative darkening so that the end - viewer can ideally see all gradings, with some difference from the brighter outdoor captures, but on the other hand, both positions result in graded images where all or most objects are well - visible (i.e., not too dark and hidden, nor clipped), and ideally have well - coordinated brightness and color. When wanting to create a perfect HDR impression, one may want to fine - tune the various lighting situations at many different shooting positions, e.g., when a specific lighting decor has been purposefully created for the HDR look of a movie, but even when having an environment with various different lighting "accidentally" (such as a show about an escape room) to capture such a technical situation in the best way, even when not considering some specific lighting impression for the final grading.

[0051] This initialization method creates a camera or camera - based capture system that is technically easier to operate. When starting to shoot, the user can focus on other aspects besides color and brightness distribution or composition, but colorimetry has not yet been reduced to a very simple technical recipe, but now an advanced recipe that allows the expectations of human video creators can be used, but in a simple short initialization process.

[0052] Advantageously, the method of setting a video - camera capture mode in a video camera (201) further comprises:

[0053] - determining, guided by the color composition, two corresponding secondary - grading functions (FsL1, FsL2) for at least two positions for calculating a corresponding second - graded image (ImRDR) from the luminance of a corresponding main capture (RW) or the first - graded image; and

[0054] - Store the secondary grading functions (FsL1, FsL2) or the parameters that uniquely define these functions in the memory (220) of the camera,

[0055] - wherein the second graded image (ImRDR) has a lower maximum luminance than the first graded image.

[0056] If there are functions that relate primary luminance or lightness to numbers, and functions that relate primary and secondary luminance or lightness, a person skilled in the art will understand what the relationships between the functions will be, for example between secondary luminance or lightness (via a selected EOTF) and numbers, since this is simply a matter of relating the input and output values along their ranges to determine the function shape. He will also understand how such a set of pairs can be parameterized, for example by slope values of a multilinear approximation. Sometimes it may be technically advantageous to use functions that relate secondary luminance or lightness to primary luminance or lightness rather than DN, since such functions can be advantageously conveyed together as metadata, for example in the SL-HDR2 or even SL-HDR1 format (after inversion). However, other applications may only expect SDR output images, i.e., pixel color matrices, without the need to convey the function(s) that generate those images. Thus, the functions FsL1 and FsL2 are typically only used inside the camera, typically whenever some output from the current shot is needed.

[0057] For dedicated broadcasts, some cameras need to output, for example, only one grading, such as some HDR grading (say 1000 nits ML_V of the target display associated with the video image), or a classical standard dynamic range (also known as LDR) output. Generally, it may be useful if the camera can output two gradings and have them already immediately in the correct levels. If those are already in a format related to these two gradings, such as the applicant's SL_HDR format (as standardized in ETSI TS 103 433), it may be further useful. This format can output video of SDR images, as communication images for, for example, consumer broadcasts or narrowcasts, and functions calculated based on the corresponding SDR images (e.g., 1000 nits HDR images). These functions can correspond to the functions of the present method / system, as described below. One advantage is that it can provide, for example, two classes of consumers for a cable TV operator, the first class having a traditional SDR TV and the second class having purchased a new HDR display.

[0058] Thus, if the main graded video is, for example, 1000 nits HDR, a secondary graded video, such as an SDR video, will be determined. The secondary grading functions (FsL1, FsL2) can work directly based on the main capture RW (i.e., based on the numbers), or advantageously, map the luminance of the main grading to the luminance of the secondary grading. In the second alternative, during the setup phase, a representative first graded image of the luminance of all image objects and a representative second image of the luminance of those image objects (i.e., juxtaposed pixels) will be determined for each position (and possibly for some orientations), and for each luminance of one of the images, the corresponding luminance of the other image in the image will be determined. Then this function will be used to calculate on the fly the luminance-to-luminance mapping of the input pixels of any captured first graded image during shooting to obtain the secondary graded image as the secondary output image. This will all be performed by the image processing circuit 207 substantially simultaneously with shooting. Some believe that the secondary grading should be merely a simple derivative of the main grading, but it can also be said that both gradings may be equally important and challenging.

[0059] Having such a configuration system for / in a camera means that given this configuration, the camera can later be operated during actual shooting. This is achieved by a method of capturing high dynamic range video in a video camera that outputs at least one (and possibly two or more) graded high dynamic range images whose pixels have assigned luminance, the method comprising:

[0060] - applying the method of setting the video camera capture mode in the video camera (201) as described above;

[0061] - determining corresponding positions for at least two positions (Pos1, Pos2) for the current capture position;

[0062] - loading the corresponding luminance assignment function (Fs1) for that position from the memory (220) of the camera; and

[0063] - applying the luminance assignment function (Fs1) to map the numbers of consecutive images captured simultaneously at the current position to the corresponding first graded images (ODR; ImHDR), and storing or outputting these images.

[0064] The corresponding positions mean that the camera operator (or another operator operating the second camera) will not be exactly in the same position (or orientation) in the scene as was selected during initialization. One must be able to move freely to take shots. This means in the vicinity of the surveyed and stored positions, i.e., generally in the same lighting conditions, for example in the same part of the scene. This can mean different things. Outdoors, under the same natural illumination, the whole world can be a set of corresponding positions, at least for example when the system operates in a way that the color composition guidance does not choose to distinguish different outdoor positions (e.g., when the photographer actually assumes that technical role, such as when the camera is operated by a layperson videographer and that person guides the camera). Indoors, there may be more complex illumination because the lights can be anywhere, the rooms may have variable shapes, there may be light-shielding objects that give different shadows, etc. A good illustrative example is a shoot where the director purposefully wants to shoot in a strongly lit room, an average-lit room, and a dimly lit room (e.g., getting only indirect light from an adjacent average-lit room through a half-open door). To achieve the maximum HDR look. With this camera (or a system of a device including the camera), one can then professionally and accurately coordinate the brightness in one or more gradings of all these rooms as desired and then move freely through them (e.g., following the actors). Even if he opens the door a bit further, if the system is well-initialized for the darker corners in the darkest room, good-quality SDR and HDR gradings will still be obtained. Various embodiments will have additional technical elements in the camera to be able to quickly and reliably determine the position, i.e., the lighting condition, at each moment of the shoot. Basically, the goal of this system is to find the lighting condition (i.e., not necessarily the light itself, let alone the average light level), rather than the lighting aspects important for image representation, i.e., the light traveling from several regions and / or objects of the scene towards the camera, i.e., the scene brightness of various important object points. It is not very relevant that the geometry of exactly the same objects is seen in any capture, i.e., the same set of, for example, circular pixels. Instead, what is important is that the same or substantially similar brightnesses occur to form the composition of the regions in the captured and later graded images. Thus, a geometric patch of pixels with a brightness distribution around L_obj can shift to the right in a later capture (regarded as a capture at the same position) because the camera operator has moved to the left (if it partially moves out of the captured image area, the area may be halved). Or, if the camera operator moves backward, the imaging area may become smaller, or when the camera operator steps backward and they suddenly appear, the imaging area may undergo perspective distortion, be partially occluded by objects in front of it, etc.Therefore, it is important that the brightness of a certain shape around L_obj will be in all these shots and, for normal shots, such as in a studio, will not suddenly go too far or too close (also given the minimum focal length of the lens) such that the area will grow from being almost the entire image on one hand to just a few pixels on the other hand. Even though those scenarios may not necessarily be a problem for the present technology, the impression for the viewer may not be perfectly maintained in grading, which generally will not be a problem of sufficient concern. Various technical elements can help determine this similarity. For example, image and / or object recognition methods can be used, which should of course reasonably identify the type of situation of the position / location characterized by, for example, 5 main different brightness regions (e.g., flame, normal object, dark object in the shadow area, lamp, and view of the outdoor sunny day). In some scenarios, there may already be such brightness sub - ranges that can identify at least two main different positions, but more robustness can be achieved if, for example, it is also verified that bright - range objects are between dark and medium - brightness objects. Additionally or alternatively, other technical means for determining the position can be used, which will imply the same brightness composition unless the scene has changed significantly (even an explosion will generally act within a set of surrounding brightnesses and can be accurately or roughly accommodated for such cases even by this method, which of course does not even need to be able to perfectly handle 100% of all possible shot situations to improve the fixed SDR method, because for an explosion, even in an impressive HDR grading, trimming a large number of non - typical pixels can be allowed).

[0065] Functions can be stored in a data structure together with position information (and possibly other function - related information, such as the maximum brightness of the output brightness range and / or the input brightness range), which can take several coding forms. For example, they can be marked as enumerated (position_1, position_2) or absolute, e.g., related to GPS coordinates or other coordinates of a positioning system, and / or have semantic information useful to the operator (e.g., "basement", "center of the hall facing the music stage during the play state", etc.). Although not always necessary, for example, the accuracy of a differential global navigation satellite system may be useful, such as for indoor use and studios with adjacent partial sets, or any positioning system with similar capabilities, such as centimeter or decimeter accuracy (although generally an aggregated characterization with an accuracy of, for example, meter magnitude will also be sufficient). This data format can be transmitted to various devices and stored in memory locations for various user - interface applications, etc.

[0066] Advantageously, the method is used in association with a camera (201) comprising a user interaction device such as a two-way switch to switch between memory locations of different illumination positions stored in the capture environment, which, when pushed in one direction, selects the previous position in a chain of linearly linked positions and, when pushed in the opposite direction, selects the next position. A two-way switch is a switch that can be moved in (at least) two directions and operates (different) functions for these two directions. For example, in practice, it can be a small joystick or the like, which in any case is considered easy to implement by camera manufacturers, for example, usually on the side or the back of the camera.

[0067] This is a simple and very fast method that relies on the user to quickly select a position. For example, and especially for only a few positions (the order of which is easy to remember), the camera operator can flick the switch before entering an indoor area. Some embodiments may use, for example, a summary brightness metric that applies a new function starting from the moment the device actually sees the first capture, where the number of photoelectrons has significantly decreased (respectively increased), which means that at that capture time, the operator has moved to a position with less light (for example, walked through a door and is now covered by a ceiling, side walls, etc.; or in a music performance, turned from facing the stage to facing the audience at the back, which first requires a change in the secondary brightness mapping function to create, for example, an SDR output feed). In the case of several image delays, in the case where a smoother change is desired (for example, considering how quickly the outdoor illumination darkens due to the geometric configuration of the entrance, or generally regarding the degree to which sudden changes are allowed, but usually there are no annoying changes anyway (the system advantageously may not require an adjustment time better than classical autoexposure); see below for longer distance transitions regarding lighting conditions, such as in a corridor)), an advanced temporal adjustment of the brightness can be achieved.

[0068] Advantageously, the camera (201) includes a speech recognition system to select a stored luminance allocation function or a secondary grading function based on an associated description of a location such as "living room". This allows the camera operator to free up their hands, which is useful in cases where they want to use them, for example, when composing, such as changing the perspective of a zoom lens. Speaking to the camera uses another part of the brain, so there is less interference with critical tasks. In the case of (even slightly) re-editing the captured footage, the camera operator can pause for a moment when selecting a new location, and those images with speech will be cut from the final video production. However, for real-time measurements, it can be positioned such that the whispers of the camera operator are difficult to record by the main camera microphone, which can be achieved, for example, by having a set of beamforming microphones with their main receiving lobes facing the camera operator, i.e., having an audio capture lobe behind the camera (while the main microphone 523 will be focused on the presenter or scene being filmed, i.e., mostly captured from the other side). Additionally, other cameras in the scene can be positioned far enough away from the whispering camera operator such that his voice is hardly recorded or at least imperceptible, and there is no need to filter it out through audio processing. The camera operator can train the whispered names of the locations while capturing one or more high-dynamic range images (o_ImHDR), and use well-distinguishable names (e.g., "shaded area under the tree in the forest" is about the longest name one might want for quick and easy operation, "tree shade" is better, if there are not too many locations that require detailed description to distinguish, such as "tree boundary", or "forest edge" is another possible location where, say, half of the hemisphere is dark and the other half is illuminated).

[0069] In impossible systems or situations, other techniques (embodiments) can be used.

[0070] Advantageously, the method ( / system) uses some location beacons that can be fixed in frequently used locations (such as a studio) or hung before shooting (e.g., in the home of a person whose decoration is being scouted as interesting), and these location beacons can be simple beacons that give, for example, three different ultrasonic sequences or microwave electromagnetic pulse sequences for identification, starting, for example, at the second one, and the camera (201) includes a position determination circuit, such as based on triangulation. Each location can also have a beacon, and then when they are properly placed, the camera can detect which beacon is closer from the arrival time one second after the clock. Alternatively, the camera can transmit its own signal to the beacon and wait for the return signal, etc.

[0071] One can also quickly stick LED flashing patterns on the ceiling. If they are infrared and the camera has a separate infrared detection, it does not even matter if they come into the field of view (if the video sensor blocks IR). The patterns can identify a room or a sub - area of the room, etc.

[0072] The camera (201) can alternatively (or additionally) also include a position recognition system based on the analysis of the respective captured images near each of at least two positions. Monitoring the amount of light at each position can be very useful. The automatic itself can detect whether some photometric measures of the aggregated situation have changed sufficiently or have approached the situation of the position. In some cases (e.g., with complex changing lighting such as in a disco or an explosion, etc.), it may be desirable to rely on the geometric recognition of color (texture) patches. For example, a red sofa or a rectangular shape against a green wallpaper can be recognized (if the wallpaper has certain objects, e.g., prints, this will make the recognition easier) as being present in a room rather than, for example, outside. In areas where people often take pictures, this can be accurately trained. When performing a quick initialization, this information can still be quickly collected by the camera (as described below). It generally provides a robust recognition of the room. The advantage of this technique is that high - quality imaging sensors are already available and there may also be some image - processing capabilities (which can also be reused for other purposes). The disadvantage is that more complex algorithms may require a dedicated processor or significant additional processing capabilities and capacities for the existing typical camera image - processing functions, but ICs are still getting more powerful year by year, so for some future cameras, this may be an option (e.g., in mobile phones, which are anyway becoming powerful computers). It is expected that future devices (e.g., phones) will have more processing power on the main board, and maybe even a neural processor. Additionally, since cameras are getting wifi and 5G connections more regularly, the identification of those sites (e.g., triangulation, time - of - flight) can also be used. If the image - analysis processing is done by external means (e.g., in the cloud) or on a local computer, the capabilities or upgradability of the camera may be less important. For identifying the shooting position, low - quality low - resolution communication images may be sufficient.

[0073] Although the innovative concept of the present invention has been very useful for single-camera capture during operation, it may be even more useful and powerful for multi-camera shooting. It is advantageous if a method of setting a video camera capture mode for output pixel brightness in one or more graded versions of an output video sequence of a specified image is provided in a secondary video camera (402), the method including setting a video camera capture mode for output pixel brightness in one or more graded versions of an output video sequence of a specified image in a first video camera (401), and communicating between the cameras to copy a set of settings (including aperture setting, shutter time setting, and analog gain setting) and any determined brightness assignment function from the memory of the first camera to the memory of the second camera.

[0074] Thus, for example, a first photographer can use his camera to discover a scene and generate typical functions for several positions of the light of interest in the scene. Then he can download these settings to other cameras before starting the actual shooting. It may be advantageous if all cameras are of the same type (i.e., the same manufacturer and version), but the method can also ideally be used with cameras that perform differently if some additional measurements are made. For example, if the second camera has a sensor with a smaller dynamic range, for example, it has a full well of 20,000 pixels and noise that has been 50 pixels, and it can still set its performance for, for example, a flame in a room relative to pixel overflow. Of course, this camera will then produce noisy black spots (however, the bright parts of the video have been well aligned), but this can be solved by using an additional post-processing brightness mapping function that slightly darkens the darkest brightness and / or denoises, etc. If one has to work with cameras that are really very different (e.g., cheap cameras that are to be destroyed during shooting), the method can always be used twice, where two camera operators independently discover the scene with the two most different cameras (and the other cameras can then copy the functions and basic capture settings based on their proximity to the respective worst cameras).

[0075] Advantageously, a method of setting a video camera capture mode in a secondary video camera (402) has one of the first video camera and the second camera being a static camera with a fixed position in a part of the shooting environment and the other camera being a movable camera, and copying the brightness assignment function for the position of the static camera to the corresponding function memory of the movable camera, or copying the brightness assignment function in the movable camera for the position of the static camera from the corresponding function memory of the movable camera to the memory of the static camera.

[0076] This can be in a classic setting for, e.g., studio broadcasts (i.e., studio cameras; but it can also be, e.g., a static camera in a live production of a sports event), where some cameras are positioned at convenient locations on the studio floor in front of the decor (set, art design), e.g., to capture parts of the set, or speakers from different directions, but, e.g., a fixed camera can also be attached to the ceiling of a room during a movie shoot for an aerial view, etc. If the static camera has all or most of the relevant objects in the field of view of, e.g., a living room with an adjacent kitchen (which can be considered a single free - range environment for the actors), the function of the static camera can be copied to the dynamic camera that may also enter the shot (or at least a part of the function of the static camera is copied, e.g., if everything except the kitchen window is determined by the static camera, then that part of, e.g., the secondary grading curve may have formed the first part of the secondary grading curve of the dynamic camera, but the dynamic camera itself from its own scene discovery can still determine the upper part of the luminance mapping function corresponding to the world outside the window 410, etc.). Vice versa, the dynamic camera operator (when loading the determined function into one or more cameras, this role can be performed under the guidance of color composition, or when copying at least one function from his camera, by the camera operator) can walk past a certain static camera and copy at least one suitable function into it (and usually also the basic capture settings for that location, such as aperture settings, etc.). In a more advanced system, the static camera can rotate, and then, e.g., two functions can be copied, one for shooting in the direction of the kitchen (which will or can include outdoor pixels), and one for shooting in the direction of the stove side. This can also be done automatically by adding a general direction code (e.g., based on a compass) to the function data structure, and then the static camera can decide for itself what to use in which situation, e.g., by dividing the angle based on which side of the direction the static camera is currently pointing between two reference angles, or it can be indicated to the static camera by the camera operator via user - interface software what to use specifically under which conditions (e.g., a standard camera can transfer its operation menu to the dynamic camera, so the operator can program the static camera by looking at the options on the display of the dynamic camera).

[0077] One advantageous way of embodying the innovative concept is in a multi - device system (200) for configuring a video camera, comprising:

[0078] - a video camera (201), for which a capture mode is set that specifies the output pixel luminance in one or more graded versions of an output video sequence of an image to be output by the video camera to a memory (208) or a communication system (209),

[0079] - Wherein, the camera includes a position-capture user interface (209) arranged such that an operator of the video camera can move to at least two positions (Pos1, Pos2) in a scene, the at least two positions having different illuminations compared to each other, and capturing at least one high-dynamic range image (o_ImHDR) for each position, the at least one high-dynamic range image (o_ImHDR) being selected via the position-capture user interface (209) as a representative primary HDR capture for each position;

[0080] - An image color composition analysis circuit (250) arranged to receive the corresponding at least one high-dynamic range image (o_ImHDR) and enable color composition guidance to analyze the at least one high-dynamic range image (o_ImHDR) to determine

[0081] a) The maximum luminance region of the image and, based thereon, determine at least one of the aperture setting, shutter time, and analog gain setting of the camera, and

[0082] b) A corresponding luminance assignment function (FL_M) of the nit values of the digital to the first graded image (ODR) of the primary captures for the at least two positions via a function determination circuit (251) for the at least corresponding first graded images (ODR) corresponding to the respective primary captures; and

[0083] - Wherein, the camera includes a function memory (220) for storing the corresponding luminance assignment functions (Fs1, Fs2) for the at least two positions determined by and received from the image color composition analysis circuit (250) or parameters uniquely defining these functions.

[0084] Unlike all components residing in a single camera (which is advantageous for ultimate portability if, for example, a team of people wants to explore an environment in, e.g., Urbex shooting), it may be advantageous if some features reside, for example, in a personal computer. On the one hand, these have the benefit of usually a large amount of computing power and the possibility of installing various software components, but on the other hand, one can connect a larger and better-quality grading monitor and can more easily shield it from ambient light (or even place it in a dedicated darkened grading room quickly set up along the scene or in an OB van). Generally, the capture mode can mean how an image is captured, but in this patent application, it is specifically also pointed out how the capture is output, i.e., what type of, e.g., 1000 nits HDR video is output (e.g., whether the darkest object in the scene is represented slightly brighter or vice versa, e.g., kept very dark). That is, it involves the possibility of specifying, roughly or precisely for all possible object brightnesses, the brightnesses that may occur in the graded optimized brightness of the captured scene corresponding to in at least one output graded video. Of course, for different dynamic range uses, one may want to output several different graded (and usually differently encoded, e.g., perceptual quantizer versus Rec. 709, etc.) videos. Therefore, the camera requires new circuitry to enable its operator to go to a certain environment of representative illumination and specify this by using a capture user interface 210 for specifying the initialization of the capture and all data from the camera side (e.g., an image color composition analysis circuit 250 residing in a personal computer can operate with a third user interface (mapping selection user interface 252), and the color composition guidance can utilize the third user interface to specify various mapping functions, i.e., shifts such as Figure 7The control points as explained). On the one hand, he will capture representative images there (a good capture is usually the main capture RW), and on the other hand, he will specify the corresponding position with at least minimal data such as an order number (e.g., position 3), but may have more information, such as semantic information that can be more easily used in various embodiments of the selection user interface 230 at a later time. This needs to be managed by the operating software of the capture UI, because the final result has, on the one hand, the basic capture settings (aperture, etc.) for the camera to operate in that lighting environment later, and on the other hand, functions that can essentially calculate various graded videos and their pixel brightness starting from the digital raw capture. This needs to be managed and stored in the corresponding location in the function memory 220 (note that the basic capture settings can also be stored there, or in some other memory). From there on, the images of any raw captures from the initialization phase have become irrelevant, and all (usually coordinated) information about the best shots in the respective positions is in the stored basic capture settings and functions, and the camera is ready for actual shooting (i.e., recording of a real talk show or shooting of a part of a movie, etc.). Then the important user interface becomes the selection user interface (230), which the camera operator can use to quickly indicate which position-related settings the camera should switch to.

[0085] A useful embodiment of a system for configuring at least one video camera (200) will have a function determination circuit (251) arranged such that color composition guidance can determine two corresponding secondary grading functions (FsL1, FsL2) for at least two positions for calculating a corresponding second graded image (ImRDR) based on a corresponding main capture (RW) or first graded image (ODR), and the camera (201) is arranged to store those secondary grading functions (FsL1, FsL2) in memory for future captures. If the camera operator switches to a new position, both the main function (Fs1) for calculating the first graded output video based on the captured DN (e.g., 3000 nits HDR main output) and the secondary function (FsL1) for calculating the secondary (e.g., SDR) output video for that position will be selected for camera operation of the lens at that position. For example, switching up could be the first position, switching right the second position, switching down the third position, and switching left the fourth position, which would be a user-friendly operation sufficient for many shooting scenarios, but if more positions are needed, more intelligent selections can be assigned to user actions, e.g., switching up is the next position depending on where the camera operator is shooting, and switching down could mean going to a room on the other side of the corridor. Or a more advanced automatic or semi-automatic system could be employed, e.g., using audio (if the camera operator has to physically walk to a distant room before continuing to shoot, he has more time to change the pre-established data for the new position than when he has to continuously change by a running photographer, e.g., running behind an actor along a staircase).

[0086] A typical secondary grading for any high dynamic range main graded video is an SDR graded video, but secondary HDR videos with lower or higher ML_V are also possible. An innovative camera will have at least memory for these various functions and their management, and in particular select the appropriate function(s) for producing a high-quality graded video output during operation. An innovative part in a computer or in a separate window running on a mobile phone (where the main part also serves as a camera, or the secondary part does not serve as a camera) etc. will have the ability to set up a user interface that typically includes the appropriate settings and functions of the camera in addition to proper communication with the camera for various positions (unless the system works completely automatically).

[0087] Accordingly, the novel camera itself includes a system for setting a video camera capture mode that specifies the output pixel luminance in one or more graded versions of the output video sequence of an image, or is configured to operate in such a system by communicating, for example, multiple HDR captures to a personal computer and receiving a corresponding luminance mapping function and storing it in a corresponding memory location, and the camera has a selection user interface (230) that is arranged to select from the memory a luminance mapping function or a secondary grading function corresponding to the capture location.

[0088] Various useful embodiments of the novel camera are especially such (various devices potentially combined in a high-end camera for selectable operation or higher reliability):

[0089] A camera including a user interaction device such as a double-throw switch (249) to switch between function memory locations for different lighting in the shooting environment, which, when pushed in one direction, selects the previous position in a chain of linearly linked positions, and when pushed in the opposite direction, selects the next position.

[0090] A camera (201) including a voice recognition system and preferably including a multi-microphone beamformer system directed at the camera operator to select a stored luminance assignment function or a secondary grading function based on an associated description of a location such as "living room".

[0091] A camera (201) including position and / or orientation determination circuitry, such as position determination based on triangulation with a positioning system in a spatial region around the at least two positions, and orientation determination circuitry such as connectable to a compass.

[0092] The camera (201) according to claim 12, 13, 14 or 15 includes a position recognition system based on an analysis of the corresponding captured images near each of the at least two positions.

[0093] The camera will typically identify various colored shapes in different positions based on basic image filtering operations such as edge detection and better level patterns of clearly distinguishable feature embeddings. Various versions of image analysis are possible, and several of them are elucidated in the following section on the details of the graphics-based teachings.

[0094] While sampling the position, the camera operator can slightly scan the environment, such as capturing images at angles around the main direction, and aggregating as if captured with a wide-angle lens. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] These and other aspects of the method and apparatus according to the present invention will become apparent and elucidated from the embodiments and examples described hereinafter and with reference to the accompanying drawings, which are only non-limiting specific illustrations for exemplifying more general concepts, and in which dashed lines are used to indicate that a component is optional and that a non-dashed component is not necessarily essential. Dashed lines can also be used to indicate elements that are interpreted as essential but hidden inside an object, or for intangible things such as the selection of an object / area (and how they can be displayed on a display).

[0096] In the drawings:

[0097] Figure 1 (In Figure 1 A) schematically shows, on the one hand, how one or more camera operators can take pictures in positions with various lighting conditions, and according to the present invention, these positions can be studied to obtain corresponding good shooting conditions for the camera, that is, a pattern of the optimal values of various brightnesses that an object can have in a scene represented in at least one output graded video as required for each position. Figure 1 B also shows how a better secondary graded output video (RDR) can be derived compared to a simple technical solution for lightness or brightness in the output video. Figure 1 C is also shown in a two-dimensional graph so that the concept of the desired brightening of, for example, a dark scene and thus a dark image object can be better seen;

[0098] Using Figure 2 , we have illustrated what the whole system generally does approximately with typical general components (related to the role of a person operating various devices) so that a technically improved camera (201) is ready to be automatically used in such a completely different lighting environment as Figure 1 shown; two separate devices are shown, but the functions of both can also reside in a single camera;

[0099] Figure 3 shows in more detail how there can be several ways to create an output video with a graded video maximum brightness of, for example, 700 nits, some methods being better and some being less suitable, and the better method being the method, system, and apparatus, in particular the novel camera, that are satisfied;

[0100] Figure 4 is an example of a complex indoor lighting environment for explaining some concepts related to the potential impact of shooting with one camera at several positions or several cameras at several positions and the orientation at any position, and these concepts can also be considered in more advanced embodiments of our present invention (the cameras shown can be the same camera operating at different times, or different cameras operating simultaneously);

[0101] Figure 5Shows a more advanced camera, in which a position - related function creation circuit and / or software are integrated, as well as some possible other circuits for selecting an appropriate position - related function during any actual shooting, and an embodiment of a display in a scenario where a function can be reasonably selected on - site;

[0102] Figure 6 Some illustrative examples of the user interface are introduced to define a main hierarchical luminance mapping function for digital creation from any originally captured video image;

[0103] Figure 7 Some further insights into the exemplary shapes of functions for two - location shooting examples (indoor and outdoor) are taught, which will be represented as a continuous video output (2000 - nit ML_V HDR - graded output video);

[0104] Figure 8 Is introduced to show how to grade a secondary, for example, SDR grade when making the main 2000 - nit HDR grade as a starting point;

[0105] Figure 9 Is introduced to schematically show an example of how a camera can detect a position by recognizing certain color patterns caused by the presence of specific distinguishing objects in one or more locations;

[0106] Figure 10 Illustrates some examples from the scenario (or capture) of video production to the final display; and

[0107] Figure 11 Shows another paradigm of coordination given different stored capture settings for two locations. For example, the second location has one stop more open for the aperture while having the same other exposure - determination parameters for the mapping function for both locations, to create, for example, a main HDR video version by merging (i.e., editing and splicing) shots from a timeline of shots taken at these two locations at different times. Detailed Description

[0108] Figure 1A shows an example where (conceptually) a first photographer 150 and a second photographer 151 can take pictures at different positions (this can be the same actual photographer taking pictures at different times, or two actual photographers taking pictures in parallel, where the cameras are initialized and set to replicate according to the present invention). For various reasons, the outdoor environment 101 can be illuminated quite differently, i.e., the illumination level and illumination spread (i.e., non-uniformity) (e.g., the sun shines on the object everywhere at the same viewing angle of its distance, or uniform illumination from an overcast sky) or equidistant light poles (113), resulting in a light distribution with a slightly reduced illumination in the middle between the poles, etc. At night, the outdoors will generally be darker than indoor shooting (and vice versa, i.e., a higher dynamic range), and during the day, it will generally be the opposite. A representative object outdoors in this scene (which must have a reasonable brightness in at least one graded output video) can be, for example, a house 110. Since grading should be optimized for at least one display situation, it may not be desirable, for example, for the house to look too bright, let alone glowing, unless it is deliberately so because they are specially illuminated, i.e., deliberately made to be perceived as such by the viewer. Another key object for monitoring the output brightness is the shrub in the shadow 114. During the day, due to the reflection of daylight on the covering, the streetlight oval area may have a brightness similar to that of the house, but at night, it may be the brightest object (much brighter in the scene so that it may be desirable to dim its relatively extra brightness (compared to the original camera that captures the ADC digits), for example, as a ratio to the brightness of an average illuminated object (such as the house) in the graded output video so that they do not become too obvious or annoying in the prepared graded viewing).

[0109] Indoor objects (such as a plant 111 (or a stool 112)) can have various brightness levels, which depend not only on how many lights illuminate the room but also on where they are hung and where the object is located. But generally, the illumination level may be about 100 times lower than outdoors (at least when it is sunny outdoors in summer, because during a winter storm shooting, some indoor objects may actually have a higher brightness than some outdoor objects).

[0110] Advanced embodiments of the system can utilize a variable definition of position-dependent functions (and position-dependent camera settings). For a basic embodiment, the idea is that it is sufficient to have a set of setting data (aperture, etc.; at least one luminance mapping function) for each position. In some cases, the same values (aperture opening, etc.) can be selected for two capture positions, but it may generally be advantageous to use different values that are on the one hand optimal for each position and on the other hand advantageously optimal relative to each other. In some advanced cases, the director can select, for example, 2 functions and perform various possible tasks. For example, when at position 1, the camera operator can still choose between function 1 or alternative function 2 and decide during operation which function works best. This can be useful both when the function implementation has small changes (i.e., has a slightly different shape) or large changes. It can be used to account for further variability in the position of the shooting environment, such as in a steam bath where there may be more or less fog. Additionally, having alternative functions for a position allows overriding the selected case. For example, the color composition director may have selected two possible functions for an outdoor position, but it is not yet known at initialization which function will be better during shooting. The camera operator or the color composition (CC) director or the camera operator collaborating with the CC director can, for example, decide to swap the first version, which is currently loaded in the main memory for this position and is the function selected by switching to this shooting position, with an alternative function, which then becomes the main function for this position for the selection UI. Or, the CC director can even fine-tune the function for the position and load it into the main position for the remainder of the shooting, thus making this a new in-operation fine-tuning grading performance for this position (usually this should be done for small changes and adjustments). Even with various functions to be selected according to capture expectations, the coordination of the functions for various positions and the ability to quickly associate one or several functions with each position also allow for a very fast but powerful capture in the sense of generating an already quite good output grading (i.e., an output image that is correctly luminance graded). Another typical example that can be classified under the category of (at least) two functions for each (generalized) position is corridor 102. Such a long and narrow environment can have different lighting at various positions along the corridor. Of course, those according to the basic system can be considered as only three different shooting positions (not actually considering any relationships), but it is better to group them together in a generalized position or position group. For example, if the corridor is only illuminated by outdoor light from the front, it will gradually darken, but at a certain position, there may also be a lamp 109 on the ceiling, which will again locally brighten (and may be in the field of view, so may be a separate object with pixel luminance and does not need to be considered in the function and possibly the basic capture settings).The camera operator can quickly switch from one situation to another, or better yet, the system (e.g., the camera itself) can do so automatically during operation (usually after an exploratory test phase, before the capture settings and functions have been set optimally for shooting).

[0111] For example, the CC guidance may have decided with the camera operator that a good first position for a first representative lighting is near the entrance of the corridor (e.g., 1 meter behind the door and facing in if the shooting will follow an actor walking in), and a second representative position is a little before the position where the light is hanging (so that we get some illumination from them, rather than maximum illumination). During shooting, the camera can then behave, for example, like this. The camera operator flicks a switch to indicate that he will move / proceed from the entrance position to the position of the lighting in the corridor (a type of position or function can be stored together for such high-level behavior, such as "gradual lighting" or "proceeding"). Then, during the creation of at least one output graded video (e.g., an HDR version), the camera can use a function with continuous adjustment between the two functions. The amount of adjustment (i.e., how far the function to be used has deviated from the entrance position function to the lighting position function) can determine, for example, exactly where the operator is standing in the corridor if the positioning embodiment allows this (other possibilities (if the delay allows this, but usually one wants a delay of about 1 second or less for live production, but this can be done in offline production) are to first use the first function for too many images, but then when reaching the second position, correct half of the previous images using a function with a gradually changing shape between their first position representative shape and their second position representative shape).

[0112] Figure 1 B and 1C can better illustrate these concepts.

[0113] Figure 1Figure B shows how one might roughly want to map from a first representation (PQ) of an image (e.g., a first grading such as an HDR grading) to a second (usually lower) dynamic range grading (RDR). Generally, in a relatively simple HDR capture, some direct positions in the first representation will be obtained, which are related to the original capture, for example. For instance, the luminance in this first representation can be set to be equal to a number multiplied by a constant. Assume the constant is such that, for example, the largest number (e.g., power(2;14)) is mapped to 4000 nits. Then, all objects will fall at luminance positions (brighter or darker, along the vertical axis of all possible luminances between 0 and 4000 nits), depending on what luminance the object has in the scene (so, for example, if the ADC images a 7000-nit luminance to its maximum number, the output of a 3000-nit scene luminance (i.e., 43%) will become 1714 nits in the first automatically assigned HDR image). In many cases, this may be quite good for the main output of the camera, for the "main HDR video", although it may not necessarily be the best main graded output video (i.e., ImHDR). When creating a lower dynamic range video, the problem of following simple video production rules may become more problematic. The dashed luminance mapping line represents a simple function (F1), such as, for example, a gamma logarithmic function (which is a function shaped starting from a power law for the darker luminances of the HDR input and then becoming logarithmic in shape for mapping the brighter input luminances to fit a smaller range of output luminances). The problem with such mappings is that generally they will not map well to a smaller dynamic range, such as SDR. When displayed on, for example, an LDR monitor, some objects will look too dark. The best-looking image appears when a human (or at least an automation that can calculate a more advanced function for each shot or lighting scenario based on good principles optimized for a colorimetric image, which is more complex than just one fixed average good mapping function) creates an optimally shaped luminance mapping function Fopt. For example, a shot of a plant indoors might be drawn too dark with a gamma logarithmic function, so we want to make the shape brighter for the darkest image objects. This can be seen in Figure 1 Figure C, Figure 1 which shows the same situation plotted in 2D rather than on two 1D luminance axes (and for luminances normalized to a maximum value of 1.0): the solid curve is above the dashed curve and boosts the pixel luminance more, as in the output image, especially for the darkest objects. Further shape fine-tuning can be done, such as dimming the luminance of the house, so that the desired between-object contrast DEL is achieved.

[0114] Although this Figure 1C illustrates the general principle, but it can also illustrate how the camera can calculate the gradual change of the function: if the dashed curve is good for the first position in the corridor and the solid curve is good for the second position, then for the intermediate positions, the camera can use the function shape that lies between the two functions (i.e., gradually move from the first shape to the second shape step by step). Several algorithms can be used to control the amount of deviation as a function of the travel distance towards the second position (usually perfect brightness determination is secondary for a visually smooth appearance). As we learn from the elucidation of the technology in this specification, not only, for example, SDR grading, but also all gradings that the camera can produce as output benefit from a good distribution of the brightness of various objects, and in particular, the empirical coordination of the brightness of objects at various positions in the (one or more) scenes that can be captured in the overall video (indoors, outdoors, natural available light, turning on additional light, etc.).

[0115] Figure 2 Conceptually shows a part of the camera and the rest of the possible devices in the initialization / mode setting system for elucidating aspects of the new method (a person skilled in the art can understand which elements can work in which combinations or individually, or be implemented by other equivalent embodiments).

[0116] The first basic part of the camera has been described above, so we describe some other typical elements of this new technical method.

[0117] The capture user interface 210 will cooperate with further control algorithms that can, for example, run on the control processor 241 (this processor depends on the type of camera, such as a slow-shifting professional camera or a fast-evolving mobile phone, etc., because a camera with a general-purpose processor can run the algorithm on the GPU, while another camera can have a dedicated ASIC or FPGA). It will at least manage the management of which position is being captured, what must be transmitted to an external device containing the image color composition analysis circuit 250 (external in this embodiment), what is expected to be received back (e.g., the brightness mapping function Fs2 transmitted in the signal S_Fs), and in which memory location the function for the second position, for example, should be stored. Although at least some or all of the functions can be integrated in another similar circuit, we assume that the camera has a dedicated communication circuit 240.

[0118] For example, assume that at least one high-dynamic range image (o_ImHDR) is output via this communication circuit 240, and the basic capture settings and mapping functions are received (i.e., input) via this circuit (the camera can further communicate with the lens via a dedicated cable, etc., but these details are not relevant to understanding the present invention).

[0119] Further assume that the connection is IP-based and over WiFi, connecting to the camera using MIMO antennas 242, or connecting to a WiFi adapter using USB (it is understood that other similar technologies can be used, such as using 5G cellular, cable-based LAN, etc.). If a single image or several images are sent, error recovery communication protocols such as Secure Reliable Transport (SRT) or Zixi are not required, but this may be useful if the functionality doubles with WiFi communication that also transmits all the images actually captured.

[0120] It should also be noted that in order to determine the luminance mapping function for grading, the received images do not need to have the highest quality, such as resolution, and there may be compression artifacts. In many applications, the actual captured video output delivered by the image processor 207 as ImHDR video images (and possibly also ImRDR video images) has also been directly compressed to the desired AV1 encoded data pool, for example, by using HEVC or VVC or AV1, but some applications / users may expect an uncompressed (graded) video output to, for example, an SD card embodiment of the video memory 208, or directly through some communication system (NETW).

[0121] Thus, with the aid of the image color composition analysis circuit 250 and via the mapping selection user interface 252, the CC guidance can be viewed on the monitoring display 253 to see how the grading looks (roughly (in incorrect colors) or graded). For example, there may be a view showing the LDR color that is generated when the shape of the secondary grading function FsL1 is changed on the fly via control points, and there may also be a second view showing the brighter HDR image, or just the LDR image. Some illustrative examples are given in other figures.

[0122] By performing an automatic image analysis of the scene, the function determines that the circuit (251) may have given a first automatic suggestion for a luminance mapping function or a secondary grading function. The CC guidance can then fine-tune the function via the UI, or perform all processing itself starting from the main capture RW or at least one HDR image. The applicant has developed an automatic meta-algorithm for mapping any HDR input image (e.g., ML_V equal to 1000 nits or 4000 nits) to, for example, a typically SDR output (RDR embodiment) image. The resulting luminance mapping function (used here as a secondary re-grading function) depends on the scene. For camera captures, the function shape will essentially depend on the lighting conditions at any location. The end result is the output of an optimized function (e.g., Fs1) transmitted in signal format S_Fs from the function determination circuit 251, which encodes the function, for example, with a plurality of parameters that uniquely define the shape (e.g., if its equation is Y_out = a*Y_in*Y_in + b*Y_in + c, the parabolic function can be characterized by the values a, b, and c). Of course, the camera and the external device will then know the function specification via the protocol, for example, because the computer runs the camera manufacturer's application.

[0123] Figure 6 Examples with good working functions are shown, which can be used to establish a primary grading (e.g., 2000 nits ML_V HDR output video) and a secondary grading (e.g., 200 nits video).

[0124] These images illustrate the underlying technical principle, but can also be the actual view that the director can see when specifying the function by moving parts of its shape in three sub-windows on the display 253. How he changes the shape is also just a detail of the embodiment: for example, if the slope of a segment needs to be changed, he can increase the slope value via a dial, or tap his finger faster to indicate where the endpoint of the line segment or the linear approximation or control line of the segment should be. Both options can be available in the same system because some users may prefer the first option while other users prefer the second option.

[0125] Assume that the mapping from the digital value (DIG_IN) to a 2000-nit output HDR video consists of two sequential mappings. For simplicity, we again normalize the digital value (DIG_IN) to 1.0 (with this normalization, it doesn't matter how many bits the ADC has, which is actually usually only a secondary issue when defining the optimal grading and the optimal grading function). First, a rough mapping RC (in the rough mapping view 620) can be set, say with two line segments at the outer ends (bright and dark input digits) and a smooth segment in the middle. The positions of the three segments can be determined by setting arrows 628 and 629. This may occur depending on which device is used, for example, by dragging with a mouse on a computer or by pen-clicking on a touch-sensitive screen connected to a camera, etc. The arrows can also be set (at least initially, before human fine-tuning) by, for example, clicking on an object of interest OOI in the main captured view, such as with a user's finger 611 on a flame in the image view 610. The span of the digits (or the luminance, if the same algorithm is used to map the luminance of the input of the primary graded video ImHDR to the output luminance of the secondary graded video ImRDR) will be represented by the up and down arrows (i.e., the positioning of arrows 628 and 629). If, for example, the three segments are automatically determined, this may also fall on a part of the middle segment. The view of the rough mapping 620 can also show a small copy of the selected area (i.e., the fireplace) as a copied object of interest OOIC in the view of the object of interest 625 located at the correct value. The CC guidance can switch through or continuously move through multiple possible slopes (B1, B2) of the line segments in the deeper color, starting from the segment 621 that still grades those objects as relatively dark, to reach its optimal segment 622, where they are graded as brighter in the main HDR output (ImHDR). This may be fine for the darkest objects, but when another key object (the fireplace) ends at a certain offset OF_i and a certain luminance span DCON_i or within-object contrast, this rough grading strategy may not yet be optimal.

[0126] Therefore, in the third view window 630, the CC guidance can fine-tune the luminance in the 2000-nit range produced by the rough grading to obtain a better-graded 2000-nit luminance for the final output (the function loaded into the camera will then be the synthesis function F2(F1(DN)).

[0127] The UI may have positioned the arrow (copy the arrows 638 and 639 to correct the new position, the horizontal position in view 630 corresponds to the vertical axis position in view 620), the second copied object of interest OOIC2, etc. to the correct new positions in the figure. For example, a simple algorithm adjusts the contrast of the flame to anchor the higher coarse grading brightness of the flame brightness range (which becomes the anchor Anch), and repeatedly flicks the button or drags the mouse to increase the slope of the lower segment to an angle higher than the diagonal, so that an offset DCO from the diagonal is achieved at the bottom brightness of the flame brightness range. This creates a customizable second grading curve (CC), which produces a larger contrast range DCON_fi in the final output 2000-nit grading (oHDR_fi) than the DCON_i of the middle 2000-nit grading (oHDR_im).

[0128] Finally (return Figure 2 ), during the actual shooting operation, the image processing circuit 207 retrieves the appropriate function(s) F_SEL (such as Fs1) from the memory, and if necessary, retrieves the corresponding FsL1 of the secondary RDR grading from the memory, and starts applying it to the captured image as long as the shot is taken at that position (or actually near that position, as determined by the camera operator or an automatic algorithm) until the shot reaches a new position. The settings of the aperture and shutter may only need to be done once before the start of shooting via the aperture signal S_ir and the shutter signal S_sh (derived from, for example, the control processor of the camera or via a communication circuit, etc.). Otherwise (and even if the values themselves do not need to be changed, the same values can be reset each time a position change is determined), when the camera operator switches to a new position, and substantially simultaneously with loading a new function to calculate one or more graded versions of the captured video as output, the correct values will be commanded via S_ir and S_sh.

[0129] Regarding the signal format for HDR output and possibly RDR output, a typical useful format can be the perceptual quantizer EOTF (standardized in SMPTE 2084) for determining the color components of non-linear R’G’B, and then, for example, matrixing based on 2020 Y’CbCr. Then, for example, VVC (MPEG common video coding) compression, or keeping the signal uncompressed for coding, etc. In the case where the secondary output is considered traditional SDR, it can use the Rec.709 format. It can be further converted to a specific format for communication, such as narrow range, packetization, metadata specification, possible encryption, etc.

[0130] Thus, the output video signal of any camera embodiment according to the present technical teachings has typically applied a first luminance mapping function (Fs1, Fs2, …) to produce a first graded real image (along a certain luminance range up to a certain selected maximum ML_V of the target display associated with the video). That is, each pixel has a luminance typically encoded via an EOTF or OETF (usually a perceptual quantizer or Rec.709). If desired, a secondary grading can also be added to the video output signal, but will typically be encoded as a function (e.g., secondary grading functions FsL1, FsL2 to calculate a secondary video image based on the primary video image). Typically, the primary grading is an HDR grading, while the secondary grading is, for example, an SDR grading. However, for a backward-compatible HDR broadcast, for example, the primary grading can also be an SDR video, and the common encoding function can be a luminance upscaling function to derive an HDR grading from the SDR graded video. In this scenario, for full backward compatibility, the SDR luminance can be encoded according to the Rec.709 OETF, but for partial backward compatibility, the SDR luminance up to 100 nits can also be encoded as lightness according to a perceptual quantizer EOTF, etc.

[0131] Figure 3 Further shown is what is typically different about our innovative techniques and working methods, i.e., what can be achieved, compared to some simpler methods that can be applied but have lower visual quality.

[0132] Assume that the camera operator has established a reasonable capture of the scene (i.e., good aperture opening, etc.), which is shown on the leftmost vertical axis. Now, for example, a 700 nits ML_V primary grading can be done in three different ways (three different technical principles).

[0133] A first representation of a 700 nits image (NDR) can be formulated by mapping the maximum possible number of the camera (i.e., the maximum value of the ADC) to the maximum value of the grading (700 nits in the selection of this example). Then, all other image luminances will be scaled proportionally (i.e., linearly, s*DN + b). If the capture is to be used as a certain version of the original capture (however, even if non-linearity might be preferred), for example, for offline later grading in the film production industry, this might be okay, but it typically will not produce a good 700 nits grading directly from the camera (usually some objects will be uncomfortably darkened due to capturing very bright scene objects). This is achievable if all camera settings are fixed (i.e., the basic capture settings), and might be a mapping function that is one-time (i.e., for the whole shoot) and the same for all positions.

[0134] For example, another possibility that is typically available when using some (potentially improved) variants of the classical automatic exposure algorithm, which determines a new optimal exposure each time there is some change in the lighting situation, is the second representation UDR. Now, the general object that will be the main luminance of most pixels (which will occur in the exposure calculation based on the average) places all Lambertian reflection objects (under the main illumination or base illumination) at the same output image luminance position (seen on the luminance axis of all possible image luminances of the UDR image, where the portrait in the light is approximately as bright as the outdoor house).

[0135] This is not ideally desired. One would want some difference in luminance (i.e., the appearance to the viewer) DL_env between the outdoor environment with stronger illumination and the average object luminance in the indoor environment, and one might expect a technology that enables one to control this. That is, one generally wants some coordination between the luminance captured at various locations and the advanced controllable luminance. This is shown in the third representation (optimized ODR), which serves as our main HDR grading (imHDR), where the selected main grading maximum luminance ML_M is equal to 700 nits (and has different shots from different locations that are optimally coordinated along this range). The dashed arrow shows one luminance mapped by the optimal luminance mapping function FL_M, which would be the optimal luminance function stored in the camera for the indoor location capturing the flame, as explained in other paragraphs of this patent application.

[0136] Figure 5 An illustration of an advanced camera that can have one or more position determination circuits is shown. The basic parts (lens, sensor, image processing circuit) are similar to other cameras. Here, we show the viewfinder 550 on which the camera operator can see some views when acting in the role of a color composition (CC) guide. This may not be as ideal as the view in a separately constructed grading compartment adjacent to the shooting scene or even in a production studio, but sometimes live broadcasts have to be done under constraints, for example, when shooting alone in Africa without a final customer. Some users want to work like this, and this embodiment can satisfy them. Alternatively, for better resolution, surround shielding, etc., the operator / CC guide can wear glasses 557 for a short time, which can have, for example, a projection device 558 and a light shield 559. What can be used is, for example, the mask used in virtual reality viewing. Also shown is the voice recognition circuit 520 or software, which is connected to at least two microphones (521, 522) that form an audio beamformer. The voice recognition does not need to be as complex as full voice recognition since only a few position descriptions (e.g., "fireplace") need to be correctly and quickly recognized. Whether the camera actually uses the in-camera recognition algorithm or uses its IP communication capabilities to have a cloud service or a computer in the production studio perform it is a detail that goes beyond the description required in this application.

[0137] An external beacon 510 is also shown. This can be a small IC with an antenna in a small box that can be glued to a wall, etc. The beacon can provide triangulation, identification whether they broadcast a specific signal sequence, etc. It will interact with the position detection circuit 511 in the camera. This circuit will, for example, perform triangulation calculations. Or, for a coarser position determination, it can simply determine whether it is in the room, such as based on recognition of signal patterns, and possibly the timing of the signal. All of these position identification systems can function similarly during the initial discovery phase of control, and specifically pre-set the camera or camera system (i.e., a camera in a system with other devices (such as a computer)) during the actual capture, so there may be the same or at least some of the same data paths for generating basic or processed measurement data (e.g., an estimate of the position) to the capture (210) and selection UI (230), respectively.

[0138] Video communication via the network to the outside world can be, for example, a contribution to a final production studio (where the video feed(s) can be mixed with other video content, e.g., and then mixed with a broadcaster), or it can be streamed to a cloud service, such as, for example, cloud storage for subsequent use, or a youth live channel, etc. The advanced camera also has circuitry for determining the functions included.

[0139] Will use Figure 9 Clarify the image analysis circuit. The idea of ​​all these techniques is that during live shooting the position-related behavior of the camera still makes it easy to operate. For some shoots there is a focus puller who can, for example, via an additional small display, select the shooting position in time before changing the focus, but in some cases the photographer has to do this completely on his own (and he is already busy with proper framing and geometric composition after, for example, fast-moving people or actions), so it is good if he can rely on or at least have the help of more technical circuits to determine the position information (and do a greater amount of work during the initialization phase of scene discovery).

[0140] But first Figure 8 In , we gave an example of building (equally easily and quickly) a secondary graded version of an image (e.g. an SDR output).

[0141] The considerations for grading the main HDR grading are usually to make all scene objects visually reasonable or impressive in high-quality image representation (i.e., not too dark and severely invisible, and / or not an excessive brightness impact of one object relative to another, the appearance intensity of light objects, etc.), which will be the main grading that is usually archived and is usually used to derive secondary grading. Thus, most objects are more or less correctly specified by brightness, which can be illustrated by the darkest object (e.g., keeping the brightness of the darkest object equal in all grading re-grading methods).

[0142] Now, the secondary grading may mainly involve different technical considerations regarding how to best squeeze the object brightness range in the main grading so that it fits well into a smaller dynamic range. Fitting well means that one tries to preserve the original appearance of the main grading as much as possible. For example, one can balance the intra-object contrast that maintains sufficient visual detail in the flame with the inter-object contrast on one hand, which tries to ensure that the flame looks sufficiently brighter than the rest of the room and does not have adjacent brightness. This may involve deviating from the equi-brightness concept and slightly darkening the darkest object to create visual contrast. In any case, even though the technical details and artistic details of curve construction may be different, the technical user interface and mathematics behind it may be the same or similar (and the camera will similarly use such functions to calculate and output the position-related (one or more) secondary grading RDRs in parallel).

[0143] For example, in the RDR grading, a typical scene object like a painted portrait can be given a normal brightness LuN. Since the painting is illuminated and also because the 400 nits ML_V maximum brightness of the RDR grading easily allows it, we may want to assign the painting pixel brightness to be approximately 75 nits instead of ~50 nits. Figure 8The circle in B is not a control point now because in this strategy, it is a point somewhere along the luminance range or the corresponding re - grading function segment (main segment F_mainL). Using this segment for many colors might be sufficient, but usually the need for compression expects a multi - segment curve for ImHDR - to - ImRDR re - grading. At least some of these can be connected so smoothly that they look like a non - linear single segment. But in this example, we also want to give some importance to the RDR grading result of the flame. So, we might want to slightly enhance the contrast of the flame. Thus, the actual second control point Cp22 can be positioned at a distance PHW from the average luminance or the minimum luminance of the flame object, for example in the middle compared to the drawing. Then, the selected segment of the main normal object luminance, for example, will continue to this Cp22 at the lower end. In this example, the lift from Cp22 to (max_ImHDR, max_ImRDR), that is, (2000, 400) is considered not only a good re - grading for the flame but also a good re - grading for all other bright objects at this location in the scene (establishing the lift segment F_boosL of the re - grading function FsL1). Note that in this example, we show the re - grading in the absolute axis system in nits ending at the corresponding ML_V value, rather than being normalized to 1.0. This is not because one kind of grading must be done in this domain and another in another domain, but to illustrate that all variants are equally possible. The dark segment of the scene (F_drkL) can again be automatically shifted from the position where CC guides its lower control point Cp21, and if it is considered fine, he will not create another control point for those objects (for example, instead of continuing to (0, 0), he can consider starting to vertically rise to (0, X nits) to brighten the darkest pixels). Figure 8 A roughly shows the expectation of RDR grading by projecting several key objects and one or more of their representative luminances, and Figure 8 B shows the determination of the actual curve for the interior, that is, the actual secondary re - grading function FsL1, which might already be coordinated with the exterior. We show the line segments of both the main grading and the secondary grading from the original numbers, but either or both of those one or more segments can also be curved, for example, having a slight curvature compared to a linear function. A linear function is easy and works well enough when, for example, applied only to the luminance channel (while usually keeping the hue and saturation or the corresponding Cb and Cr basically unchanged), but some might prefer curve segments for the grading curve. Grading can also be applied to the transformation of the 3 color components, such as matrixing, that is, applied to non - linear R’G’B’ coefficients or any color representation, since this is usually an aspect of color science in a camera, but for understanding the principle of the new teaching, these details can be left to the knowledge of colorimetric technicians.

[0144] Figure 9 Illustrates some illustrative examples of how various embodiments of the position recognition circuit (540) for a camera can roughly or more precisely identify the location (and possibly the orientation) where the camera operator is currently taking a picture. After decades of research, the development of image analysis technology has been huge, so several alternative algorithms can be used. Therefore, we will illustrate this technical element with only some examples. Figure 9 A shows that in addition to just determining that we are taking a picture "there" in a room (usually the basic capture parameters and functions have been determined in such a way that they are favorable for any way of taking pictures in that room, and possibly also in adjacent rooms, rather than outside), advanced embodiments can also use 3D scene estimation techniques to determine the position of the camera in the room and the orientation in which the camera is taking pictures. The accuracy of this measurement does not of course need to be as high as, for example, depth map estimation, so we can use many techniques and cheaper techniques that require less computation. We do not need to know exactly the centimeter accuracy where the lens center is located, because the mapping function and the basic capture parameters are considered to work equally well whether 10% of the window is in the field of view or (e.g., zoomed) even 100% (i.e., all pixels of the currently indoor captured image are imaging outdoor objects for lighting). Here, you can again see the main difference in the methods of classical auto-exposure techniques. If zoomed in on the window, the classical auto-exposure techniques will of course produce completely different settings (in fact, they will produce outdoor settings instead of seeing the outdoor from the indoor settings). Therefore, the geometry can be determined, for example, after basic object feature extraction and / or analysis has been completed, by calculating the distance on the sensor between objects DP, the shift of the objects on the sensor related to the camera rotation, etc.

[0145] Since usually we only need to identify the room by recognizing a few typical objects, we focus on Figure 9 B and 9C.

[0146] Figure 9B shows an example of a jump-out feature of interest (i.e., it is found that it can be of interest in terms of the "smoothness" or clutter of other features), the red bricks of the chimney. For example, if red is a color that rarely appears in the room, it may have been counted as a jump-out feature, at least as a starting feature. These bricks can even be determined independently of size (i.e., independently of position) by looking for red corners on the mortar (if size-related features such as rectangles are desired, or for example the overall shape of the fireplace, which can be encoded as distances and angles of linear boundary segments or the like, the algorithm can zoom in and out of the image or parts of it several times, or apply other techniques). Thus, for example, two adjacent bricks have been generalized into such an adjacent pattern, the way of which can be determined by (as a non-limiting example) the G-criterion or the generalized G-criterion (see, for example, Sahli and Mertens: Model-based car tracking through the integration of search and estimation Proc. Of SPIE Conf. on Enhanced and Synthetic Vision 1998, p. 160-).

[0147] The idea behind the G-criterion is that there are elements (usually pixels in image processing) with some properties, such as the red component in the simple example of the fireplace, but it can also be a complex aggregated property generated by pre-computation of other properties. Element properties usually have a distribution of possible values (for example, the red component value can depend on the lighting). And they are geometrically distributed in the image, i.e., there are positions with red brick pixels and other positions where there are none. Figure 9 C clarifies the concept of the principle.

[0148] That is, we take a measure that is high, for example, for red bricks (saturated color) and low for unsaturated mortar. For the mortar, roughly R = G = B, and for the bricks R >> G, B.

[0149] Therefore, we can adopt a discriminative property P, such as the function R - (G + B) / 2 (or it can be the ratio of R / (R + G + B)). One would expect a "double-lobe" histogram, where one type of "object" is centered around one value, such as 1 / 3, while the other type is centered around another value, such as 1 / 2 < P <= 1.

[0150] Now, select two sampling regions R1 and R2 for a traveling sampling filter that examines several positions in the image.

[0151] Now the G-criterion is calculated as:

[0152] G = sum of all possible values Pi[abs_value_of(number_occurer_Pi_in_R1 minus number_occurer_Pi_in_R2)] / normalization equation 1

[0153] The idea is also to shape the regions to be sensitive to what is desired, so R1 could be an L-shaped mortar region around the brick and R2 is the block of the brick within it.

[0154] Then, what will happen when the G-criterion detector is located on such a brick boundary.

[0155] If we run the possible P values Pi from low to high, we see that about 1 / 3 of the low values will occur a lot in the mortar, so if all pixels are completely achromatic, we have for example (theoretically) A_R1, which is the area or amount of pixels in the L-shaped region R1. In R2, there will be no such achromatic pixels. So, the first term of the sum becomes A_R1. We take the dimensions of the second region to be usually the same (in pixels, but not the shape). Assume the brick only has the maximum red pixel R = 255, G = B = 0. Then, there are many Pi values that will give zero occurrences (Np) in either region and contribute nothing to the sum. Finally, there are pixels that only occur in the red brick region R2, i.e., with value Pimax = 1. There will again be those A_R2 = A_R1. So the sum will be 2*A_R1. If we take the normalization factor to also be 2*A_R1, then the G-criterion will give 1.0 for detection. If we position this analysis filter over all brick pixels, both regions will only contain P values equal to 1, so only one bin will count A_R1 - A_R2 = 0.

[0156] So, the G-criterion detects what is present and where by producing a value close to 1.0 if it exists. If not, then 0. The statistics of the G-criterion are a bit complex, but the power is that any (or several) attributes P can be input as needed. For example, if the room is characterized by having wallpaper with yellow stripes on black, next to a uniformly painted wall, the cumulative sum or derivative of the stripe pattern can be calculated. For example, if yellow is classified as +1 and blue as -1, the P feature can be calculated by measuring the pixel colors at the middle positions of multiple colored wallpaper bands, and obtaining the local representative feature P = M1 + (-1)*M2 + M3 + (-1)*M4, where the binary measurements M1 - M4 depend on the underlying pixel colors, i.e., for the wallpaper, P = 1 + (-1)*(-1) + 1 + (-1)*(-1) = 4. Any texture metric or color metric or geometric metric can be constructed and used in the G-criterion. Also, the shape of the sampling region can be determined at will (only the amount of sampled pixels should be the same for easy comparison and normalization).

[0157] The generalized G criterion does not compare the presence of a feature in one location to an adjacent (e.g., neighboring) location in the image, but rather to an overall feature pattern. For example, if a red block is known to have a P value much higher than 1 / 3, or for subtraction above zero, then the occurrence bins of various reds can be compared to a reference bin of zero. That is, regardless of whether the color block appears in the image, a unique red region R1 anywhere in the image can be compared to a virtual region R2 consisting of all Pimin=0 values ​​(itself), and can have a rectangular shape.

[0158] The G criterion may be only a first stage to select candidates, and a more detailed algorithm may be performed if increased certainty is required.

[0159] Thus, typically, upon initialization, the location identification circuit (540) takes one or more images from the location, e.g., typically a master capture RW with a basic capture setup for the location. It begins to identify salient features, such as rare colors, corners, etc. It may build slightly more structured low-level computer vision features for these salient objects (e.g., a brick detector with a G standard). It may store several representations of these representative objects, such as a small portion of the image to be associated, a description of a shape boundary, etc. It may build various mid-level computer vision descriptions for the location and store them.

[0160] During the determination phase of the position during the actual capture, the position identification circuit (540) will perform one or more of these calculations to build an estimate of which position the camera resides in. It may cross-check by performing some additional calculations, for example, to check if this indoor position is likely not somewhere in an outdoor scene by checking some color texture patterns typical of the outdoors on some taken copies of the currently captured image.

[0161] Note that this recognition of typical structures in anything (here, images of various locations) is also typical of what is learned in the hidden layers of a neural network. In fact, as NN processors become more common and cheaper, such ICs may be used, at least for cases where the effort of doing sufficient learning can be afforded (e.g., if this location in these locations is going to be used frequently, because you are, for example, a corporate communications officer, and this is your corporate campus). But other variants may rely on the user capturing only a few images that he considers representative for recognizing scenes (something a human is very good at, e.g., he will capture a portrait and an image of the underside of a fireplace, which is unlikely to be seen outside in the forest), and then when he starts drinking coffee before taking a picture, the circuit's algorithm starts doing a direct (non-learned) statistical image analysis.

[0162] Figure 11Another example shows how to coordinate all capture (mode) parameters to produce a good HDR video output with the possibility of taking in-motion shots in an easy way. The basic capture settings and various functions will generally be coordinated at least in the sense that when the camera is set to its behavior for each position (i.e., when setting the aperture and the mapping function to obtain, for example, a graded HDR luminance or brightness from the captured numbers), the result will look good when directly displaying the entire shot (i.e., all sequential shots at various positions). We have illustrated this with another example where, for example, a camera operator can freely walk between a first (strongly) illuminated room (Ro_1) and a dark second room (Ro_2). The objects in this second room will generally be indirectly illuminated through an opening in the wall (1105), such as an open door, which can also serve as a viewport for a part of the other room if the camera is in the first room and pointed in that direction (and the photographer can use it to move between the rooms). For a future ultimately near-infinite capture dynamic range, there may be no difficulty in the basic capture (i.e., no critical choice for the value of the aperture opening, etc.), and then only the mapping function will be predefined (since given one or more basic capture setting values, an 800 nits ML_V output video and / or a 1500 nits video and / or a 3000 nits video, etc. still have to be produced based on the numbers captured at each moment). But in this example, we assume that at least one near-future camera has a less-than-perfect capture dynamic range and has no opportunity or no clear choice about not using fill lighting in the dark second room. In the first room, during the discovery initial phase, the CC guidance and / or the camera operator of course consider the sun, and at least some of the sunlit clouds (1104) outside the small window can be clipped above the full pixel and the maximum number of digits DN (which is here considered to come from an 18-bit ADC). Thus, the region of maximum luminance is determined as a set of pixels (respectively their luminance) that are just darker than those pixels that might have all been clipped to the same maximum luminance (in this example, some parts of the clouds where some gray-scale value variations of the cloud luminance still want to be encoded). This luminance of the clouds will form a good aggregated maximum since there are no scene areas or important objects of higher luminance at other positions. In various gradings, or at least in the only HDR video output, this aggregated maximum can be mapped onto the TDML value ML_V. There is a bright light (1103) that brightly illuminates some key objects 1102 (e.g., silverware) on the table. The bright light can be reasonably captured without clipping, as well as all other objects in the illuminated room down to the object 1101 with average illumination. In the dark second room, when using the same basic capture settings (e.g., the first aperture value Ir1), the hollow object 1111 can be captured with small numbers but still faithfully enough (i.e., generally with little noise).However, the monster 1110 in the shadow behind something may be captured too dark in this capture setting and not ideal. Therefore, we can set the camera memory values such that each time the photographer starts shooting in the second room (not necessarily towards the camera room, because in such a partial view surrounded by brighter pixels from the first room, it may be fine if not many details from the dark room are seen), a larger aperture opening Ir2 (e.g., 2x more open) is used. Thus, the mapping of the 1000 nits main HDR video output towards anything in the first room will occur together with a function of the first position (indicated in the function named p1) and together with its aperture setting (i.e., the first aperture value Ir1), i.e., it will use the function F_p1_Ir1, for which an input / output value pair is shown from a digital range to the possible brightness range in the 1000 nits HDR grading. Similarly, other digital numerical values will be mapped to other output brightnesses, together forming a mapping function that generally strictly increases. When shooting in the dark room, we can define functions starting from the numbers as they would occur in the case of the first aperture setting, but we know that we will use DN as captured in the case of the second aperture setting Ir2, which is chosen to be optimal for this room because it captures the darkest objects without noise and there are no brightest objects (if we point the camera through the door opening towards the first room object, we usually will not see the outside world through the window and not too much of the bright room, so if very high scene brightness clipping may not be too much of a problem, otherwise one may want to optimize the total set of brightnesses in the shooting environment together, only when shooting in the "second room rotated to open" position, some quality of the darkest objects may be lost due to noise, but this is usually okay too because when no monster is visible and if very dark and very bright pixels will be shown together on the display, the viewer will not see minute test details anyway). When starting from the numbers based on Ir2, the shape of the function of the second room position F_p2_Ir2 will be different from when starting from the DN based on Ir1, but their relationship will be that during the initial stage, the CC guidance has specified that dark room objects (such as the hollow object 1111, say a barrel) need to have brightness pixels around some dark brightness L_drk, which needs to have a specific value according to requirements and in particular an amount or ratio that is darker than the bright brightness L_bri of the bright room object pixels. Thus, from this description, we can see how the discovery phase can coordinate all the required values for two or more shooting positions to ultimately obtain a coordinated graded brightness such that the actual shooting is free and there is no worry about the colorimetric issues of the output video or video version.The various sub - ranges of the characteristic scene area in the first position will be coordinated by placing them at a certain distance from the sub - ranges of the characteristic area in the second position. For at least some of the sub - ranges, this is done in practice by determining the function shape.

[0163] The algorithm components disclosed herein can actually be (fully or partially) implemented as hardware (e.g., part of a dedicated IC) or software running on a special digital signal processor or a general - purpose processor, etc.

[0164] Those skilled in the art should understand from our introduction which components can be optional improvements and can be implemented in combination with other components, and how the (optional) steps of the method correspond to the corresponding devices of the apparatus, and vice versa. The term "apparatus" in this application is used in its broadest sense, i.e., a set of devices that allows the achievement of a specific goal, and can thus be, for example, a small circuit part of an IC or a dedicated device (such as a device with a display) or part of a networked system, etc.

[0165] A computer program product representation should be understood to cover any physical implementation of a set of commands that enables a general - purpose or a special - purpose processor to input the commands into the processor after a series of loading steps (which may include intermediate conversion steps, such as translation into an intermediate language and a final processor language), and execute any characteristic function of the present invention. In particular, a computer program product can be implemented as data on a carrier such as a disk or a tape, data present in a memory, data traveling wired or wirelessly via a network connection, or program code on paper. In addition to the program code, the characteristic data required for the program can also be embodied as a computer program product.

[0166] Some of the steps required for the operation of the method may already exist in the functions of the processor rather than being described in the computer program product, such as data input and output steps.

[0167] It should be noted that the above - mentioned embodiments illustrate rather than limit the present invention. In cases where those skilled in the art can easily implement the mapping of the presented examples to other areas of the claims, we have not mentioned all these options in depth for the sake of brevity. In addition to the combinations of the elements of the present invention combined in the claims, other combinations of the elements are also possible. Any combination of the elements can be implemented in a single dedicated element.

[0168] Any reference signs between parentheses in the claims are not intended to limit the claims. The word "comprising" does not exclude the presence of elements or aspects not listed in the claims. The word "a" or "an" before an element does not exclude the presence of a plurality of such elements.

Claims

1. A method for controlling a video camera (201), including setting, in one or more hierarchical output video sequences of an image, a video camera capture mode that specifies output pixel brightness for corresponding positions in a scene to be captured, the hierarchical output video sequences being characterized as having different maximum pixel brightnesses, the method comprising: - an operator of the video camera moves to at least two positions (Pos1, Pos2) in the scene having different illuminations compared to each other, and captures at least one high dynamic range image (o_ImHDR) for each of the at least two positions of the scene; - color composition guidance analyzes the at least one captured high dynamic range image for each of the at least two positions to determine corresponding maximum luminance regions, and determines at least one of the following based on the maximum luminance: the aperture setting, shutter time, and analog gain setting of the camera; - for each of the at least two positions (Pos1, Pos2), uses at least one of the determined aperture setting, shutter time setting, and analog gain setting of the camera to capture a corresponding primary capture (RW), and stores the determined aperture setting, shutter time setting, and analog gain setting for subsequent image capture when the video camera resides at the corresponding position; - determines at least a corresponding first hierarchical image (ODR) of the corresponding primary capture, including determining an adjustable luminance allocation function (FL_M) for the corresponding position, the adjustable luminance allocation function mapping the digital values of the primary capture to nit values of the first hierarchical image (ODR); and - stores the corresponding luminance allocation functions (Fs1, Fs2) for the at least two positions or parameters uniquely defining these functions in corresponding memory locations (221, 222) of the camera.

2. The method for controlling a video camera (201) according to claim 1, further comprising: - determining two corresponding secondary hierarchical functions (FsL1, FsL2) for the at least two positions for calculating corresponding second hierarchical images (ImRDR) based on the corresponding primary capture (RW) or luminance of the first hierarchical image; and - storing the secondary hierarchical functions (FsL1, FsL2) or parameters uniquely defining these functions in the memory (220) of the camera, - wherein the second hierarchical image (ImRDR) has a lower maximum brightness than the first hierarchical image.

3. A method for capturing high dynamic range video in a video camera, the video camera outputting at least one hierarchical high dynamic range image having pixels with assigned brightnesses, the method comprising: - applying, in an initial step, the method for setting the video camera capture mode according to claim 1 or 2; and subsequently during the capture of the video: - determining the corresponding positions for the at least two positions (Pos1, Pos2) for the current capture position; - loading, from the memory (220) of the camera, the corresponding luminance allocation function (Fs1) for that position; and - Apply the luminance allocation function (Fs1) in the image processing circuit (207) to map the digital values of consecutive images captured at the current position to corresponding first graded images (ODR; ImHDR), and store or output those images.

4. The method according to claim 3, wherein, the camera (201) includes a user interaction device such as a two-way switch (249) to switch between function memory locations of stored positions of different illuminations in the shooting environment, which selects the previous position in the chain of linearly linked positions when pushed in one direction and selects the next position when pushed in the opposite direction, and retrieves the luminance allocation function corresponding to the selected position from the memory of the image processing circuit (207).

5. The method according to claim 3, wherein, the camera (201) includes a voice recognition system to select the stored luminance allocation function or secondary grading function based on an associated description of a position such as "living room".

6. The method according to claim 3, wherein, the camera (201) includes a position and / or orientation determination circuit, such as based on triangulation using a positioning system placed at least temporarily in the spatial region around the at least two positions.

7. The method according to claim 3, wherein, the camera (201) includes a position recognition system based on the analysis of corresponding captured images near each of the at least two positions.

8. A method of controlling a video camera capture mode for setting the output pixel luminance in one or more graded versions of an output video sequence of a specified image by a secondary video camera (402), the method comprises: setting a video camera capture mode for setting the output pixel luminance in one or more graded versions of an output video sequence of a specified image in a first video camera (401) according to claim 1 or 2; and communicating between the cameras to copy a set of settings and any determined luminance allocation function from the memory of the first camera to the memory of the second camera, the set of settings including the aperture setting, shutter time setting, and analog gain setting.

9. The method according to claim 8, wherein, one of the first video camera and the second camera is a static camera having a fixed position in a part of the shooting environment, and the other camera is a movable camera, and copy the luminance allocation function for the position of the static camera to the corresponding function memory of the movable camera, or copy the luminance allocation function in the movable camera for the position of the static camera from the corresponding function memory of the movable camera to the memory of the static camera.

10. A system (200) for configuring a video camera, comprising: - A video camera (201), for which a capture mode is set that specifies the output pixel luminance in one or more hierarchical versions of an output video sequence of images output by the video camera to a memory (208) or a communication system (209). - wherein, the camera includes a position capture user interface (209), which is arranged such that an operator of the video camera can move to at least two positions (Pos1, Pos2) in a scene having different illuminations compared to each other, and capture at least one high dynamic range image (o_ImHDR) for each position, and the at least one high dynamic range image is selected via the position capture user interface (209) as a representative main HDR capture for each position; - An image color composition analysis circuit (250), which is arranged to receive the corresponding at least one high dynamic range image (o_ImHDR), and enables color composition guidance to analyze the at least one high dynamic range image (o_ImHDR) to determine: - a) the region of the maximum luminance of the image, and based on this, determine at least one of the following: the aperture setting, shutter time, and analog gain setting of the camera, and - b) for the at least two positions, for at least the corresponding first hierarchical image (ODR) corresponding to the respective main capture, determine the respective luminance assignment function (FL_M) of the digital value of the main capture and the nit value of the first hierarchical image (ODR) via a function determination circuit (251); and - wherein the camera includes a function memory (220) for storing the respective luminance assignment functions (Fs1, Fs2) for the at least two positions determined by and received from the image color composition analysis circuit (250), or parameters that uniquely define these functions.

11. The system for configuring a video camera (200) according to claim 10, characterized in that, the function determination circuit (251) is arranged such that the color composition guidance can determine two corresponding secondary hierarchical functions (FsL1, FsL2) for the at least two positions for calculating a corresponding second hierarchical image (ImRDR) based on the respective main capture (RW) or the first hierarchical image (ODR), and the camera (201) is arranged to store those secondary hierarchical functions (FsL1, FsL2) in a memory for future captures.

12. A camera, which includes the system according to claim 10 or 11, or is configured to operate the system according to claim 10 or 11, wherein, the camera has a selection user interface (230), which is arranged to select from the memory a luminance mapping function or a secondary hierarchical function corresponding to a capture position.

13. The camera according to claim 12, wherein, It includes a user interaction device such as a double-throw switch (249) to switch between function memory locations of different illuminations at stored positions in the shooting environment. When pushed in one direction, it selects the previous position in the chain of linear link positions, and when pushed in the opposite direction, it selects the next position.

14. The camera (201) according to claim 12 includes a voice recognition system (520), and preferably includes a multi-microphone beamformer system facing the camera operator to select a stored brightness assignment function or a secondary classification function based on an associated description of a position such as "living room".

15. The camera (201) according to claim 12, 13 or 14 includes a position and / or orientation determination circuit (511), such as the position determination is based on triangulation using a positioning system placed in the spatial region around the at least two positions, and such as the orientation determination circuit can be connected to a compass.

16. The camera (201) according to claim 12, 13, 14 or 15 includes a position recognition circuit (540), which is based on the analysis of the respective captured images near each of the at least two positions.

Images