Imaging apparatus, image processing method, and non-transitory computer readable medium

Abstract

An imaging apparatus includes circuitry configured to: acquire an input image from an image sensor; generating information related to color conversion of the acquired input image based on a first color setting parameter and a second color setting parameter, the first color setting parameter being determined according to at least one first light source configured to provide illumination light for the acquired input image, the second color setting parameter is determined according to at least one second light source configured to provide illumination light for the acquired input image; and initiating output of the output data according to the generated information related to the color conversion of the acquired input image.

Description

Technical Field

[0001] The present technology relates to an imaging device, an image processing method, and a non-transitory computer-readable medium, and relates to a technology that can be used in the field of video production, for example. Background Art

[0002] As a shooting technique for producing video contents such as movies, a technique is known in which performers perform in front of a so-called green background, and the background video is composited thereafter.

[0003] Furthermore, in recent years, a shooting system has been developed and referred to as so-called virtual production, in-camera VFX, or light-emitting diode (LED) wall virtual production: this shooting system can image the performers and the background by having the monitor display a background video in a studio equipped with a large monitor instead of shooting against a green background, and allowing the performers to perform in front of the background video.

[0004] Patent Document 1 below discloses a system technology for capturing images of performers and objects performing in front of a background video.

[0005] Citation List

[0006] Patent Literature

[0007] Patent Document 1: U.S. Patent Application Publication No. 2020 / 0145644 Summary of the Invention

[0008] Technical issues

[0009] By using a camera to shoot the performers and background video in addition to displaying the background video on a large display, advantages over shooting with a green background are, for example, that there is no need to separately synthesize the background video after shooting, and that performers or staff can perform and judge the quality of the performance by visually understanding the scene.

[0010] In addition, a large display showing a background video can also serve as lighting. Therefore, as lighting for shooting, the light of a separate lighting device and the light from the display can be used in combination, and in some cases, it can be considered to use only the light from the display as lighting.

[0011] However, the display used for background video display has spectral characteristics different from those of lighting used in general shooting, such as sunlight or LED lighting. Therefore, when light from the display is used for lighting, there are cases where the captured video is not suitable for color reproduction.

[0012] Therefore, the present disclosure proposes a technique that enables capturing video of appropriate chromaticity even when a display is used as illumination.

[0013] Solution to the problem

[0014] According to the present disclosure, an imaging device is provided, the imaging device including a circuit configured to: acquire an input image from an image sensor; generate information related to color conversion of the acquired input image based on first color setting parameters and second color setting parameters, the first color setting parameters being determined based on at least one first light source configured to provide illumination light for the acquired input image, the second color setting parameters being determined based on at least one second light source configured to provide illumination light for the acquired input image; and

[0015] Based on the generated information related to the color conversion of the acquired input image, output of the output data is initiated.

[0016] In an environment where lighting with different spectral characteristics is used, particularly when a lighting device is used and when a display device having spectral characteristics different from those of the lighting device and displaying a background image is used as the lighting device, switching between color conversion processing for color reproduction, etc. can be performed, for example.

[0017] Furthermore, according to the present disclosure, an image processing method includes: acquiring an input image; generating information related to color conversion of the acquired input image based on a first color setting parameter and a second color setting parameter, the first color setting parameter being determined based on at least one first light source configured to provide illumination light for the acquired input image, and the second color setting parameter being determined based on at least one second light source configured to provide illumination light for the acquired input image; and outputting output data based on the generated information related to color conversion of the acquired input image.

[0018] In addition, according to the present disclosure, a non-transitory computer-readable medium is provided, on which a program is embodied. When the program is executed by a computer, the computer performs an information processing method, the method including: acquiring an input image; generating information related to color conversion of the acquired input image based on a first color setting parameter and a second color setting parameter, the first color setting parameter being determined based on at least one first light source configured to provide illumination light for the acquired input image, and the second color setting parameter being determined based on at least one second light source configured to provide illumination light for the acquired input image; and outputting output data based on the generated information related to color conversion of the acquired input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] [ Figure 1 ] Figure 1 It is an explanatory diagram of a photographing system according to an embodiment of the present technology.

[0020] [ Figure 2 ] Figure 2 It is an explanatory diagram of a background video according to the position of a camera of a shooting system according to an embodiment.

[0021] [ Figure 3 ] Figure 3 It is an explanatory diagram of a background video according to the position of a camera of a shooting system according to an embodiment.

[0022] [ Figure 4 ] Figure 4 It is an explanatory diagram of the video content production process according to the embodiment.

[0023] [ Figure 5 ] Figure 5 is a block diagram of a photographing system according to an embodiment.

[0024] [ Figure 6 ] Figure 6 is a flowchart of background video generation of a shooting system according to an embodiment.

[0025] [ Figure 7 ] Figure 7 is a block diagram of a photographing system using a plurality of imaging devices according to an embodiment.

[0026] [ Figure 8 ] Figure 8 is a block diagram of an information processing apparatus according to an embodiment.

[0027] [ Figure 9 ] Figure 9 This is an explanatory diagram of the types of LED walls used in the embodiment.

[0028] [ Figure 10 ] Figure 10 This is an explanatory diagram of the types of LED walls used in the embodiment.

[0029] [ Figure 11 ] Figure 11 This diagram illustrates the spectral characteristics of various types of light.

[0030] [ Figure 12 ] Figure 12 is a block diagram of an imaging device according to an embodiment.

[0031] [ Figure 13 ] Figure 13 This is an illustration of color temperature and tint.

[0032] [ Figure 14 ] Figure 14 It is an explanatory diagram of the processing of the white balance unit of the imaging device according to the embodiment.

[0033] [ Figure 15 ] Figure 15 It is an explanatory diagram of processing by the matrix unit of the imaging device according to the embodiment.

[0034] [ Figure 16 ] Figure 16 It is an explanatory diagram of a process corresponding to LED lamp lighting according to an embodiment.

[0035] [ Figure 17 ] Figure 17 It is an explanatory diagram of a process corresponding to LED wall lighting according to the embodiment.

[0036] [ Figure 18 ] Figure 18 It is an explanatory diagram of processes corresponding to LED lamp lighting and LED wall lighting according to the embodiment.

[0037] [ Figure 19 ] Figure 19 It is an explanatory diagram of the processing of the white balance unit according to the embodiment.

[0038] [ Figure 20 ] Figure 20 It is an explanatory diagram of the processing of the matrix unit according to the embodiment.

[0039] [ Figure 21 ] Figure 21 is an explanatory diagram of a user interface according to an embodiment.

[0040] [ Figure 22 ] Figure 22 This is an explanatory diagram of a case where the lighting in the embodiment is an LED lamp.

[0041] [ Figure 23 ] Figure 23 This is an explanatory diagram of a case where the lighting in the embodiment is an LED wall.

[0042] [ Figure 24 ] Figure 24 This is an explanatory diagram of a case where the lighting in the embodiment includes LED lamps and LED walls.

[0043] [ Figure 25 ] Figure 25 is a flowchart of a color conversion process according to an embodiment.

[0044] [ Figure 26 ] Figure 26 is a flowchart of a process according to the operation of an embodiment.

[0045] [ Figure 27 ] Figure 27 is a flowchart of a process performed at the time of shooting according to an embodiment.

[0046] [ Figure 28 ] Figure 28 is an explanatory diagram of an example of processing executed by original development software according to an embodiment.

[0047] [ Figure 29 ] Figure 29 is a flow chart of an example of a process performed by original development software according to an embodiment.

[0048] [ Figure 30 ] Figure 30 is an explanatory diagram of an example of processing executed by original development software according to an embodiment.

[0049] [ Figure 31 ] Figure 31 is a flow chart of an example of a process performed by original development software according to an embodiment.

[0050] [ Figure 32 ] Figure 32 is an explanatory diagram of an example of processing executed by original development software according to an embodiment.

[0051] [ Figure 33 ] Figure 33 is a flow chart of an example of a process performed by original development software according to an embodiment.

[0052] [ Figure 34 ] Figure 34 is an explanatory diagram of an example of processing executed by original development software according to an embodiment.

[0053] [ Figure 35 ] Figure 35 is a flow chart of an example of a process performed by original development software according to an embodiment.

[0054] [ Figure 36 ] Figure 36 is a flowchart of an example of a process for parameter correction and output performed by original development software according to an embodiment. DETAILED DESCRIPTION

[0055] Hereinafter, the embodiments will be described in the following order.

[0056] <1. Filming System and Content Production>

[0057] <2. Configuration of Information Processing Equipment>

[0058] <3. Lighting form>

[0059] <4. Configuration and Processing Example of Image Capture Device>

[0060] <5. Processing Examples Using Metadata and Original Development Software>

[0061] <6. Summary and Modification Examples>

[0062] In this disclosure, "video" or "image" includes both still images and moving images. Furthermore, "video" may refer not only to video data displayed on a display, but also to video data that is not being displayed on a display. This also applies to "image."

[0063] For example, in an embodiment, although the background video before being displayed in the display, the captured video captured by the camera device, the background video switched using the switch and the captured video are not actually displayed videos but video data, for the sake of convenience of description, they will be expressed as "background video", "captured video", etc.

[0064] <1. Shooting System and Video Content Production>

[0065] A shooting system and production of video content to which the technology of the present disclosure can be applied will be described.

[0066] Figure 1 The photographing system 500 is schematically shown. The photographing system 500 is a system for performing photographing as virtual production, and the drawing shows a part of equipment installed in a photographing studio.

[0067] In the filming studio, a performance area 501 is provided where performers 510 perform performances and other activities. A large display device is provided on at least one of the rear, left / right sides, and top of performance area 501. Although the type of display device is not particularly limited, the accompanying drawings illustrate an example of an LED wall 505 as an example of a large display device.

[0068] An LED wall 505 is formed by arranging a plurality of LED panels 506 by connecting them horizontally and vertically to form a large panel. Although the size of the LED wall 505 described herein is not particularly limited, it can be a size required or sufficient to display a background when filming a performer 510.

[0069] At desired positions such as the upper side, lateral side, etc. of the performance area 501 , a desired number of lamps 580 are provided, and the performance area 501 is illuminated.

[0070] Near the performance area 501, a camera 502 is installed, for example, for filming movies or other video content. A photographer 512 can move the camera 502 and manipulate its shooting direction, angle of view, and other aspects. It is conceivable that the camera 502 can be moved and its angle of view manipulated remotely. Furthermore, the camera 502 can be moved and its angle of view changed automatically or autonomously. Therefore, the camera 502 may be installed on a camera platform or mobile device.

[0071] Performers 510 in performance area 501 and the video displayed on LED wall 505 are captured together by camera 502. For example, by displaying a landscape as background video vB on LED wall 505, a video similar to a video of the performer 510 actually being at the location of the landscape and performing a performance can be captured.

[0072] An output monitor 503 is provided near the performance area 501. The video captured by the camera 502 is displayed in real time as a monitor video vM on the output monitor 503. This allows a director or a staff member producing the video content to check the video being captured.

[0073] In this manner, the filming system 500 for filming a performance of a performer 510 with an LED wall 505 as a background in a filming studio has several advantages over filming with a green background.

[0074] For example, when shooting against a green background, it is difficult for the performer to imagine the background and the scene, which may affect the performance. In contrast, by displaying the background video vB, it becomes easier for the performer 510 to perform, thereby improving the quality of the performance. In addition, the director and other staff can easily determine whether the performance of the performer 510 matches the background and the scene.

[0075] Furthermore, post-production after filming becomes more efficient than when shooting with a green background. This is because there are situations where so-called chroma key compositing can be configured to be unnecessary, as well as situations where color correction and reflection compositing can be configured to be unnecessary. Furthermore, even when chroma key compositing becomes necessary during filming, it is possible to display only the green and blue video, thus eliminating the need for a physical background screen, which also contributes to improved efficiency.

[0076] In the case of shooting against a green background, the green tint of the performer's body, clothing, and objects increases, and therefore needs to be corrected. In addition, in the case of shooting against a green background, if there are objects such as glass, mirrors, snow domes, etc. that reflect the surrounding scene, it is necessary to generate and synthesize an image of the reflection video, which is a laborious operation.

[0077] In contrast, Figure 1 In the case where the shooting system 500 shown in is used to shoot, the green chromaticity is not increased, and therefore, no correction is required. In addition, by displaying the background video vB, the reflection of an actual object such as glass can be naturally captured, and therefore, no reflected video needs to be synthesized.

[0078] Here, we will refer to Figure 2 and Figure 3 The background video vB is described. Even when the background video vB is displayed on the LED wall 505 and filmed together with the performer 510, the background of the captured video becomes unnatural when the background video vB is simply displayed. This is because the actual stereoscopic background having depth is formed in a two-dimensional form as the background video vB.

[0079] For example, camera 502 can capture performer 510 in performance area 501 from various directions and can also perform zoom operations. Performer 510 does not stay in one location. However, although the visual performance of the background of performer 510 actually seen changes depending on the position, shooting direction, and viewing angle of camera 502, such changes cannot be captured in background video vB, which is a flat video. Therefore, background video vB is changed so that the background, including parallax, has a visual performance similar to the actual visual performance.

[0080] Figure 2 shows a view of a performer 510 captured by a camera 502 from a position on the left side of the figure, and Figure 3 The camera 502 is shown shooting a view of the performer 510 from a position on the right side of the figure. In each figure, the shooting area video vBC is shown within the background video vB.

[0081] Furthermore, a portion other than the shooting area video vBC in the background video vB will be referred to as an “outer frustum,” and the shooting area video vBC will be referred to as an “inner frustum.”

[0082] The background video vB described here means the entire video displayed as the background including the shooting area video vBC (inner frustum).

[0083] The range of the captured area video vBC (inner frustum) corresponds to the range actually captured by the camera 502 within the display surface of the LED wall 505. The captured area video vBC is a video that expresses the scene seen when the camera 502 is actually positioned at a viewpoint corresponding to the position, capturing direction, and viewing angle of the camera 502.

[0084] More specifically, for the shooting area video vBC, three-dimensional (3D) background data (which is a 3D model) as a background is prepared, and the 3D background data is sequentially rendered in real time based on the viewpoint position of the camera 502 .

[0085] Furthermore, in reality, the range of the captured area video vBC is slightly wider than the range captured at that point in time by the camera 502. This is to prevent the video of the outer frustum from being reflected due to a rendering delay, or to avoid the influence of diffracted light from the video of the outer frustum when the captured range slightly changes due to panning, tilting, zooming, etc. of the camera 502.

[0086] In this way, the video of the shooting area video vBC, which has been rendered in real time in this way, is synthesized with the video of the outer frustum. Although there are cases where the video of the outer frustum used in the background video vB is pre-rendered based on the 3D background data, and cases where the video is rendered in real time for each frame or at intervals, the entire background video vB is generated by embedding the video of the shooting area video vBC (inner frustum) into a portion of the video of the outer frustum.

[0087] In addition, although there is a case where the video of the outer frustum is rendered per frame similarly to the inner frustum, here, a still video will be used as an example, and in the following description, the case of rendering only the first frame of the video of the outer frustum will be mainly described as an example.

[0088] Accordingly, even when the camera 502 moves to the front / back / left / right side or performs a zoom operation, the background of the range filmed together with the performer 510 is captured as a video according to the change in the viewpoint position or field of view (FOV) accompanying the actual movement of the camera 502.

[0089] like Figure 2 and Figure 3 As shown, a monitor video vM including a performer 510 and a background, i.e., a captured video, is displayed on the output monitor 503. The background in the monitor video vM is the capture area video vBC. In other words, the background included in the captured video is a video that has been rendered in real time.

[0090] In this manner, the photographing system 500 according to the embodiment not only displays the background video vB in a plane but also changes the background video vB including the photographing area video vBC in real time, making it possible to photograph a video in the case of actually photographing a landscape.

[0091] Furthermore, by not rendering the entire background video vB displayed in the LED wall 505 in real time but rendering only the shooting area video vBC which is the range reflected by the camera 502 in real time, a plan to reduce the processing load of the system can be performed.

[0092] Here, as virtual production using the filming system 500 to perform filming, a process of producing video content will be described. Figure 4 As shown in FIG, the video content production process is mainly divided into three steps, namely pre-production ST1, production ST2 and post-production ST3.

[0093] Pre-production ST1 is a process of producing 3D background data for displaying background video vB. As described above, background video vB is generated by rendering in real time using 3D background data during imaging. Therefore, 3D background data as a 3D model is produced in advance.

[0094] Examples of technology used to produce 3D background data include full computer graphics (CG), point cloud data scanning, and photogrammetry.

[0095] Full CG is a technology for creating 3D models using computer graphics. This technology requires the most processing and takes the longest time of the three technologies, and is used appropriately when it is desired to set an unrealistic video or a video that is difficult to actually shoot as the background video vB.

[0096] Point cloud data scanning is a technology that generates a 3D model from point cloud data by, for example, measuring the distance to a location using LiDAR, capturing a 360-degree image from the same location using a camera, and then applying the color data captured by the camera to the points whose distances were measured using LiDAR. This technology allows the creation of 3D models in a relatively short time compared to full CG. Furthermore, it is easier to create highly accurate 3D models than with photogrammetry.

[0097] Photogrammetry is a technique for measuring the size and shape of an object by analyzing parallax information from two-dimensional images taken from multiple viewpoints. This technique allows for the creation of 3D models in a short time.

[0098] Furthermore, in the generation of 3D data using photogrammetry, point cloud information acquired by LiDAR can be used.

[0099] In pre-production ST1, for example, a 3D model serving as 3D background data is created using these techniques. Obviously, these techniques can be used in combination. For example, portions of a 3D model created using point cloud data scanning and photogrammetry can be created and synthesized using CG.

[0100] ST2 is made in Figure 1The process of filming is performed in the filming studio shown. As the element technologies in such a case, there are real-time rendering, background display, camera tracking, lighting control, etc.

[0101] like Figure 2 and Figure 3 As shown, real-time rendering is a rendering process for obtaining the shooting area video vBC at each time point (each frame of the background video vB). This is to perform rendering on the 3D background data produced in the pre-production ST1 at each time point according to the viewpoint such as the position of the camera 502.

[0102] By performing real-time rendering in this manner, the background video vB including each frame of the shooting area video vBC is generated and displayed in the LED wall 505 .

[0103] Camera tracking is performed to acquire shooting information using the camera 502, and to track the position information, shooting direction, viewing angle, etc. of the camera 502 at each point in time. By providing this shooting information, including this information, to the rendering engine in association with each frame, real-time rendering based on the viewpoint position of the camera 502, etc., can be performed.

[0104] The shooting information is information associated or corresponding to the video as metadata, and is assumed to include position information of the camera 502 at each frame timing, camera direction, angle of view, focal length, F value (aperture value), shutter speed, lens information, and the like.

[0105] In lighting control, by controlling the lighting state in the photographing system 500, more specifically, controlling the light quantity, light color, lighting direction, etc. of the lamp 580, for example, lighting control is performed according to the time setting and location setting of the scene to be photographed.

[0106] Post-production ST3 refers to various processing performed after shooting, such as video correction, video adjustment, editing, video effects, etc.

[0107] As video correction, color gamut conversion, color matching between the camera and the material, etc. can be performed.

[0108] As adjustments of the video, color adjustment, brightness adjustment, contrast adjustment, etc. can be performed.

[0109] As a clip editor, you can perform cutting, sequence adjustment, time length adjustment, etc.

[0110] As the video effect, synthesis of CG video, special effect video, and the like can be performed.

[0111] Subsequently, the configuration of the photographing system 500 used in producing ST2 will be described. Figure 5It is shown in Figure 1 、 Figure 2 and Figure 3 A block diagram of the configuration of the photographing system 500, an overview of which has been described.

[0112] Figure 5 The shooting system 500 shown in FIG includes the above-mentioned LED wall 505 using multiple LED panels 506, a camera 502, an output monitor 503, and a lamp 580. Figure 5 As shown, the filming system 500 includes a rendering engine 520 , an asset server 530 , a synchronization generator 540 , an operation monitor 550 , a camera tracker 560 , an LED processor 570 , a lighting controller 581 , and a display controller 590 .

[0113] The LED processor 570 is provided corresponding to one or more LED panels 506 , and performs video display driving of the one or more LED panels 506 corresponding thereto.

[0114] The synchronization generator 540 generates a synchronization signal for synchronizing between the frame timing of the display device according to the LED panel 506 and the frame timing of the shooting according to the camera 502, and supplies the generated synchronization signal to each LED processor 570, the camera 502 and the rendering engine 520.

[0115] The camera tracker 560 generates shooting information according to the camera 502 at each frame timing, and supplies the generated shooting information to the rendering engine 520. For example, the camera tracker 560 detects relative position information of the position of the camera 502 relative to the LED wall 505 or a predetermined reference position and the shooting direction of the camera 502 as one piece of shooting information, and supplies this information to the rendering engine 520.

[0116] As a specific detection technology based on the camera tracker 560, there is a method in which reflective plates are randomly arranged on the ceiling, and the position is detected based on the reflected light of infrared light emitted from the camera tracker 560 assembled together with the camera 502 to the reflective plates. In addition, as a detection technology, there is a technology in which the device position of the camera 502 itself is estimated using gyroscope information installed in the platform of the camera 502 or the main body of the camera 502 and image recognition of the video captured by the camera 502.

[0117] Furthermore, there are cases where the angle of view, focal length, F value, shutter speed, lens information, and the like are supplied from the imaging device 502 to the rendering engine 520 as shooting information.

[0118] The asset server 530 is a server that stores the 3D model created in the pre-production ST1, that is, the 3D background data, in a recording medium and can read the 3D model as needed. In other words, the asset server 530 serves as a database (DB) for the 3D background data.

[0119] The rendering engine 520 performs processing to generate the background video vB to be displayed on the LED wall 505. Therefore, the rendering engine 520 reads the required 3D background data from the asset server 530. Then, the rendering engine 520 generates a video of the outer frustum used in the background video vB by rendering the 3D background data in a form viewed from pre-specified spatial coordinates.

[0120] Furthermore, the rendering engine 520 recognizes the viewpoint position and the like of 3D background data using the photographing information supplied from the camera tracker 560 or the camera 502 , and performs rendering of the photographing area video vBC (inner frustum).

[0121] In addition, the rendering engine 520 combines the shooting area video vBC that changes dynamically according to the movement of the camera 502 with the outer frustum to generate the background video vB as one frame of video data. Then, the rendering engine 520 sends the generated one frame of video data to the display controller 590.

[0122] The display controller 590 generates a divided video signal nD obtained by dividing the video data of one frame into video portions to be displayed in each LED panel 506, and transmits the divided video signal nD to each LED panel 506. At this time, the display controller 590 can perform calibration based on individual differences / manufacturing errors in color rendering and the like between display units.

[0123] In addition, without providing the display controller 590 , such processing may be performed by the rendering engine 520 . In other words, the rendering engine 520 may generate the divided video signal nD, perform calibration, and transmit the divided video signal nD to each LED panel 506 .

[0124] Each LED processor 570 drives the LED panel 506 based on the received divided video signal nD, thereby displaying the entire background video vB on the LED wall 505. The background video vB includes a shooting area video vBC rendered according to the position of the camera 502 at that point in time.

[0125] In this manner, the camera 502 can capture the performance of the performer 510 (including the background video vB displayed on the LED wall 505). The video captured by the camera 502 is recorded in a recording medium provided within the camera 502 or in an external recording device (not shown), supplied to the output monitor 503 in real time, and displayed as a monitor video vM.

[0126] In the operation monitor 550, an operation image vOP for controlling the rendering engine 520 is displayed. The engineer 511 can perform necessary settings and operations related to rendering of the background video vB while viewing the operation image vOP.

[0127] The lighting controller 581 controls the light intensity, color, and direction of the light 580. For example, the lighting controller 581 can control the light 580 asynchronously with the rendering engine 520, or synchronously with the capture information and rendering process. Thus, the lighting controller 581 can control the light emission based on instructions from the rendering engine 520, a main controller (not shown), or the like. Furthermore, the lighting controller 581 can control the light 580 from the rendering engine 520.

[0128] exist Figure 6 An example of processing by the rendering engine 520 in the photographing system 500 of such a configuration is shown in FIG.

[0129] In step S10 , the rendering engine 520 reads the 3D background data used at this time from the asset server 530 , and expands the read 3D background data used at this time into the internal work area.

[0130] At this stage, there is a case where a video serving as an outer frustum is generated.

[0131] Thereafter, the rendering engine 520 repeats the processing of step S30 to step S60 until it is determined in step S20 that the display of the background video vB based on the read 3D background data is completed.

[0132] In step S30, the rendering engine 520 acquires photographing information from the camera tracker 560 or the camera 502. Based on this, the position and state of the camera 502 reflected in the current frame are checked.

[0133] In step S40, the rendering engine 520 performs rendering based on the shooting information. In other words, based on the position, shooting direction, and viewing angle of the camera 502 reflected in the current frame, the viewpoint position of the 3D background data is identified and rendering is performed. At this time, video processing can be performed that reflects the focal length, F-number, shutter speed, lens information, etc. Based on this rendering, video data as the shooting area video vBC (inner frustum) can be obtained. The outer frustum can be pre-generated as a fixed video in step S10, or it can be generated for each frame in step S40.

[0134] In step S50, the rendering engine 520 performs a process of combining the outer frustum, which is the entire background video, with the video reflecting the viewpoint of the camera 502, namely, the shooting area video vBC. For example, this process combines the video generated while reflecting the viewpoint of the camera 502 with the video of the entire background rendered at a specific reference viewpoint. Consequently, a single frame of background video vB is generated, which is displayed on the LED wall 505, namely, the background video vB including the shooting area video vBC.

[0135] The processing of step S60 is performed by the rendering engine 520 or the display controller 590. In step S60, the rendering engine 520 or the display controller 590 generates divided video signals nD for each frame of the background video vB, which are divided into videos to be displayed on each LED panel 506. There is a case where calibration is performed. Then, each divided video signal nD is sent to each LED processor 570.

[0136] According to the above-described process, at each frame timing, the background video vB including the shooting area video vBC captured by the camera 502 is displayed on the LED wall 505 .

[0137] Despite Figure 5 Only one camera 502 is shown in FIG. 5 , but a plurality of cameras 502 may be used to perform photography. Figure 7 The configuration example of the case where a plurality of cameras 502a and 502b are used is shown. Each of the cameras 502a and 502b is configured to independently perform shooting in the performance area 501. In addition, the synchronization between the cameras 502a and 502b and the LED processor 570 is maintained by the synchronization generator 540.

[0138] The output monitors 503 a and 503 b are provided corresponding to the cameras 502 a and 502 b and are configured to display videos captured by the corresponding cameras 502 a and 502 b as monitor videos vMa and vMb, respectively.

[0139] Furthermore, camera trackers 560a and 560b are provided corresponding to cameras 502a and 502b, and detect the positions and shooting directions of the corresponding cameras 502a and 502b. Shooting information from camera 502a and camera tracker 560a, as well as shooting information from camera 502b and camera tracker 560b, is sent to the rendering engine 520.

[0140] The rendering engine 520 may perform rendering for acquiring the background video vB for each frame using photographing information of one or both of the camera 502 a side and the camera 502 b side.

[0141] Despite Figure 7 An example using two cameras 502a and 502b is shown in FIG, but shooting may be performed using three or more cameras 502.

[0142] Here, when a plurality of cameras 502 are used to render and display the shooting area video vBC (inner frustum) corresponding to each camera 502 using shooting information, there is a case where the shooting area videos vBC interfere with each other. Figure 7 In the example using two cameras 502a and 502b, as shown, although the shooting area video vBC corresponding to camera 502a is shown, when using the video from camera 502b, the shooting area video vBC corresponding to camera 502b is also required. In this case, simply displaying the shooting area videos vBC corresponding to cameras 502a and 502b would interfere with each other. Therefore, a plan for displaying the shooting area videos vBC is necessary.

[0143] <2. Configuration of Information Processing Equipment>

[0144] Next, we will refer to Figure 8 A configuration example of the information processing apparatus 70 that can be used for pre-production ST1 , production ST2 , and post-production ST3 is described.

[0145] Information processing device 70 is a computer device or the like that can perform information processing, particularly video processing. More specifically, information processing device 70 includes a personal computer, a workstation, a portable terminal device such as a smartphone or tablet, a video editing device, and the like. Furthermore, information processing device 70 may be a computer device configured as a server device or an arithmetic operation device in cloud computing.

[0146] In the case of this embodiment, more specifically, the information processing apparatus 70 can be used as a 3D model production apparatus that produces a 3D model in pre-production ST1.

[0147] Furthermore, the information processing device 70 can also function as the rendering engine 520 or the asset server 530 configuring the shooting system 500 used in producing ST2. Furthermore, the information processing device 70 can also function as the control system, signal processing system, and interface system included in the camera 502.

[0148] Furthermore, the information processing apparatus 70 can also be used as a video editing apparatus that performs various video processes in post-production ST3.

[0149] Figure 8 A central processing unit (CPU) 71 of the illustrated information processing device 70 executes various processes according to a program stored in a read-only memory (ROM) 72 or a nonvolatile memory unit 74 (e.g., an electrically erasable programmable read-only memory (EEP-ROM) or the like) or a program loaded from a storage unit 79 into a random access memory (RAM) 73. In the RAM 73, data and the like required for the CPU 71 to execute various processes and the like are appropriately stored.

[0150] The video processing unit 85 is configured as a processor that performs various video processing. For example, the video processing unit is formed as a processor that can perform one or more processes including 3D model generation processing, rendering DB processing, color brightness adjustment processing and color conversion processing, video editing processing, video analysis / detection processing, etc.

[0151] This video processing unit 85 can be realized using, for example, a CPU which is a unit separate from the CPU 71 , a graphics processing unit (GPU), a general purpose graphics processing unit (GPGPU), an artificial intelligence (AI) processor, or the like.

[0152] In addition, the video processing unit 85 can also be provided as a function inside the CPU 71.

[0153] The CPU 71, the ROM 72, the RAM 73, the nonvolatile memory unit 74, and the video processing unit 85 are interconnected via a bus 83. Furthermore, to this bus 83, an input / output interface 75 is connected.

[0154] An input unit 76 formed of operators and operating devices is connected to the input / output interface 75. For example, as the input unit 76, various operators and operating devices such as a keyboard, mouse, keys, trackball, dial, touch panel, touch pad, remote controller, etc. are assumed.

[0155] The user's operation is detected by the input unit 76 , and a signal corresponding to the input operation is analyzed by the CPU 71 .

[0156] A microphone is also assumed as the input unit 76. Voice spoken by the user can be input as operation information.

[0157] Furthermore, a display unit 77 formed of a liquid crystal display (LCD) or an organic electroluminescence (EL) panel or the like and a voice output unit 78 formed of a speaker or the like are connected to the input / output interface 75 integrally or separately.

[0158] The display unit 77 is a display unit that performs various displays, and is configured using, for example, a display device provided in the casing of the information processing apparatus 70 , a separate display device connected to the information processing apparatus 70 , or the like.

[0159] The display unit 77 performs display of various images, operation menus, icons, messages, and the like on a display device based on instructions from the CPU 71 , in other words, performs display as a graphical user interface (GUI).

[0160] A storage unit 79 and a communication unit 80 configured by a hard disk drive (HDD), a solid state drive (SSD), or the like may be connected to the input / output interface 75 .

[0161] The storage unit 79 can store various data and programs. In the storage unit 79, a DB can be configured.

[0162] The communication unit 80 performs communication processing via a transmission line such as the Internet, and performs communication with various devices such as an external DB, an editing device, an information processing device, etc. using wired / wireless communication, bus communication, etc. For example, in the case of the information processing device 70 used in post-production ST3, access to a DB such as the asset server 530 can be performed using the communication unit 80. In this way, the information processing device 70 can obtain the captured video vC captured by the camera 502 (see Figure 12 etc.) and the accompanying metadata MT (see Figure 30 wait).

[0163] Furthermore, a drive 81 is connected to the input / output interface 75 as needed, and a removable recording medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted.

[0164] By using the drive 81, video data, various computer programs, and the like can be read from the removable recording medium 82. The read data is stored in the storage unit 79, or the video or voice included in the data is output to the display unit 77 and the voice output unit 78. In addition, the computer program and the like read from the removable recording medium 82 are installed in the storage unit 79 as needed.

[0165] In this information processing apparatus 70, for example, software for processing according to the present embodiment can be installed through network communication using the communication unit 80 or the removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0166] <3. Lighting form>

[0167] The form of the lighting in production ST2 will be described. As described above, the lamp 580 exists as a lighting device in the photographing system 500. As the lamp 580, for example, an LED lamp is used.

[0168] At the same time, the LED wall 505 displaying the background video vB emits video light from each LED panel 506 and can thus be used as lighting for the foreground of the performer 510, for example.

[0169] Figure 9 A. Figure 9 B. Figure 10 A and Figure 10 B shows an example of a lighting form.

[0170] Figure 9 A is a case where the LED wall 505 is a flat wall type. In this case, it is difficult for the light from the LED wall 505 to be emitted to the front of the performer 510, and therefore, it can be considered to use the lamp 580 as the lighting for the performer 510.

[0171] Figure 9 B is a case where LED wall 505 is a cave-type. For example, there is a case where LED wall 505 widely covers the vicinity of performer 510, has a ceiling, and displays background video vB near and above. In this cave-type case, the light from LED wall 505 serves as sufficient illumination for performer 510, and therefore, lamp 580 may not be used.

[0172] Figure 10 A is a case where the LED wall 505 is formed in a curved wall type, where the lateral side is surrounded from the rear side of the performer 510. In this case, light from the LED wall 505 can be emitted to the performer 510. Therefore, as lighting, both the lamp 580 and the light from the LED wall 505 can be used.

[0173] Furthermore, for example, in the case of a curved wall type, in order to make the reflection of lighting and scenery on objects (for example, reflection on mirrors, glass, etc. as objects existing in the performance area 501) similar to the actual scenery, as shown in FIG. Figure 10As shown in FIG. 5B , a rectangular LED panel 505 a may be installed on the ceiling as part of the LED wall 505. In addition, there is also a case where an LED panel covering the entire ceiling is formed to form a nearly cave-like shape.

[0174] In addition, if Figure 1 As shown, in a flat wall type rather than a curved wall type, there are cases where LED panels can be installed on the rear side and the lateral side. In such cases, there are examples where LED panels 505a are installed on the ceiling and examples where LED panels cover the entire ceiling. In addition, in such an LED wall 505, there are cases where light from both the lamp 580 and the LED wall 505 can be used as lighting.

[0175] For example, as described above, as lighting, there are a case where the lamp 580 is used, a case where only the LED wall 505 is used, and a case where both the lamp 580 and the LED wall 505 are used as lighting.

[0176] The above description is merely an example. While the lighting form can be determined according to the type of LED wall 505, there are cases where it is not necessary to determine the lighting form according to the type of LED wall 505. For example, in the case of a cave-type LED wall 505, there are cases where lamps 580 are used, and in the case of a flat-wall-type LED wall 505, there are cases where the LED wall 505 is used as lighting.

[0177] Meanwhile, in the photographing system 500, for example, Figure 9 As shown in A, it is assumed that the lamp 580 is used as lighting, and in the captured video data (captured video vC to be described below) captured by the camera 502, there is no problem in color reproduction of the video of the captured portion of the performer 510 and the like.

[0178] This is because the lamp 580 (e.g., an LED lamp) used for lighting has a spectrum characteristic close to that of natural light and can obtain natural color reproduction. In other words, this is because the camera 502 is generally designed so that natural color reproduction can be obtained when lighting is performed using the lamp 580.

[0179] The LED wall 505 used to display the background video vB has spectral characteristics that are very different from natural light.

[0180] exist Figure 11, the spectral characteristics of natural light (sunlight) are represented by a dotted line 110, the spectral characteristics of the lamp 580 (LED lamp for lighting) are represented by a dashed line 111, and the spectral characteristics of the light from the LED wall 505 are represented by a solid line 112. The LED lamp for lighting has a component close to that of sunlight in a wide wavelength region. On the other hand, the light from the LED wall 505 has components only in the wavelength regions of R (red), G (green), and B (blue). Therefore, for example, the component near 580nm (near yellow) is small. Based on this, when the LED wall 505 is set as lighting, the color reproducibility of the face and skin of the performer 510 in the captured video vC is reduced. More specifically, the skin color becomes redder, etc.

[0181] Therefore, in this embodiment, the image pickup device 502 or the information processing apparatus 70 used in post-production ST3 is configured to be able to perform switching of the color conversion process according to lighting.

[0182] For example, the imaging device 502 or the information processing apparatus 70 used in post-production is configured so that, when the lamp 580 is used as lighting, a first color conversion process can be performed using first color setting parameters set for the lamp 580, and, when the LED wall 505 is used as lighting, a second color conversion process can be performed using second color setting parameters set for the LED wall 505. In other words, the imaging device 502 or the information processing apparatus 70 is configured to selectively perform the first and second color conversion processes.

[0183] Furthermore, the camera 502 or the information processing apparatus 70 is configured to also be capable of executing the third color conversion process performed using both the first color setting parameters and the second color setting parameters.

[0184] <4. Configuration and Processing Example of Image Capture Device>

[0185] Hereinafter, the color conversion processing performed by the camera 502 will be described. Figure 12 The configuration of the imaging device 502 is described.

[0186] The camera 502 includes a lens unit 20 , an image sensor 21 , a user interface unit 22 (hereinafter referred to as “UI unit 22 ”), a communication unit 23 , a recording control unit 24 , a control unit 25 , a memory unit 26 , a display unit 27 , and a signal processing unit 30 .

[0187] The lens unit 20 is configured to include various lenses such as an incident end lens, a zoom lens, a focus lens, a condenser lens, etc. and an aperture mechanism, which performs exposure control by adjusting the opening amount according to the lens and the aperture (aperture) so that sensing is performed in a state where the signal charge is not saturated and is within the dynamic range.

[0188] The lens unit 20 is configured as, for example, an interchangeable lens. Here, the lens unit may be configured as a type that is integrated with the main body of the imaging device 502.

[0189] The image sensor 21 includes, for example, a charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) type sensing element and a signal processing circuit for photoelectrically converting signals. Using the image sensor 21 having a two-dimensional arrangement of sensing elements, light from a subject incident through the lens unit 20 is photoelectrically converted. Then, for example, correlated double sampling (CDS) processing, automatic gain control (AGC) processing, and analog / digital (A / D) conversion processing are performed on the electrical signals resulting from the photoelectric conversion. The image sensor 21 outputs the video data acquired through this processing to the signal processing unit 30.

[0190] The signal processing unit 30 is configured using, for example, a microprocessor dedicated to digital signal processing such as a digital signal processor (DSP), a microcomputer.

[0191] This signal processing unit 11 performs various signal processes on the video data sent from the image sensor 21 in accordance with a control signal SG from the control unit 25 .

[0192] The signal processing unit 30 includes a sensor signal processing unit 31, a white balance unit 32, a color separation unit 33, a matrix unit 34, and a development processing unit 35. Hereinafter, "white balance" will be referred to as "WB." For example, the white balance unit 32 will be referred to as "WB unit 32." Furthermore, in the figure, "matrix" will be referred to as "MTX."

[0193] The sensor signal processing unit 31 performs black level adjustment processing, bit allocation adjustment processing, defective pixel correction processing, and the like on the video data from the image sensor 21 .

[0194] The WB unit 32 performs white balance adjustment processing according to the color setting parameters. The details will be described below.

[0195] In addition, the color setting parameters described here are the value of color temperature (in Kelvin) and the tint value. Hereinafter, the color setting parameters will be expressed as "color temperature KL" and "tint value TT".

[0196] Figure 13The CIE 1931 xy chromaticity diagram is shown, along with a blackbody radiation curve 120 and an arrow 121 indicating the direction of its normal line. Color temperature KL is a unit that uses a quantitative value to express the color of light emitted by a light source and is the absolute temperature represented by blackbody radiation curve 120. Tone value TT is a parameter used to adjust hue and chromaticity and is a control value used to correct blackbody radiation curve 120 in the direction indicated by arrow 121.

[0197] In other words, in the color adjustment process, a color on the blackbody radiation curve 120 is specified by using the color temperature KL, and a deviation from the blackbody radiation curve 120 is specified by using the hue value TT.

[0198] The color separation unit 33 performs color separation processing on the video data after white balance adjustment. For example, when the color filters of the pixels of the image sensor 21 have a Bayer arrangement, the color separation unit 33 is configured as a debayer processing unit. Furthermore, when mosaic filters are used, the color separation unit 33 is configured as a demosaic processing unit. Furthermore, when other color filters such as stripe filters are used, for example, the color separation unit 33 is also configured to perform color separation depending on the form of the color filters.

[0199] The matrix unit 34 performs matrix color conversion processing using, for example, 3×3 matrix coefficients.

[0200] The matrix unit 34 performs processing based on the color temperature KL and the tone value TT as color setting parameters.

[0201] exist Figure 12 In the illustrated signal processing unit 30 , the WB unit 32 and the matrix unit 34 perform color conversion processing for color reproduction and color generation.

[0202] The development processing unit 35 performs some or all of gamma processing, lookup table (LUT) processing, resolution conversion processing, encoding processing for recording or communication, and the like, thereby performing conversion into video data of a final output format.

[0203] For example, the video data on which the encoding process for recording has been performed is recorded on a recording medium by the recording control unit 24 .

[0204] In addition, the video data on which the encoding process for communication has been performed is transmitted from the communication unit 23 to another device. Figure 5 In the illustrated configuration, video data is sent to a rendering engine 520 .

[0205] Furthermore, the video data on which resolution conversion of the monitor or the like has been performed is supplied to the output monitor 503 as the above-mentioned monitor video vM.

[0206] In the present disclosure, such video data acquired by the camera 502 is collectively referred to as “captured video vC”.

[0207] Furthermore, for example, video data at the output stage of the sensor signal processing unit 31 may be recorded on a recording medium as raw video data vCraw by the recording control unit 24 and transmitted from the communication unit 23 .

[0208] The raw video data vCraw is video data having a frame arranged according to the pixel arrangement of the image sensor 21, wherein a color filter is provided in each pixel. Although there are some different cases, this means that the raw video data is video data that can be returned to the original pixel arrangement of the image sensor 21. Therefore, the video data maintained in the pixel arrangement read from the image sensor 21 is also the raw video data vCraw, and for example, video data divided into four channels such as G1 (green first pixel), G2 (green second pixel), R and B for compression is included in the raw video data vCraw. In addition, the frame corresponding to the pixel arrangement of the image sensor 21 does not need to correspond to the pixel arrangement of all pixels of the image sensor 21. In addition, not only R, G and B pixels but also white pixels may be included.

[0209] Although such raw video data vCraw is expressed differently, the raw video data is one of the captured videos vC.

[0210] In addition, the captured video vC (including the original video data vCraw) also includes metadata MT of the video data of the captured video vC (see Figure 30 wait).

[0211] The communication unit 23 performs data communication or network communication with an external device in a wired or wireless manner based on a control signal SG from the control unit 25. For example, in the case of the capture system 500, the captured video vC (including metadata MT) is transmitted to the rendering engine 520. The captured video vC transmitted to the rendering engine 520 is stored in the asset server 530 for post-production, for example.

[0212] Furthermore, the communication unit 23 can transmit the captured video vC not only to the rendering engine 520 but also to various external display devices, recording devices, playback devices, and the like. Furthermore, the communication unit 23 can function as a network communication unit. For example, the communication unit can communicate using various networks such as the Internet, a home network, or a local area network (LAN), and can transmit and receive various data to and from servers, terminals, and the like over the network. For example, it is conceivable that the captured video vC can be transmitted as a video stream.

[0213] The recording control unit 24 performs processing of recording the captured video vC (including metadata MT) on a recording medium formed of, for example, a nonvolatile memory, based on a control signal SG from the control unit 25 .

[0214] The form of the recording control unit 24 can be considered in various ways. For example, the recording control unit 24 can be configured to perform recording processing on a flash memory built into the camera 502, or can be configured using a memory card (for example, a portable flash memory) that can be attached to / detached from the camera 502 and an access unit that accesses the memory card to store or read data. In addition, as a form built into the camera 502, the recording control unit can be implemented using an HDD, an SSD, or the like.

[0215] The display unit 27 performs various displays for the camera operator 512. The display unit 27 is, for example, a monitor or an electronic viewfinder (EVF) monitor on the housing of the camera 502. The display unit 27 displays various information such as the captured video vC and a user interface. For example, the display unit 27 displays various operation menus, icons, and messages on a screen as a graphical user interface (GUI) based on a control signal SG from the control unit 25.

[0216] The UI unit 22 includes a display, a touch sensor, etc. that performs image display for the user (cameraman 512, etc.), and outputs operation information according to various operations such as tap operations, drag operations, etc. of the user on the operation image to the control unit 16.

[0217] Furthermore, the UI unit 22 may be formed by providing a touch panel on the display unit 27. Furthermore, the UI unit 22 may be configured as an operation device separate from the camera 502.

[0218] The control unit 25 is configured using a microcomputer including a CPU, and performs overall control of the imaging device 502 .

[0219] The control unit 25 performs, for example, recording control of moving images and still images corresponding to the operation of the camera 502, control of shutter speed, gain, aperture, etc., focus control, zoom control, instructions for various signal processing in the signal processing unit 11, reproduction operation control of recorded image files, communication control, display control, etc.

[0220] The memory unit 26 stores information used for processing performed by the control unit 25 etc. As the illustrated memory unit 26 , for example, a ROM, a RAM, a flash memory, etc. are generally shown.

[0221] The memory unit 26 may be a memory area built in a microchip serving as the control unit 25 , or may be configured using a separate memory chip.

[0222] The ROM, flash memory, etc. of the memory unit 26 store programs and the like used by the control unit 25. In addition to an operating system (OS) for the CPU to control each unit and content files such as image files, the ROM, flash memory, etc. also store application programs, firmware, etc. used for various operations.

[0223] By executing this program, the control unit 25 controls the entire imaging device 502 .

[0224] The RAM of the memory unit 26 is used as a work area of the control unit 25 by temporarily storing data, programs, and the like used when performing various data processing executed by the CPU of the control unit 25 .

[0225] The WB unit 32 and the matrix unit 34 of the image pickup device 502 configured as described above will be described.

[0226] exist Figure 14 The configuration of the WB unit 32 is shown in . The WB unit 32 is configured to include a coefficient setting unit 41 and multipliers 42 , 43 , and 44 .

[0227] For example, the color temperature KL and the tone value TT are input from the control unit 25 to the coefficient setting unit 41. The control unit 25 selects the color temperature KL and the tone value TT as color setting parameters stored in the memory unit 26, for example, and indicates the values thereof to the WB unit 32. Alternatively, the control unit 25 indicates the color temperature KL and the tone value TT indicated by the user operation using the UI unit 22 to the WB unit 32.

[0228] The coefficient setting unit 41 calculates R gain, G gain, and B gain for white balance adjustment by performing a function operation based on the supplied color temperature KL and tone value TT.

[0229] The R value (Rin), the G value (Gin), and the B value (Bin) in the video data from the sensor signal processing unit 31 are input to the multipliers 42 , 43 , and 44 , respectively.

[0230] Multiplier 42 multiplies the R value (Rin) by the R gain and outputs the R value (Rout) after WB adjustment. Multiplier 43 multiplies the G value (Gin) by the G gain and outputs the G value (Gout) after WB adjustment. Multiplier 44 multiplies the B value (Bin) by the B gain and outputs the B value (Bout) after WB adjustment.

[0231] exist Figure 15The configuration of the matrix unit 34 is shown in . The matrix unit 34 is configured to include a coefficient setting unit 45 and a matrix coefficient processing unit 46.

[0232] Similar to the WB unit 32 , the color temperature KL and the tone value TT are input from the control unit 25 to the coefficient setting unit 45 .

[0233] The coefficient setting unit 45 calculates nine matrix coefficients for 3×3 operations of R values, G values, and B values by performing function operations based on the supplied color temperature KL and tone value TT.

[0234] The R value (Rin), G value (Gin), and B value (Bin) in the video data from the color separation unit 33 are input to the matrix coefficient processing unit 46. The matrix coefficient processing unit 46 performs a matrix operation on the R value (Rin), G value (Gin), and B value (Bin) using matrix coefficients, and outputs the R value (Rout), G value (Gout), and B value (Bout) after color conversion.

[0235] Color conversion processing according to color reproduction of lighting using the image pickup device 502 having such a configuration will be described below.

[0236] In addition, in the following description, reference is made to Figure 16 、 Figure 17 、 Figure 18 、 Figure 28 、 Figure 30 、 Figure 32 and Figure 34 , as the configuration of the imaging device 502, only the lens unit 20, the image sensor 21, the WB unit 32, and the matrix unit 34 are extracted and shown.

[0237] Figure 16 Shown in use Figure 9 FIG. 5A shows a case where a light 580 is used as lighting for a performer 510 in a photograph of a flat wall type LED wall 505 shown in FIG.

[0238] In this case, the color temperature KL1 and the tone value TT1 as the color temperature KL and the tone value TT, which are color reproduction coefficients optimized for the illumination light of the lamp 580 , are indicated by the control unit 25 to the WB unit 32 and the matrix unit 34 .

[0239] According to this, the WB unit 32 performs white balance adjustment processing corresponding to the illumination light of the lamp 580 , and the matrix unit 34 performs matrix color conversion processing corresponding to the illumination light of the lamp 580 .

[0240] According to this, for example, the monitor video vM displayed on the output monitor 503 based on the shot video vC becomes a video in which the skin of the performer 510 has natural chromaticity as under natural light.

[0241] Figure 17 Shown in use Figure 9 In the shooting of the cave-type LED wall 505 shown in B, the LED wall 505 is used as lighting for the performer 510.

[0242] In this case, the color temperature KLw and the tone value TTw, which are color reproduction coefficients optimized for the light of the LED wall 505 , are indicated as the color temperature KL and the tone value TT by the control unit 25 to the WB unit 32 and the matrix unit 34 .

[0243] According to this, the WB unit 32 performs white balance adjustment processing corresponding to the illumination light according to the LED wall 505 , and the matrix unit 34 performs matrix color conversion processing corresponding to the illumination light according to the LED wall 505 .

[0244] According to this, for example, the monitor video vM displayed on the output monitor 503 based on the shot video vC becomes a video in which the skin of the performer 510 has natural chromaticity as under natural light.

[0245] In other words, the control unit 25 changes the color temperature KL and the hue value TT, which are color setting parameters instructed to the WB unit 32 and the matrix unit 34, according to the lighting conditions at the time of shooting. Consequently, whether the lamp 580 is used as lighting or the LED wall 505 is used as lighting, the captured video vC can be configured as a video with natural chromaticity.

[0246] Subsequently, the case where the lamp 580 and the LED wall 505 are used together for lighting will be described. Figure 10 A and Figure 10 In the case of the curved wall type shown in B, lighting is considered to be used together. In addition, as Figure 18 As shown, in the case where three plane-type walls are provided on the rear and two sides of the LED wall 505, the lamp 580 and the LED wall 505 are also considered to be used together as lighting.

[0247] In addition, the use of the light of the lamp 580 and the light of the LED wall 505 together as lighting is not limited to the above case. Figure 9 The plane wall type shown in A, Figure 9 In the case where a ceiling is provided in a cave type, a curved wall type, etc. as shown in B, there is a case where the light of the lamp 580 and the light of the LED wall 505 are used together for lighting. Figure 18This is merely an example showing a case where the light of the lamp 580 and the light of the LED wall 505 are used together as lighting.

[0248] When the light of the lamp 580 and the light of the LED wall 505 are used together as lighting, the control unit 25 indicates two pairs of color temperature KLl and hue value TTl (which are color reproduction coefficients optimized for the lighting light of the lamp 580) and color temperature KLw and hue value TTw (which are color reproduction coefficients optimized for the light of the LED wall 505) to the WB unit 32 and the matrix unit 34.

[0249] Furthermore, the control unit 25 instructs the mixing ratio MixR to the WB unit 32 and the matrix unit 34. The mixing ratio MixR is a ratio that reflects the color setting parameters of the lamp 580 and the color setting parameters of the LED wall 505 in the color conversion process.

[0250] For example, the control unit 25 reads the mixing ratio MixR set according to the lighting device used together in the memory unit 26 and can instruct the WB unit 32 and the matrix unit 34 .

[0251] Alternatively, for example, by acquiring measured values of the illumination light intensities of the lamp 580 and the LED wall 505 , the control unit 25 may give an instruction with the mixing ratio MixR set according to the ratio thereof.

[0252] Furthermore, the control unit 25 can give an instruction at the mixing ratio MixR set according to the operation of the user using the UI unit 22 .

[0253] Then, the WB unit 32 performs white balance adjustment processing (mixed WB processing) in which the color temperatures KL1 and KLw and the tone values TT1 and TTw are reflected according to the mixing ratio MixR.

[0254] The matrix unit 34 performs a matrix color conversion process (mixing MTX process) in which the color temperatures KL1 and KLw and the tone values TT1 and TTw are reflected according to the mixing ratio MixR.

[0255] exist Figure 19 The configuration of the WB unit 32 in the case of performing such processing is shown in . The WB unit 32 is configured to include coefficient setting units 41a and 41b, a mixing coefficient setting unit 47, and multipliers 42, 43, and 44.

[0256] The color temperature KLw and the tone value TTw are input to the coefficient setting unit 41a, which calculates R gain Rg1, G gain Gg1, and B gain Bg1 for white balance adjustment corresponding to the illumination light of the LED wall 505 using a function operation based on the color temperature KLw and the tone value TTw.

[0257] The color temperature KL1 and the tone value TT1 are input to the coefficient setting unit 41b, which calculates R gain Rg2, G gain Gg2, and B gain Bg2 for white balance adjustment corresponding to the illumination light of the lamp 580 using a function operation based on the color temperature KL1 and the tone value TT1.

[0258] The mixing coefficient setting unit 47 performs α mixing of the R gain, G gain, and B gain at a mixing ratio MixR, and sets the R gain, G gain, and B gain used for white balance adjustment.

[0259] In other words, the mixing coefficient setting unit 47 sets the R gain by mixing the R gains Rg1 and Rg2 at a mixing ratio MixR. Furthermore, the mixing coefficient setting unit 47 sets the G gain by mixing the G gains Gg1 and Gg2 at a mixing ratio MixR. Furthermore, the mixing coefficient setting unit 47 sets the B gain by mixing the B gains Bg1 and Bg2 at a mixing ratio MixR.

[0260] The R gain, G gain, and B gain set in this manner become coefficients of the multipliers 42, 43, and 44. Then, the R value (Rin), G value (Gin), and B value (Bin) in the video data from each sensor signal processing unit 31 are multiplied by the R gain, G gain, and B gain, and the R value (Rout), G value (Gout), and B value (Bout) after WB adjustment are output to the multipliers 42, 43, and 44.

[0261] exist Figure 20 The configuration of the matrix unit 34 is shown in . The matrix unit 34 is configured to include coefficient setting units 45a and 45b, a mixing coefficient setting unit 48, and a matrix coefficient processing unit 46.

[0262] The color temperature KLw and the tone value TTw are input to the coefficient setting unit 45a, which calculates nine matrix coefficients for 3×3 matrix color conversion corresponding to the illumination light of the LED wall 505 using a function operation based on the color temperature KLw and the tone value TTw.

[0263] The color temperature KL1 and the tone value TT1 are input to the coefficient setting unit 45b, which calculates nine matrix coefficients for 3×3 matrix color conversion corresponding to the illumination light of the lamp 580 using a function operation based on the color temperature KL1 and the tone value TT1.

[0264] The mixing coefficient setting unit 48 performs α mixing of the nine coefficients from each of the coefficient setting units 45 a and 45 b at a mixing ratio Mix R, thereby setting nine matrix coefficients. These matrix coefficients are used by the matrix coefficient processing unit 46.

[0265] The matrix coefficient processing unit 46 performs matrix operations on the R value (Rin), G value (Gin), and B value (Bin) in the video data from the color separation unit 33 using matrix coefficients, and outputs the R value (Rout), G value (Gout), and B value (Bout) after color conversion.

[0266] By configuring the WB unit 32 and the matrix unit 34 as described above, it is possible to perform Figure 18 Mixed WB treatment and mixed MTX treatment as shown in .

[0267] By setting the mixing ratio MixR at the ratio of the light intensities of the illumination light of the LED wall 505 and the illumination light of the lamp 580 , appropriate color reproduction processing is performed, and a captured video vC of natural chromaticity can be acquired.

[0268] Furthermore, the mixing ratio MixR does not necessarily need to be set precisely to the ratio of the illumination light intensities of the LED wall 505 and the lamp 580. The mixing ratio may be a ratio for causing appropriate color reproduction to a certain extent. Furthermore, by setting the mixing ratio MixR to deviate from the illumination light intensity ratio, color reproduction can be performed actively.

[0269] Although the case where illumination is used together has been described above, it is assumed that the WB unit 32 and the matrix unit 34 have Figure 19 and Figure 20 In the case of the configuration shown in Figure 16 In the case corresponding only to the illumination light of the lamp 580 as shown in , the mixing ratio MixR between the color temperature KLw and the color temperature KL1 and between the hue value TTw and the hue value TT1 can be configured to be 0:100.

[0270] In addition, in Figure 17 In the case corresponding only to the lighting light of the LED wall 505 , the mixing ratio MixR between the color temperature KLw and the color temperature KL1 and between the hue value TTw and the hue value TT1 can be configured to be 100:0.

[0271] exist Figure 18 In the case of the illustrated configuration, it can be considered that the control unit 25 sets the color temperatures KLw and KL1, the tone values TTw and TT1, and the mixing ratio MixR according to the user operation using the UI unit 22.

[0272] For example, the UI unit 22 performs the following operations: Figure 21 The GUI display 50 shown in FIG. 50 includes a color temperature operator 51 and a hue operator 53 corresponding to the LED wall 505 , a color temperature operator 52 and a hue operator 54 corresponding to the lamp 580 , and a mixing ratio operator 55 .

[0273] Furthermore, such an operator may not be a touch operator using a display, but may be an operator using an actual physical joystick, slider, dial, or the like.

[0274] For example, the color temperature operator 51 and the tint operator 53 corresponding to the LED wall 505 are set to color temperature and tint values for optimizing color reproduction when using the illumination light of the LED wall 505 as initial values, and the user can arbitrarily change these values from the initial values.

[0275] For example, the color temperature operator 51 and the tint operator 53 corresponding to the lamp 580 are set to color temperature and tint values for optimizing color reproduction using illumination light of the lamp 580 as initial values, and the user can arbitrarily change these values from the initial values.

[0276] In the mixing ratio operator 55 , the mixing ratio MixR can be arbitrarily set between 0:100 and 100:0.

[0277] The control unit 25 acquires operation information from such UI unit 22 , determines the color temperatures KLw and KL1, the tone values TTw and TT1, and the mixing ratio MixR, and instructs the WB unit 32 and the matrix unit 34 .

[0278] For example, in the case where only the lamp 580 is used for lighting, as shown in FIG. Figure 22 As shown, the user operates the mixing ratio operator 55 to the lamp (LED lamp) side. Then, the WB unit 32 and the matrix unit 34 perform color conversion according to the color temperature KL1 and the tone value TT1.

[0279] In addition, in the case where only the LED wall 505 is used for lighting, as shown in FIG. Figure 23 As shown, the user operates the mixing ratio operator 55 to the side of the LED wall 505. Then, the WB unit 32 and the matrix unit 34 perform color conversion according to the color temperature KLw and the hue value TTw.

[0280] In addition, when the LED wall 505 and the lamp 580 are used together as lighting, for example, the lighting light intensity ratio is 50:50, as shown in FIG. Figure 24 As shown, the user operates the mixing ratio operator 55 to an intermediate value (e.g., 50:50). Then, the WB unit 32 and the matrix unit 34 perform color conversion so that the color temperatures KL1 and KLw, and the hue values TT1 and TTw, respectively, reflect 50%. Obviously, if the ratio of the illumination light intensities is different, the mixing ratio operator 55 can be operated accordingly.

[0281] According to the above operation, a captured video vC with natural chromaticity can be obtained.

[0282] In actual shooting scenes, the ratio of the illumination light intensities of the LED wall 505 and the lamp 580 is not always measured. Therefore, for example, it is assumed that the user selects the mixing ratio operator 55 while viewing the monitor video vM. Figure 24 The mixing ratio operator 55 is operated in the state set to the intermediate value shown, and the state is adjusted to a state in which color reproducibility is considered to be optimal.

[0283] Furthermore, the operation of the blending ratio operator 55 can be performed not for natural chromaticity color reproduction but for positive color generation. By operating the blending ratio operator 55, a captured video vC having target chromaticity instead of natural chromaticity can be obtained in video production.

[0284] Furthermore, by also performing the operations of the color temperature operators 51 and 52 and the hue operators 53 and 54 , color generation can be performed.

[0285] It will be described how the WB unit 32 and the matrix unit 34 can perform the following operations: Figure 18 An example of processing by the imaging device 502 of the configuration of the hybrid processing is shown.

[0286] Figure 25 FIG. 5 shows a process of setting color setting parameters executed by the control unit 25 of the camera 502. For example, at a point in time before shooting starts or at a point in time when the lighting form changes, the control unit 25 is considered to execute Figure 25 The processing shown.

[0287] In step S101, the control unit 25 determines the lighting situation. For example, if system setting information is transmitted between devices in the imaging system 500 and the form of the LED wall 505 and the status of the lights 580 are perceived, the control unit 25 can determine the lighting situation for the image. Furthermore, if the operator inputs information regarding the lighting situation, such as a lighting pattern, to the imaging device 502, the control unit 25 checks the lighting pattern and other information.

[0288] As a result of the illumination determination, if it is determined that only the lamp 580 is used as the illumination for this shooting, the control unit 25 advances the process from step S102 to step S110, and reads the color temperature KL1 and tone value TT1 for color reproduction corresponding to the lamp 580 from the memory unit 26, for example. Then, in step S111, the control unit 25 instructs the WB unit 32 and the matrix unit 34 on the color temperature KL1 and tone value TT1 as color setting parameters. Furthermore, as the mixing ratio MixR in this case, the color temperature KL1 and tone value TT1 are set to 100%.

[0289] As a result of the lighting determination, if it is determined that only the LED wall 505 is used as the lighting for this shooting, the control unit 25 advances the process from step S103 to step S120 and, for example, reads the color temperature KLw and tone value TTw for color reproduction corresponding to the LED wall 505 from the memory unit 26. Then, in step S121, the control unit 25 instructs the WB unit 32 and the matrix unit 34 on the color temperature KLw and tone value TTw as color setting parameters. Furthermore, as the mixing ratio MixR in this case, the color temperature KLw and tone value TTw are set to 100%.

[0290] As a result of the lighting judgment, when it is identified that both the wall 582 and the LED wall 505 are used as lighting in this shooting, the control unit 25 causes the processing to proceed from step S104 to step S130, and reads, for example, the color temperature KLl and tint value TTl for color reproduction corresponding to the lamp 580 and the color temperature KLw and tint value TTw for color reproduction corresponding to the LED wall 505 from the memory unit 26.

[0291] Furthermore, in step S131, the control unit 25 sets the mixing ratio MixR. For example, the ratio of the illumination light intensity of the LED wall 505 and the lamp 580 may be obtained, and the mixing ratio MixR may be set based on this illumination light intensity ratio. If the illumination light intensity ratio is unclear, the mixing ratio MixR is set to MixR = 50:50, a preset ratio, or the like.

[0292] Then, in step S132, the control unit 25 instructs the color temperature KL1, tone value TT1, color temperature KLw, and tone value TTw as color setting parameters to the WB unit 32 and the matrix unit 34. In addition, the control unit 25 also gives instructions regarding the mixing ratio MixR set in step S131.

[0293] For example, according to the above-described processing, the signal processing unit 30 of the image pickup device 502 is set to a state in which color conversion processing for appropriate color reproduction according to the lighting form at the time of shooting is performed at a point in time before shooting.

[0294] If the result of the lighting determination is unclear, the control unit 25 ends the process from step S104. Figure 25 processing.

[0295] Although the processing example of parameter setting is described above, as a modified example, for example, it is conceivable to perform only Figure 16 The status of the control light 580 shown in Figure 17The processing example of switching between the states of the control LED wall 505 shown in FIG. In other words, the processing example is an example in which the mixing process is not performed. In this case, the processing example becomes Figure 25 The processing example in which steps S104, S130, S131, and S132 are not provided.

[0296] In addition, despite Figure 25 In the example, the control unit 25 automatically advances the process to the parameter setting process based on the judgment results of steps S102, S103, and S104. However, the control unit 25 may be configured to suggest parameter settings to the user. For example, the control unit 25 displays a recommendation of color setting parameters on the display unit 27 based on the lighting judgment of step S101. Based on this, the user can perform parameter setting operations using the UI unit 22.

[0297] As a processing example in which mixing processing is also performed, in the parameter setting processing, a processing example can be considered in which, while the color temperature KLl, the tone value TTl, the color temperature KLw, and the tone value TTw are continuously indicated to the WB unit 32 and the matrix unit 34, the mixing ratio MixR is set according to the result of the lighting judgment, and the mixing ratio MixR is indicated to the WB unit 32 and the matrix unit 34.

[0298] exist Figure 25 In the processing of and the processing of the modified examples thereof, although parameter setting for the WB unit 32 and the matrix unit 34 is performed, there is a case where the parameter setting is changed according to an operation using the UI unit 22.

[0299] For example, the control unit 25 executes Figure 26 In step S150, the control unit 25 monitors the user operation on the GUI display 50 and checks whether an operation of one of the color temperature operators 51 and 52, the tint operators 53 and 54, or the mixing ratio operator 55 has been performed.

[0300] If any of the operators has been executed, the control unit 25 proceeds to step S151 and performs parameter change. In other words, the control unit 25 changes the setting of one of the color temperature KL1, the tone value TT1, the color temperature KLw, the tone value TTw, and the mixing ratio MixR to the value set for the operator, and instructs the WB unit 32 and the matrix unit 34 to change the parameter.

[0301] According to such processing, the parameters of the color conversion processing of the signal processing unit 30 can be changed according to the operation of the operator at any time. For example, at the time of preview of shooting, etc., the operator can perform adjustments of color reproducibility and color generation by performing operations on the GUI display 50 while viewing the monitor video vM.

[0302] Figure 27 An example of processing by the control unit 25 during shooting is shown.

[0303] The control unit 25 advances the process to step S161 according to the start of shooting, and gives an instruction to start the video processing using the signal processing unit 30. Accordingly, the signal processing unit 30 starts referring to the video signal. Figure 12 Description of the process.

[0304] Furthermore, in step S162 , the control unit 25 generates metadata MT, and instructs the signal processing unit 30 to associate the metadata with the frames of the captured video vC.

[0305] In particular, in the case of this embodiment, the control unit 25 performs a process of associating metadata related to color setting parameters with the captured video vC or the raw video data vCraw.

[0306] Specific examples of the metadata MT include the following Figure 28 、 Figure 30 and Figure 32 Metadata describing MT1, MT2, MT3, MT4 and MT5.

[0307] Furthermore, metadata MT related to color setting parameters MT may not necessarily be associated with frames of the captured video vC. For example, it is conceivable that metadata MT may be associated with the captured video vC on a project basis, more specifically, on a project basis, where a frame rate, recording type, etc. are set. Furthermore, metadata MT may be associated on a clip basis of the captured video vC or on a file basis, which is referred to as a recording unit.

[0308] Furthermore, in the case of performing association with a file library or in the case of performing serial digital interface (SDI) output, the metadata MT may be considered to be included in a header or a footer of baseband original video data or compressed original video data.

[0309] In step S163, the control unit 25 causes the signal processing unit 30 to start recording and transmitting the captured video vC, the raw video data vCraw, and the metadata MT. Accordingly, the signal processing unit 30 starts recording the captured video vC and the metadata MT, or the raw video data vCraw and the metadata MT, on a recording medium in association with each other, and transmits and outputs the recorded data.

[0310] Then, when the shooting ends, the control unit 25 advances the process from step S164 to step S165, and gives a shooting end instruction to the signal processing unit 30. According to this, the signal processing unit 30 ends the process during the shooting.

[0311] According to the above-described processing, the captured video vC becomes a video in which color reproduction or color generation based on the color setting parameters corresponding to the lighting form is performed.

[0312] Furthermore, raw video data vCraw is recorded or transmitted, with which information related to color setting parameters is associated as metadata MT.

[0313] <5. Processing Examples Using Metadata and Raw Development Software>

[0314] Subsequently, an example of processing of executing raw video data vCraw in post-production ST3 will be described.

[0315] In post-production ST3, it is assumed that the information processing apparatus 70 serving as a video editing apparatus performs processing based on the original development software 90. Here, four video processing examples performed in the information processing apparatus 70 using the original development software 90 will be described.

[0316] In addition, here, although the processing based on the original development software 90 will be described, in the following, reference may be made to Figures 28 to 36 The described processing is understood as processing performed by the information processing device 70 based on the original color adjustment software.

[0317] Figure 28 The information processing device 70 is shown as being realized by the original development software 90 used in the post-production ST3 together with the camera 502. Here, as the processing functions of the information processing device 70, a WB unit 92 and a matrix unit 94 are shown. The WB unit 92 and the matrix unit 94 are for executing Figure 19 Processing and Figure 20 The processing function.

[0318] Figure 28 An example is shown in which metadata MT1 and MT2 are associated with raw video data vCraw from the camera 502 .

[0319] The metadata MT1 is the light source ID. For example, the metadata MT1 is Figure 25 The identification information of the lighting device identified in the lighting determination process of step S101 is shown. More specifically, the light source ID is the identification information of the lamp 580 and the LED wall 505. In addition, the light source ID can be information used to identify a specific model, such as the model of the LED lamp, the model of the LED panel 506 configuring the LED wall 505, etc.

[0320] The metadata MT2 represents color setting parameters. In other words, the metadata MT2 is the color temperatures KL1 and KLw and the tone values TT1 and TTw.

[0321] In this case, the information processing device 70 of the post-production ST3 can read the metadata MT1 and MT2 and perform color conversion processing when processing the raw video data vCraw. For example, the CPU 71 of the information processing device 70 executes the color conversion processing based on the raw development software 90. Figure 29 processing.

[0322] In step S201, the CPU 71 reads the color temperatures KL1 and KLw and the tone values TT1 and TTw as metadata MT2 associated with the raw video data vCraw, which is a processing target.

[0323] In step S202 , the CPU 71 sets the color temperatures KL1 and KLw and the tone values TT1 and TTw as color setting parameters of the WB unit 92 and the matrix unit 94 .

[0324] In step S203, the CPU 71 sets the mixing ratio MixR and instructs the WB unit 92 and the matrix unit 94. For example, based on the light source ID as metadata MT1, the CPU 71 estimates the lighting equipment used and the lighting light intensity ratio in consideration of using lighting together, and sets the mixing ratio MixR.

[0325] Accordingly, in the information processing device 70, color conversion processing using the color conversion parameters at the time of shooting is performed by the WB unit 92 and the matrix unit 94, and the adjusted video vE as the processing result of the color conversion processing is displayed on the monitor device 99 used in post-production ST3.

[0326] Furthermore, although the case where the mixing ratio MixR is estimated based on the information of the light source ID for color reproduction has been described, for example, it is possible to prepare Figure 21 The GUI shown in FIG. 14 is used, and the mixing ratio MixR can be set according to the user's operation.

[0327] According to this, video processing can be performed in which the operator can arbitrarily set the mixing ratio MixR while using the color temperature KL and the tone value TT at the time of shooting.

[0328] Furthermore, while the color temperatures KL1 and KLw and the tone values TT1 and TTw at the time of photographing are set as initial settings, the operator can arbitrarily change the values of the initial settings.

[0329] Figure 30 This is an example in which metadata MT1 , MT2 , and MT3 are associated with raw video data vCraw from the camera 502 .

[0330] Metadata MT3 is the mixing ratio MixR set during shooting.

[0331] In this case, the information processing device 70 of the post-production ST3 can read the metadata MT1, MT2 and MT3 and perform color conversion processing when processing the raw video data vCraw. For example, the CPU 71 of the information processing device 70 executes the color conversion processing based on the raw development software 90. Figure 31 processing.

[0332] In step S211, the CPU 71 reads the color temperatures KL1 and KLw and the tone values TT1 and TTw as metadata MT2 associated with the original video data vCraw (which is a processing target), and reads the mixing ratio MixR as metadata MT3.

[0333] In step S212 , the CPU 71 sets the color temperatures KL1 and KLw and the tone values TT1 and TTw as parameters of the WB unit 92 and the matrix unit 94 .

[0334] In step S213 , the CPU 71 instructs the WB unit 92 and the matrix unit 94 of the mixing ratio MixR as metadata MT3 .

[0335] According to this, in the information processing device 70 , color conversion processing similar to that at the time of shooting is performed by the WB unit 92 and the matrix unit 94 , and the video vE after adjustment as a processing result of the color conversion processing is displayed in the monitor device 99 .

[0336] In this example, metadata MT1 as the light source ID is not necessarily required. However, by including metadata MT, it is possible to perform adjustment of the color temperature and tint value or the mixing ratio according to the light source type of the lighting device used.

[0337] Furthermore, in this case, for example, Figure 21 The GUI shown, and the mixing ratio MixR, color temperatures KL1 and KLw, and tone values TT1 and TTw can be set according to the user's operation.

[0338] According to this, it is possible to perform video processing in which the operator arbitrarily sets parameters while setting parameter settings similar to those at the time of shooting as initial values.

[0339] Figure 32 This is an example in which metadata MT4 and MT5 are associated with the raw video data vCraw from the camera 502 .

[0340] The metadata MT4 is the value of the white balance coefficient (R gain, G gain, and B gain) of the WB unit 32 at the time of shooting. In the description, the metadata MT4 will be expressed as a white balance coefficient Kw.

[0341] The metadata MT5 is 9 matrix coefficients for 3×3 matrix color conversion used by the matrix unit 34 at the time of shooting. In the description, the metadata MT5 will be expressed as matrix coefficients Km.

[0342] In this case, the information processing device 70 of the post-production ST3 can read the metadata MT4 and MT5 and perform color conversion processing when processing the raw video data vCraw. For example, the CPU 71 of the information processing device 70 executes the color conversion processing based on the raw development software 90. Figure 33 processing.

[0343] In step S221 , the CPU 71 reads the white balance coefficient Kw and the matrix coefficient Km as metadata MT4 and MT5 associated with the raw video data vCraw (which is the processing target).

[0344] In step S222 , the CPU 71 indicates the white balance coefficient Kw and matrix coefficient Km that have been read to the WB unit 92 and the matrix unit 94 .

[0345] The white balance coefficient Kw and the matrix coefficient Km are coefficient values reflecting the mixing ratio MixR at the time of shooting.

[0346] Therefore, even when no instructions are given regarding the color temperatures KLl and KLw, the tone values TTl and TTw, and the mixing ratio MixR, a color conversion process similar to that at the time of shooting is performed by the WB unit 92 and the matrix unit 94, and the adjusted video vE as a result of the processing of the color conversion process is displayed in the monitor device 99.

[0347] However, in this case, it is also possible to prepare, for example, Figure 21 The GUI shown, and the mixing ratio MixR, color temperatures KL1 and KLw, and tone values TT1 and TTw can be set according to the user's operation.

[0348] According to this, it is possible to perform video processing in which the operator can arbitrarily set parameters from a state in which color conversion similar to that at the time of photographing is performed, and perform color generation.

[0349] Figure 34 This is an example in which metadata related to color conversion processing is not associated with the raw video data vCraw from the camera 502 .

[0350] In this case, the information processing device 70 that performs post-production ST3 prepares, for example, Figure 21 GUI shown, and for example when processing raw video data vCraw Figure 35 processing.

[0351] In step S231, the CPU 71 monitors the user's operation related to the color conversion process. If there is an operation, the CPU 71 accepts the operation information in step S232.

[0352] In step S233, the CPU 71 sets the color temperatures KL1 and KLw and the tone values TT1 and TTw according to the operation, and instructs the WB unit 92 and the matrix unit 94.

[0353] Furthermore, in step S234 , the CPU 71 sets the mixing ratio MixR according to the operation, and instructs the WB unit 92 and the matrix unit 94 .

[0354] According to the description presented above, it is possible to perform video processing in which an operator performs color generation on raw video data vCraw regardless of color conversion at the time of shooting.

[0355] exist Figure 29 、 Figure 31 and Figure 33 In the processing of , an example of setting color setting parameters (color temperature and tint value) or setting processing coefficients (white balance coefficient Kw and matrix coefficient Km) according to metadata MT is shown. Then, in this case, after such setting, further, it has been described that the operator can use Figure 21 The GUI shown performs operations to correct color.

[0356] Therefore, the information processing device 70 may be configured to execute the Figure 36 The processing and Figure 29 、 Figure 31 or Figure 33 processing.

[0357] Will describe Figure 36 For example, in Figure 29 、 Figure 31 or Figure 33 After setting the color setting parameters or processing coefficients in the process of , the CPU 71 of the information processing device 70 checks in step S251 whether the color setting of the original video data vCraw has been completed. In other words, it checks whether the operator has performed an operation to complete the color processing.

[0358] The CPU 71 monitors the operator's correction operation in step S252 until the completion of the operation is detected. The operator can use Figure 21 The illustrated GUI arbitrarily operates the color temperature and tint value of the LED wall 505, the color temperature and tint value of the lamp 580, and the mixing ratio MixR.

[0359] The CPU 71 returns the process to step S251 when there is no such operation, and advances the process to step S253 and accepts operation information if there is any one of the operations.

[0360] Upon accepting the operation information, the CPU 71 sets the color temperatures KL1 and KLw and the tone values TT1 and TTw according to the operation in step S254 and instructs the WB unit 92 and the matrix unit 94 .

[0361] Furthermore, in step S255, the CPU 71 sets the mixing ratio MixR according to the operation, and instructs the WB unit 92 and the matrix unit 94. Then, the process returns to step S251.

[0362] According to the above-described process, color correction by the operator is arbitrarily performed until color setting is completed.

[0363] When the operator performs an operation to complete the color setting, such as an operation to instruct the output (recording, transmission, uploading, etc.) of the raw video data vCraw at a certain point in time, the CPU 71 advances the process from step S251 to step S256 and generates metadata MT. In this case, the metadata MT includes all or some of the color temperatures KL1 and KLw, the tonal values TT1 and TTw, the mixing ratio MixR, the white balance coefficient Kw, the matrix coefficient Km, and the like at that point in time.

[0364] Then, in step S257 , the CPU 71 outputs the raw video data vCraw and the metadata MT in association with each other.

[0365] For example, the CPU 71 controls recording on a recording medium, transmission to another device, uploading, etc., in a state where the raw video data vCraw and the metadata MT are associated with each other. In this case, in addition to the raw video data vCraw and the metadata MT, all video data for which color setting has been processed by the WB unit 92 and the matrix unit 94 or all video data for which development has been processed can be output.

[0366] According to the above processing, metadata MT including the color setting parameters and processing coefficients after correction is set as output content together with the original video data vCraw. Therefore, in the subsequent video processing stage, the processing performed by the information processing device 70 can be reproduced.

[0367] Although Figure 36 The processing has been described in Figure 29 、 Figure 31 or Figure 33 But for example, Figure 34In the case of the example described in the preceding example, the above processing can be performed instead of Figure 35 processing.

[0368] In addition, although Figure 36 The processing has been described as being performed by the information processing apparatus 70 based on the raw development software 90 (or raw color adjustment software), but it may be processing performed by the control unit 25 of the camera 502. In other words, this is an example in which the camera 502 outputs metadata MT (including raw video data vCraw) including color setting parameters and processing coefficients after the operator's color correction operation.

[0369] <6. Summary and Modification Examples>

[0370] According to the above-described embodiment, the following effects can be obtained.

[0371] The imaging device 502, which is a shooting device according to the embodiment, includes an image sensor 21 and a signal processing unit 30. The signal processing unit 30 can perform a first color conversion process on the video data acquired by the image sensor 21, the first color conversion process being performed using first color setting parameters (color temperature KL1 and tint value TT1) set for a first lighting (lamp 580). Furthermore, the signal processing unit 30 can perform a second color conversion process using second color setting parameters (color temperature KLw and tint value TTw) set for a second lighting (LED wall 505) having different spectral characteristics from the first lighting and using a display device that displays a background video. The signal processing unit 30 is configured to selectively perform these first and second color conversion processes.

[0372] In a shooting environment where LED wall 505 (secondary lighting) is used, there are cases where lamp 580 (primary lighting such as an LED lamp) is used, and cases where LED wall 505 is used as lighting. The light from LED wall 505 does not have spectral characteristics optimized for lighting use. Even in this case, by performing the second color conversion processing using color setting parameters set for the spectral characteristics of LED wall 505, the color reproduction of the captured video vC can be improved.

[0373] In addition, in this embodiment, although the background video vB of the video displayed on the LED wall 505 is a virtual video obtained by rendering using a 3D model, the technology of the present disclosure can also be applied to the case of displaying a shot video on the LED wall 505, for example.

[0374] In the embodiment, an example is described in which the signal processing unit 30 is configured to be capable of executing the third color conversion process performed using both the first color setting parameters and the second color setting parameters.

[0375] For example, in the case of using both the LED wall 505 and the lamp 580, by performing color conversion processing in which the parameters KLw and TTw for the LED wall 505 and the parameters KLl and TTl for the lamp 580 are reflected, color reproduction of the captured video vC can be improved.

[0376] In addition, the WB unit 32 and the matrix unit 34 have Figure 19 and Figure 20 With the illustrated configuration, also in the case where the mixing ratio MixR of the color temperature KL and the tone value TT is 0:100 or 100:0, the processing of the WB unit 32 and the matrix unit 34 corresponds to the first color conversion processing or the second color conversion processing.

[0377] In a case where the mixing ratio MixR is neither 0:100 nor 100:0, in other words, the color conversion process in which all the color temperatures KLw and KL1 and the tone values TTw and TT1 are reflected becomes the third color conversion process described in the present disclosure.

[0378] In the embodiment, an example is described in which the signal processing unit 30 sets the degree of reflection of the first color setting parameter and the second color setting parameter on the third color conversion process according to the mixing ratio MixR.

[0379] When both the LED wall 505 and the lamp 580 are used, their contributions to the illumination light vary. Therefore, the ratio of the reflections of the color setting parameters KLw and TTw for the LED wall 505 and the color setting parameters KLl and TTl for the lamp 580 is set according to the mixing ratio MixR. This allows for color conversion processing based on the ratio of the intensities of the illumination lights from the LED wall 505 and the lamp 580.

[0380] In the embodiment, an example has been described in which the signal processing unit 30 of the camera 502 performs processing to associate the first color setting parameter and the second color setting parameter with the video data as metadata MT. Furthermore, an example has been described in which the information processing device 70 used in post-production ST3 reads the first color setting parameter and the second color setting parameter from the metadata associated with the video data and applies them to the color conversion processing.

[0381] For example, as in Figure 28 and Figure 30 In the example shown, color setting parameters KLw and TTw for the LED wall 505 and color setting parameters KL1 and TT1 for the lamp 580 are associated with the raw video data vCraw as metadata MT2.

[0382] According to this, when processing of the raw video data vCraw is performed by the information processing apparatus 70 installed with the raw development software 90, color reproduction processing using all or some of the color setting parameters KLw, TTw, KL1, and TT1 at the time of shooting can be performed.

[0383] Furthermore, only the color setting parameters KL1 and TT1 for the lamp 580, which are the first color setting parameters, can be associated with the raw video data vCraw as metadata MT2. Conversely, only the color setting parameters KLw and TTw for the LED wall 505, which are the second color setting parameters, can be associated with the raw video data vCraw as metadata MT2. In other words, a process is performed to associate at least one of the first color setting parameters and the second color setting parameters with the video data.

[0384] Furthermore, the process of associating with the captured video data is a process in which, for example, color setting parameters KLw, TTw, KLl, TTl, etc. are set as metadata MT, set as data included in a video file along with the original video data vCraw, and recorded on a recording medium or transmitted to an external device. Alternatively, even when such metadata MT is data of a file different from the file of the original video data vCraw, the data can be associated with each other by performing a process of maintaining the data in a managed state by associating the data with each other through the allocation of the same ID, etc.

[0385] In the embodiment, an example has been described in which the signal processing unit 30 of the camera 502 performs processing to associate the mixing ratio MixR of the first color setting parameters and the second color setting parameters in the third color reproduction processing with video data. Furthermore, an example has been described in which the information processing device 70 used in post-production ST3 reads the mixing ratio MixR from metadata associated with the video data and applies the mixing ratio to the color conversion processing.

[0386] For example, as in Figure 30 In the example shown, the mixing ratio MixR is associated with the original video data vCraw as metadata MT3.

[0387] According to this, when the raw video data vCraw is subsequently processed, color reproduction processing using the mixing ratio MixR at the time of shooting can be performed. In particular, by associating the mixing ratio MixR with the color setting parameters KLw, TTw, KLl, and TTl, when the raw video data vCraw is processed by the information processing device 70 in which the raw development software 90 is running, color reproduction processing using a mixing ratio similar to the mixing ratio at the time of shooting can be performed.

[0388] In the embodiment, an example has been described in which the signal processing unit 30 of the camera 502 performs processing for associating processing coefficients for color conversion processing based on first and second color setting parameters with video data. Furthermore, an example has been described in which the information processing device 70 used in post-production ST3 reads processing coefficients (white balance coefficient Kw and matrix coefficient Km) for color conversion processing based on first and second color setting parameters from metadata associated with video data and applies them to the color conversion processing.

[0389] For example, as in Figure 32 In the example of , the white balance coefficient Kw and the matrix coefficient Km set based on the color setting parameters KLw, TTw, KLl and TTl at the time of shooting are associated with the original video data vCraw as metadata MT4 and MT5, and can be used in post-production ST3.

[0390] According to this, when processing of the raw video data vCraw is performed by the information processing apparatus 70 running the raw development software 90, color reproduction processing similar to that at the time of shooting can be performed.

[0391] In the embodiment, the signal processing unit 30 of the camera 502 and the information processing device 70 used in post-production ST3 perform white balance adjustment processing as one of the color conversion processing.

[0392] According to white balance adjustment, natural chromaticity can be obtained by absorbing the difference between the color temperatures of the light sources of illumination.

[0393] In the embodiment, the signal processing unit 30 of the camera 502 and the information processing device 70 used in post-production ST3 perform a matrix color conversion process as one of the color conversion processes.

[0394] According to the matrix color conversion process, natural chromaticity can be acquired by absorbing the difference between the spectral characteristics of the light sources of illumination.

[0395] Furthermore, processing equivalent to the matrix color conversion processing may be performed as 3D LUT conversion processing using a three-dimensional lookup table (3D LUT).

[0396] In the embodiment, an example is described in which the UI unit 22 is included that can give instructions for the first color setting parameters and the second color setting parameters, and the signal processing unit 30 is configured to be able to perform the first color conversion process or the second color conversion process using the first color setting parameters and the second color setting parameters indicated by the UI unit 22.

[0397] The user can specify the first color setting parameters (color temperature KL1 and tint value TT1) or the second color setting parameters (color temperature KLw and tint value TTw) using the UI unit 22. This allows the user to perform color conversion processing for color reproduction according to lighting. Furthermore, the user can instruct not only simple color reproduction but also color conversion processing for so-called desired color generation.

[0398] exist Figure 21 In the example of , although the UI is configured so that the mixing ratio MixR can also be specified, a UI configured so that the mixing ratio MixR cannot be specified is also conceivable.

[0399] In an embodiment, an example is described in which a UI unit 22 is included that is configured to be able to indicate first color setting parameters, second color setting parameters, and ratio information, and the signal processing unit 30 is configured to be able to perform a third color conversion process using the first color setting parameters, second color setting parameters, and ratio information indicated by the UI unit 22.

[0400] The user can specify the mixing ratio MixR and the color setting parameters KL1, TT1, KLw, and TTw using the UI unit 22. Accordingly, the user can perform color conversion processing for color reproduction and color generation even when both the lamp 580 and the LED wall 505 are used as lighting.

[0401] Furthermore, a UI can be considered in which the color setting parameters KL1, TT1, KLw, and TTw are preset as fixed values, for example, and only the mixing ratio MixR can be arbitrarily specified by the user.

[0402] In an embodiment, the first color setting parameter and the second color setting parameter include a color temperature and a tint value.

[0403] For example, by setting processing coefficients as white balance adjustment and matrix color conversion processing based on color temperature and tint value, appropriate color reproduction according to lighting is achieved.

[0404] In an embodiment, the camera 502 outputs raw video data vCraw. In addition, for example, by performing Figure 36 In the processing shown in the image capture device 502, the control unit 25 and the signal processing unit 30 correct at least one of the first color setting parameter and the second color setting parameter, and the corrected first color setting parameter or the corrected second color setting parameter or the processing coefficients of the WB unit 32 and the matrix unit 34 can be associated with the raw video data vCraw.

[0405] According to this, the color setting parameters and processing coefficients reflecting the operator's correction can be set as output content together with the original video data vCraw. Therefore, in the subsequent processing, the color setting corrected by the camera 502 can be reproduced.

[0406] In the embodiment, an example is described in which the control unit 25 of the camera 502 automatically performs one of the first color conversion process and the second color conversion process or suggests one of the first color conversion process and the second color conversion process to the user based on the shape of the display device or the correspondence / non-correspondence of the LED lights. In other words, Figure 25 In the process of , parameter setting is performed according to the lighting judgment of step S101. Alternatively, as described in a modified example thereof, a recommendation of color setting parameters is given to the user. Accordingly, in the case where color processing with good reproducibility is required, the burden on the user can be reduced.

[0407] The program according to the embodiment is, for example, a program that causes a processor such as a CPU, a DSP, or the like, or a device including these to execute the above-described Figure 29 、 Figure 31 、 Figure 33 、 Figure 35 and Figure 36 In other words, the program according to the embodiment is a program that causes the information processing device 70 to perform, on video data obtained by the camera 502 (for example, raw video data vCraw), a first color conversion process performed using first color setting parameters for a first lighting setting, and a second color conversion process performed using second color setting parameters for a second lighting setting having spectral characteristics different from those of the first lighting and using a display device that displays the background video vB.

[0408] According to such a program, for example, by using the information processing device 70 used as a video editing device in post-production ST3, it is possible to perform color conversion processing for color reproduction related to the captured video vC. In particular, the color reproduction of the captured video vC can be improved in accordance with the case where the lamp 580 is used as lighting and the case where the LED wall 505 is used as lighting.

[0409] Furthermore, the program according to the embodiment is also a program that causes the information processing apparatus 70 to execute the third color conversion process that is executed using two pairs of the first color setting parameters and the second color setting parameters.

[0410] For example, corresponding to the case of using both the LED wall 505 and the lamp 580, when post-producing ST3, etc., the information processing device 70 is used to perform color conversion processing that reflects the parameters KLw and TTw for the LED wall 505 and the parameters KLl and TTl for the lamp 580, thereby improving the color reproduction of the captured video vC.

[0411] The program according to the embodiment is software that causes the information processing apparatus 70 to execute development processing on raw video data vCraw, which is video data captured by the imaging device 502. In other words, the program is so-called raw development software 90. This realizes a program suitable for processing the raw video data vCraw in post-production ST3.

[0412] In addition, the captured video vC processed in the post-production ST3 is not limited to the raw video data vCraw described above. For example, similar processing can be performed on the captured video vC after the development processing.

[0413] Furthermore, for example, video data after processing by the WB unit 32 of the camera 502 may be output as raw video data vCraw, and it may be considered that processing similar to this is performed on such raw video data vCraw.

[0414] In the program according to the embodiment, an example is described in which the information processing device 70 performs a process of correcting at least one of the first color setting parameter and the second color setting parameter and associating the corrected first color setting parameter or the corrected second color setting parameter with the raw video data vCraw (see Figure 36 ).

[0415] Thus, the color setting parameters and processing coefficients reflecting the operator's correction can be configured as output content together with the original video data vCraw. Thus, the color setting corrected by the information processing device 70 based on the original development software 90 can be reproduced in subsequent processing.

[0416] An example is described in which a program according to an embodiment causes the information processing apparatus 70 to apply first color setting parameters and second color setting parameters as metadata associated with video data according to the camera 502 to color conversion processing. Figure 28 and Figure 29 Describe the post-production processing of ST3.

[0417] Furthermore, one of the first color setting parameter and the second color setting parameter may be applied to the color conversion process.

[0418] Furthermore, an example is described in which the program according to the embodiment causes the information processing apparatus 70 to apply ratio information of the first color setting parameter and the second color setting parameter as metadata associated with the video data according to the camera 502 to the color conversion process. Figure 30 and Figure 31 Describe the post-production processing of ST3.

[0419] Furthermore, an example is described in which the program according to the embodiment causes the information processing apparatus 70 to apply processing coefficients (white balance coefficient Kw and matrix coefficient Km) of a color conversion process corresponding to first color setting parameters and second color setting parameters as metadata associated with video data according to the camera 502. Accordingly, reference is performed. Figure 32 and Figure 33 Describe the post-production processing of ST3.

[0420] Furthermore, one of a processing coefficient according to the first color setting parameters and a processing coefficient according to the second color setting parameters may be applied to the color conversion process.

[0421] Furthermore, an example is described in which the program according to the embodiment causes the information processing apparatus 70 to execute the color conversion process according to the first color setting parameter and the second color setting parameter specified in accordance with the operation of the user.

[0422] For example, for raw video data vCraw, regardless of the presence / absence of metadata such as parameters related to color conversion processing, such as Figure 21 The UI shown is also provided in the information processing device 70, and therefore, as shown in FIG. Figure 34 and Figure 35 As described, color conversion processing for reproduction or color generation can be performed according to a user's operation.

[0423] The program according to the above-described embodiment may be recorded in advance in an HDD or a ROM inside a microcomputer having a CPU or the like, which is a recording medium built into a device such as a computer device. In addition, such a program may be temporarily or permanently stored (recorded) in a removable recording medium such as a floppy disk, a compact disk read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disk (DVD), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such a removable recording medium may be provided as so-called packaged software.

[0424] Furthermore, such a program can be installed from a removable recording medium to a personal computer or the like, or can be downloaded from a download site via a network such as a local area network (LAN), the Internet, or the like.

[0425] In addition, such a program is suitable for widely providing the information processing device 70 according to the embodiment. For example, by downloading the program to a personal computer, a communication device, a portable terminal device such as a smartphone, a tablet computer, a mobile phone, a game device, a video device, a personal digital assistant (PDA), etc., such a device can be configured to function as the information processing device 70 according to the present disclosure.

[0426] Furthermore, the effects described in this specification are merely examples, the effects are not limited thereto, and other effects may exist.

[0427] Furthermore, the present technology can adopt the following configurations while remaining within the technical scope of the present disclosure.

[0428] (1) An imaging device comprising:

[0429] A circuit configured to:

[0430] Get the input image from the image sensor,

[0431] Information related to color conversion of the acquired input image is generated based on:

[0432] first color setting parameters determined based on at least one first light source configured to provide illumination light for the acquired input image, and

[0433] second color setting parameters determined based on at least one second light source configured to provide illumination light for the acquired input image, and

[0434] Based on the generated information related to the color conversion of the acquired input image, output of the output data is initiated.

[0435] (2) An imaging device according to (1), wherein the at least one first light source includes one or more first lighting devices that display one or more background images in the acquired input image, and the at least one second light source includes at least one lighting device of a different light source type other than the one or more first lighting devices.

[0436] (3) The imaging apparatus according to (1) or (2), wherein the one or more first lighting devices include an arrangement of one or more light emitting diode (LED) panels that display the one or more background images.

[0437] (4) The imaging device according to any one of (1) to (3), wherein the content of the one or more background images is determined based on information indicating the area of the one or more LED panels included in the acquired input image.

[0438] (5) The imaging device according to any one of (1) to (4), wherein

[0439] The first color setting parameter is determined according to a spectral characteristic of the at least one first light source, and the spectral characteristic of the at least one first light source is different from a spectral characteristic of the at least one second light source.

[0440] (6) The imaging apparatus according to any one of (1) to (5), wherein the at least one first light source includes one or more first lighting devices that display one or more background images in the acquired input image.

[0441] (7) The imaging device according to any one of (1) to (6), wherein the first color setting parameter and the second color setting parameter are reflected in the color conversion according to a mixing ratio set between the first color setting parameter and the second color setting parameter.

[0442] (8) The imaging device according to any one of (1) to (7), wherein the blending ratio is set according to a user's operation.

[0443] (9) The imaging device according to any one of (1) to (8), wherein the mixing ratio is set according to a ratio of the intensity of the illumination light provided by the at least one first light source to the intensity of the illumination light provided by the at least one second light source.

[0444] (10) The imaging device according to any one of (1) to (9), wherein the first color setting parameter and the second color setting parameter include at least one of a color temperature or a hue value of the acquired input image.

[0445] (11) The imaging device according to any one of (1) to (10), wherein the generated information includes metadata corresponding to the first color setting parameter and the second color setting parameter.

[0446] (12) The imaging device according to any one of (1) to (11), wherein the output data includes the metadata in an output image based on the acquired input image.

[0447] (13) An imaging device according to any one of (1) to (12), wherein the output data includes at least one metadata file corresponding to the metadata, and the at least one metadata file is separated from and associated with at least one output image file based on the acquired input image.

[0448] (14) The imaging device according to any one of (1) to (13), wherein the output data includes raw image data.

[0449] (15) The imaging device according to any one of (1) to (14), wherein the circuit is further configured to perform color conversion based on the first color setting parameter and the second color setting parameter to obtain a color-converted image included in the output data.

[0450] (16) The imaging device according to any one of (1) to (15), wherein the output data includes the color-converted image and original image data.

[0451] (17) An image processing method comprising:

[0452] Get the input image;

[0453] generating information related to color conversion of the acquired input image based on first color setting parameters determined according to at least one first light source configured to provide illumination light for the acquired input image and second color setting parameters determined according to at least one second light source configured to provide illumination light for the acquired input image; and

[0454] Output data is output according to the generated information related to color conversion of the acquired input image.

[0455] (18) The image processing method according to (17),

[0456] Wherein, the acquiring includes acquiring the input image from another information processing device, the first color setting parameter and the second color setting parameter are each acquired in association with the acquired input image, and the method further includes performing color conversion based on the acquired first color setting parameter and the acquired second color setting parameter to obtain a color-converted image included in the output data.

[0457] (19) The information processing method according to (17) or (18), further comprising:

[0458] correcting at least one of the first color setting parameter and the second color setting parameter; and

[0459] The corrected first color setting parameters or the corrected second color setting parameters are associated with the original image data.

[0460] (20) A non-transitory computer-readable medium having a program embodied thereon, wherein when the program is executed by a computer, the computer is caused to perform an information processing method, the method comprising:

[0461] Get the input image;

[0462] generating information related to color conversion of the acquired input image based on first color setting parameters determined according to at least one first light source configured to provide illumination light for the acquired input image and second color setting parameters determined according to at least one second light source configured to provide illumination light for the acquired input image; and

[0463] Output data is output according to the generated information related to color conversion of the acquired input image.

[0464] Reference Signs List

[0465] 21 Image Sensor

[0466] 22UI Unit

[0467] 23 communication units

[0468] 24 recording control unit

[0469] 25 control units

[0470] 26 memory cells

[0471] 27 display units

[0472] 30 signal processing units

[0473] 31 sensor signal processing unit

[0474] 32 white balance processing units (WB processing units)

[0475] 33 color separation units

[0476] 34 matrix processing units

[0477] 35 Development processing unit

[0478] 41, 41A and 41b coefficient setting units

[0479] 42, 43, 44 multipliers

[0480] 45 coefficient setting unit

[0481] 46 matrix coefficient processing units

[0482] 47, 48 Mixing coefficient setting unit

[0483] 50GUI display

[0484] 51 color temperature operator

[0485] 52 color temperature operator

[0486] 53 Tone Operators

[0487] 54 color tone operators

[0488] 55 Mixing ratio operator

[0489] 70 Information processing equipment

[0490] 71CPU

[0491] 90 original development software

[0492] 92 white balance processing units (WB processing units)

[0493] 94 matrix processing units

[0494] 502 camera device

[0495] 505LED Wall

[0496] 580 lights

Claims

1. An imaging device comprising: A circuit configured to: Get the input image from the image sensor, Information related to color conversion of the acquired input image is generated based on: first color setting parameters determined based on at least one first light source configured to provide illumination light for the acquired input image, and second color setting parameters determined based on at least one second light source configured to provide illumination light for the acquired input image, and Based on the generated information related to the color conversion of the acquired input image, output of the output data is enabled.

2. The imaging device according to claim 1, in, The at least one first light source includes one or more first lighting devices that display one or more background images in the acquired input image, and The at least one second light source includes at least one lighting device of a different light source type than the one or more first lighting devices.

3. The imaging device according to claim 2, in, The one or more first lighting devices include an arrangement of one or more light emitting diode (LED) panels that display the one or more background images.

4. The imaging device according to claim 3, in, Contents of the one or more background images are determined based on information indicating areas of the one or more LED panels included in the acquired input image.

5. The imaging device according to claim 1, in, determining the first color setting parameter according to spectral characteristics of the at least one first light source, and The spectral characteristics of the at least one first light source are different from the spectral characteristics of the at least one second light source.

6. The imaging device according to claim 5, in, The at least one first light source includes one or more first lighting devices that display one or more background images in the acquired input image.

7. The imaging device according to claim 5, in, The first color setting parameter and the second color setting parameter are reflected in the color conversion according to a mixing ratio set between the first color setting parameter and the second color setting parameter.

8. The imaging device according to claim 7, in, The mixing ratio is set according to a user's operation.

9. The imaging device according to claim 7, in, The mixing ratio is set according to a ratio of an intensity of the illumination light provided by the at least one first light source to an intensity of the illumination light provided by the at least one second light source.

10. The imaging device according to claim 1, in, The first color setting parameter and the second color setting parameter include at least one of a color temperature or a hue value of the acquired input image.

11. The imaging device according to claim 1, in, The generated information includes metadata corresponding to the first color setting parameter and the second color setting parameter.

12. The imaging device according to claim 11, in, The output data includes the metadata in an output image based on the acquired input image.

13. The imaging device according to claim 11, in, The output data includes at least one metadata file corresponding to the metadata, the at least one metadata file being separate from and associated with at least one output image file based on the acquired input image.

14. The imaging device according to claim 1, in, The output data includes raw image data.

15. The imaging device according to claim 1, in, The circuit is further configured to perform color conversion based on the first color setting parameter and the second color setting parameter to obtain a color converted image included in the output data.

16. The imaging device according to claim 15, in, The output data includes the color-converted image and original image data.

17. An image processing method, comprising: Get the input image; Information related to color conversion of the acquired input image is generated based on: first color setting parameters determined based on at least one first light source configured to provide illumination light for the acquired input image, and second color setting parameters determined based on at least one second light source configured to provide illumination light for the acquired input image; as well as Output data is output according to the generated information related to color conversion of the acquired input image.

18. The image processing method according to claim 17, in, The acquiring includes acquiring the input image from another information processing device, wherein the first color setting parameter and the second color setting parameter are each acquired in association with the acquired input image, and Wherein, the method further includes performing color conversion based on the acquired first color setting parameter and the acquired second color setting parameter to obtain a color-converted image included in the output data.

19. The image processing method according to claim 17, further comprising: correcting at least one of the first color setting parameter and the second color setting parameter; as well as The corrected first color setting parameters or the corrected second color setting parameters are associated with the original image data.

20. A non-transitory computer-readable medium having a program embodied thereon, wherein when the program is executed by a computer, the program causes the computer to function as an information processing method, the method comprising: Get the input image; Information related to color conversion of the acquired input image is generated based on: first color setting parameters determined based on at least one first light source configured to provide illumination light for the acquired input image, and second color setting parameters determined based on at least one second light source configured to provide illumination light for the acquired input image; as well as Output data is output according to the generated information related to color conversion of the acquired input image.

Images