Information processing device, information processing method, and program

Abstract

To check the hue etc. of a taken image at a time before taking a virtual image of a display with a camera.SOLUTION: An information processing device comprises a determination unit which determines whether an artifact occurs in an image taken with a camera by using an interval between light emitters of a display in an image taking system for taking an image of the display with the camera, a pixel interval in the image sensor of the camera, the properties of the optical low-pass filter of the camera, the focal distance, the F value, and the focusing distance of a lens of the camera, and the distance between the display and the camera.SELECTED DRAWING: Figure 35

Description

[Technical field]

[0001] The present technology relates to an information processing device, an information processing method, and a program, and relates to technology that can be used, for example, when producing a video using a virtual image. [Background technology]

[0002] 2. Description of the Related Art As a shooting technique for producing video content such as movies, a technique is known in which actors perform in front of a so-called green screen, and a background image is later composited into the performance. In recent years, instead of shooting against a green screen, a shooting system has been developed in which a large display device is installed in a studio, a background image is displayed on the display device, and the performers perform in front of the display device, allowing the performers and the background to be filmed. This is known as Virtual Production, In-Camera VFX, or LED Wall Virtual Production. The following Patent Document 1 discloses a system technique for filming a performer performing in front of a background image. Furthermore, Patent Document 2 listed below discloses a technique for arranging a film-like optical member to prevent moire when photographing a large display device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2020 / 0145644 [Patent Document 2] JP 2014-202816 A Summary of the Invention [Problem to be solved by the invention]

[0004] Displaying the background image on a large display and then shooting the performers and background with a camera has many advantages over shooting on a green screen, such as not having to separately composite the background image after shooting and allowing performers and staff to visually understand the scene and judge the performance and whether it was good or bad.

[0005] However, when capturing images of a display, artifacts such as moire occur due to interference between the pixels of the camera's image sensor and the pixels of the display panel. To avoid this, the camera needs to blur the display instead of focusing on it. However, because the conditions under which moire occurs are unknown, it is necessary to change the positions of the camera and performers, adjust the lenses, and carry out trial and error on the set, which reduces the efficiency of video production.

[0006] Therefore, the present disclosure proposes a technique that enables occurrence of artifacts such as moire to be determined more accurately. [Means for solving the problem]

[0007] The information processing device according to the present technology is configured to calculate a pixel value based on an image captured by a camera using an interval between light-emitting elements of a display in an imaging system that captures an image of the display with a camera, an interval between pixels of an image sensor of the camera, characteristics of an optical low-pass filter of the camera, a focal length, an F-number, and a focusing distance of a lens of the camera, and a distance between the display and the camera. Moire The device further includes a determination unit that performs a process of determining whether or not the above occurs. That is, the information processing device uses the above-mentioned values ​​of the intervals between the light-emitting elements, etc., and when an image of the display is captured by a camera before or during shooting, etc., the captured image is added with more. Re Determine whether or not this occurs. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram of an imaging system according to an embodiment of the present technology. [Diagram 2]11A and 11B are explanatory diagrams of background images according to camera positions of the imaging system according to the embodiment. [Diagram 3] 11A and 11B are explanatory diagrams of background images according to camera positions of the imaging system according to the embodiment. [Figure 4] FIG. 2 is an explanatory diagram of a video content production process according to an embodiment. [Diagram 5] FIG. 1 is a block diagram of a photographing system according to an embodiment. [Figure 6] 11 is a flowchart of background image generation in the imaging system according to the embodiment. [Figure 7] FIG. 1 is a block diagram of an imaging system using multiple cameras according to an embodiment. [Figure 8] FIG. 1 is a block diagram of an information processing apparatus according to an embodiment. [Figure 9] FIG. 2 is an explanatory diagram of pre-visualization according to an embodiment. [Figure 10] FIG. 2 is a block diagram of a configuration example including a determination unit according to an embodiment. [Figure 11] FIG. 11 is an explanatory diagram of a camera position operation image according to an embodiment. [Figure 12] 1 is an explanatory diagram of a moire alert according to an embodiment. [Figure 13] 11A to 11C are explanatory diagrams of operations in a camera position operation video according to an embodiment. [Figure 14] 11 is a flowchart of a moire alert process according to an embodiment. [Figure 15] 11 is a flowchart of a moire alert process during shooting according to an embodiment. [Figure 16] FIG. 2 is an explanatory diagram of an image on an image sensor. [Figure 17] FIG. 2 is an explanatory diagram of an optical LPF (Low Pass Filter). [Figure 18] FIG. 2 is an explanatory diagram of the characteristics of an optical LPF. [Figure 19] FIG. 2 is an explanatory diagram of a pixel width. [Figure 20] FIG. 2 is an explanatory diagram of the blurring of an image on an image sensor. [Figure 21] FIG. 1 is an explanatory diagram of the approximation of a regular dodecagon. [Figure 22] FIG. 1 is an explanatory diagram of an image of an LED (Light Emitting Diode). [Diagram 23] This is an explanatory diagram of how an LED image is split into four by an optical LPF. [Figure 24] FIG. 13 is an explanatory diagram of the state in which the image of the LED is further blurred at a diameter δ. [Diagram 25] FIG. 2 is an explanatory diagram of a pixel of an image sensor. [Figure 26] FIG. 2 is an explanatory diagram of the brightness sensed by each pixel of the image sensor. [Figure 27] FIG. 4 is an explanatory diagram of amplitude characteristics. [Figure 28] FIG. 13 is an explanatory diagram of amplitude characteristics approximated by an envelope. [Figure 29] 1 is an explanatory diagram showing a state in which a camera captures an image of a panel surface from an oblique direction. [Diagram 30] FIG. 2 is an explanatory diagram of the range captured by the camera. [Diagram 31] 1 is an explanatory diagram of the shortest and longest distances captured by a camera. [Diagram 32] FIG. 2 is an explanatory diagram of distances on an image sensor. [Diagram 33] FIG. 13 is an explanatory diagram of distance calculation using a sub-camera. [Diagram 34] FIG. 13 is an explanatory diagram of distance calculation using a sub-camera. [Diagram 35] 11 is a flowchart of a moire determination process according to an embodiment. [Diagram 36] FIG. 5 is an explanatory diagram of an LED wall 505. [Figure 37] FIG. 11 is an explanatory diagram of an arrangement that makes moire more noticeable. [Figure 38] 13 is a flowchart of another example of the moire determination process according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The embodiments will be described below in the following order. <1. Filming system and content production> <2. Configuration of information processing device> <3. Configuration with artifact determination unit> <4. Judgment process> <5. Summary and examples of modifications>

[0010] In this disclosure, "video" or "image" includes both still images and videos. Furthermore, "video" does not only refer to the state in which it is displayed on a display, but may also refer comprehensively to video data that is not displayed on a display. For example, in the embodiments, background images before being displayed on a display, images captured by a camera, etc. are not images that are actually displayed but are video data, but for convenience they will be referred to as "background images", "captured images", etc.

[0011] <1. Filming system and video content production> A shooting system and video content production to which the technology of the present disclosure can be applied will be described. 1 is a schematic diagram of a photography system 500. This photography system 500 is a system for performing photography as a virtual production, and the figure shows some of the equipment arranged in a photography studio.

[0012] The photography studio is provided with a performance area 501 where performers 510 perform acts and other performances. Large display devices are placed at least on the back, left and right sides, and top of this performance area 501. The type of device used for the display device is not limited, but the figure shows an example in which an LED wall 505 is used as an example of a large display device.

[0013] A single LED wall 505 is formed by arranging a plurality of LED panels 506 connected vertically and horizontally to form a large panel. The size of the LED wall 505 is not particularly limited, but may be a size necessary or sufficient for displaying the background when filming the performer 510.

[0014] A required number of lights 580 are placed at required positions, such as above or to the sides of the performance area 501, to illuminate the performance area 501.

[0015] A camera 502 for shooting movies or other video content is placed near the performance area 501. The camera 502 can be moved by a cameraman 512, and the cameraman 512 can also control the shooting direction, angle of view, etc. Of course, it is also conceivable that the movement of the camera 502 and the angle of view can be controlled by remote control. The camera 502 may also move and change the angle of view automatically or autonomously. For this reason, the camera 502 may be mounted on a pan head or a moving body.

[0016] Camera 502 captures performer 510 in performance area 501 and the image displayed on LED wall 505. For example, by displaying a landscape as background image vB on LED wall 505, it becomes possible to capture an image similar to that of performer 510 actually being in the location of that landscape and performing.

[0017] An output monitor 503 is placed near the performance area 501. The video captured by the camera 502 is displayed in real time on this output monitor 503 as a monitor video vM. This allows the director and staff producing the video content to check the video being captured.

[0018] In this way, the imaging system 500 that images the performance of the performer 510 with the LED wall 505 as a backdrop in a photography studio has various advantages over green screen photography.

[0019] For example, when shooting against a green screen, it can be difficult for the performer to imagine the background or the situation of the scene, which can affect the performance. By displaying the background image vB, the performer 510 can act more easily, improving the quality of the performance. It is also easy for the director and other staff to judge whether the performance of the performer 510 matches the background or the situation of the scene.

[0020] In addition, post-production is more efficient than with green screen shooting. This is because there are cases where so-called chromakey compositing is unnecessary, and there are also cases where color correction and reflection compositing are unnecessary. Even if chromakey compositing is required during shooting, it is also more efficient because there is no need to add a physical background screen, since it is only necessary to display green and blue images.

[0021] When shooting on a green screen, the performers' bodies, costumes, and objects tend to have a green tint, which needs to be corrected. Also, when shooting on a green screen, if there are objects that reflect the surrounding scenery, such as glass, mirrors, or snow globes, it is necessary to generate and composite images of the reflections, which is a time-consuming task.

[0022] In contrast, when shooting with the shooting system 500 in Fig. 1, the green hue does not increase, so correction for this is not necessary. Also, by displaying the background image vB, reflections on actual objects such as glass are naturally captured and shot, so there is no need to synthesize the reflected image.

[0023] Here, background image vB will be explained with reference to Figures 2 and 3. Even if background image vB is displayed on LED wall 505 and shooting is performed together with performer 510, simply displaying background image vB will result in an unnatural background in the shot image. This is because the background, which is actually three-dimensional and has depth, is displayed as two-dimensional background image vB.

[0024] For example, camera 502 can capture images of performer 510 in performance area 501 from various directions and can also zoom in. Performer 510 does not stand still in one place. In this case, the actual appearance of the background of performer 510 should change depending on the position of camera 502, the shooting direction, the angle of view, etc., but such changes cannot be obtained with background image vB as a flat image. Therefore, the background image vB is changed so that the background, including parallax, appears the same as how it actually appears.

[0025] Fig. 2 shows a state in which a camera 502 is shooting a performer 510 from a position on the left side of the figure, and Fig. 3 shows a state in which a camera 502 is shooting a performer 510 from a position on the right side of the figure. In each figure, a shooting area image vBC is shown within a background image vB. The portion of the background image vB excluding the shooting area image vBC is called the "outer frustum," and the shooting area image vBC is called the "inner frustum." The background image vB described here refers to the entire image displayed as the background, including the shooting area image vBC (inner frustum).

[0026] The range of this shooting area image vBC (inner frustum) corresponds to the range actually captured by the camera 502 within the display surface of the LED wall 505. The shooting area image vBC is an image that expresses a scene that can be seen when the position of the camera 502 is actually used as the viewpoint, depending on the position, shooting direction, angle of view, etc. of the camera 502.

[0027] Specifically, for the shooting area video vBC, 3D background data, which is a 3D (three dimensions) model, is prepared as the background, and the 3D background data is rendered sequentially in real time based on the viewpoint position of the camera 502. In reality, the range of the shooting area image vBC is set to be a little wider than the range shot by the camera 502 at that point in time. This is to prevent the image of the outer frustum from being captured due to a drawing delay when the shooting range changes slightly due to panning, tilting, zooming, etc. of the camera 502, and to avoid the influence of diffracted light from the image of the outer frustum. The image of the shooting area image vBC thus rendered in real time is composited with the image of the outer frustum. The image of the outer frustum used in the background image vB may be rendered in advance based on 3D background data, or may be rendered in real time for each frame or for each intermittent frame, and the image of the shooting area image vBC (inner frustum) is incorporated into a part of the image of the outer frustum to generate the overall background image vB. In some cases, the outer frustum image is also rendered every frame, just like the inner frustum, but here we will take a still image as an example, and in the following explanation we will mainly use the case where only the first frame of the outer frustum image is rendered.

[0028] As a result, even if the camera 502 is moved back and forth or left and right, or zoomed, the background of the area being photographed along with the performer 510 will be captured as an image that corresponds to the changes in viewpoint position and FOV (Field of View) that accompany the actual movement of the camera 502.

[0029] 2 and 3, the output monitor 503 displays a monitor image vM including a performer 510 and the background, which is the captured image. The background in this monitor image vM is the shooting area image vBC. In other words, the background included in the captured image is a real-time rendered image.

[0030] In this manner, in the embodiment of the shooting system 500, the background image vB is not simply displayed two-dimensionally, but the background image vB including the shooting area image vBC is changed in real time so that an image similar to that obtained when an actual landscape is shot can be captured.

[0031] In addition, it may be possible to reduce the processing burden on the system by rendering in real time only the shooting area image vBC as the range captured by the camera 502, rather than the entire background image vB displayed on the LED wall 505.

[0032] Here, we will explain the process of creating video content as a virtual production in which shooting is performed using the shooting system 500. As shown in Fig. 4, the video content production process is roughly divided into three stages: asset creation ST1, production ST2, and post-production ST3.

[0033] Asset creation ST1 is a process for creating 3D background data for displaying the background image vB. As described above, the background image vB is generated by rendering in real time using 3D background data during shooting. For this purpose, 3D background data is created in advance as a 3D model.

[0034] Examples of techniques for creating 3D background data include full computer graphics (CG), point cloud scanning, and photogrammetry.

[0035] Full CG is a method of creating 3D models using computer graphics. Of the three methods, this is the most labor-intensive and time-consuming method, but it is best used when you want to use unrealistic images or images that are difficult to actually shoot as background images vB.

[0036] Point cloud data scanning is a method to measure distance from a certain position using LiDAR, take 360-degree images from the same position with a camera, and generate a 3D model from point cloud data by placing the color data captured by the camera on the points measured by the LiDAR. 3D models can be created in a shorter time than full CG. It is also easier to create high-resolution 3D models than photogrammetry.

[0037] Photogrammetry is a photogrammetry technique that obtains dimensions and shapes by analyzing parallax information from 2D images taken from multiple viewpoints. It can create 3D models in a short time. In addition, when generating 3D data using photogrammetry, point cloud information obtained by LIDAR may also be used.

[0038] In Asset Creation ST1, for example, these methods are used to create 3D models that will become 3D background data. Of course, the above methods can be used in combination. For example, a part of a 3D model created by point cloud data scanning or photogrammetry can be created in CG and then composited.

[0039] Production ST2 is the process of shooting in a studio as shown in Figure 1. The elemental technologies in this case include real-time rendering, background display, camera tracking, and lighting control.

[0040] Real-time rendering is a rendering process for obtaining the shooting area image vBC at each point in time (each frame of the background image vB) as explained in Figures 2 and 3. This is a process for rendering the 3D background data created in asset creation ST1 from a viewpoint according to the position of the camera 502 at each point in time.

[0041] In this way, real-time rendering is performed to generate the background image vB for each frame including the shooting area image vBC, and the generated background image vB is displayed on the LED wall 505.

[0042] Camera tracking is performed to obtain shooting information from the camera 502, and tracks the position information, shooting direction, angle of view, etc. of the camera 502 at each point in time. By providing the shooting information including this to the rendering engine in correspondence with each frame, real-time rendering according to the viewpoint position of the camera 502, etc. can be performed.

[0043] The shooting information is information that is linked or associated with the video as metadata. The shooting information is expected to include position information of the camera 502 at each frame timing, the camera direction, angle of view, focal length, F-number (aperture value), shutter speed, lens information, and the like.

[0044] Illumination control refers to controlling the state of illumination in the photographing system 500, specifically controlling the amount of light, emission color, illumination direction, etc. of the light 580. For example, illumination control is performed according to the time setting and location setting of the scene to be photographed.

[0045] Post-production ST3 indicates various processes carried out after shooting, such as image correction, image adjustment, clip editing, and image effects.

[0046] Image correction may involve color gamut conversion or color matching between cameras or materials. Image adjustments may include color adjustments, brightness adjustments, contrast adjustments, and the like. Clip editing may involve cutting clips, adjusting the order, adjusting the length of time, and the like. As a video effect, CG images and special effect images may be combined.

[0047] Next, the configuration of the imaging system 500 used in production ST2 will be described. FIG. 5 is a block diagram showing the configuration of the imaging system 500 the outline of which is explained in FIG. 1, FIG. 2 and FIG.

[0048] 5 includes the above-mentioned LED wall 505 including a plurality of LED panels 506, a camera 502, an output monitor 503, and a light 580. The photography system 500 further includes a rendering engine 520, an asset server 530, a sync generator 540, an operation monitor 550, a camera tracker 560, an LED processor 570, a lighting controller 581, and a display controller 590, as shown in FIG.

[0049] Each of the LED processors 570 is provided corresponding to one or more LED panels 506, and drives the corresponding one or more LED panels 506 to display images.

[0050] The sync generator 540 generates a synchronization signal for synchronizing the frame timing of the image displayed by the LED panel 506 with the frame timing of the image captured by the camera 502 , and supplies this to each LED processor 570 , the camera 502 , and the rendering engine 520 .

[0051] The camera tracker 560 generates shooting information by the camera 502 at each frame timing, and supplies it to the rendering engine 520. For example, the camera tracker 560 detects, as one piece of shooting information, the position of the LED wall 505 or position information of the camera 502 relative to a predetermined reference position, and the shooting direction of the camera 502, and supplies these to the rendering engine 520. A specific detection method using the camera tracker 560 is to randomly place reflectors on the ceiling and detect the position from the reflected light of infrared light irradiated from the camera tracker 560 attached to the camera 502. Another detection method is to estimate the self-position of the camera 502 by using gyro information mounted on the pan head or main body of the camera 502, or by image recognition of the video captured by the camera 502.

[0052] In addition, the camera 502 may supply the rendering engine 520 with shooting information such as the angle of view, focal length, F-number, shutter speed, and lens information.

[0053] The asset server 530 is a server that can store the 3D model created by the asset creation ST1, i.e., the 3D background data, on a recording medium and can read out the 3D model as needed. In other words, it functions as a DB (database) for the 3D background data.

[0054] The rendering engine 520 performs processing to generate the background image vB to be displayed on the LED wall 505. To this end, the rendering engine 520 reads out necessary 3D background data from the asset server 530. The rendering engine 520 then generates an image of the outer frustum used in the background image vB by rendering the 3D background data as viewed from pre-specified spatial coordinates. Furthermore, the rendering engine 520 uses the shooting information supplied from the camera tracker 560 and the camera 502 to identify the viewpoint position and the like with respect to the 3D background data, and renders the shooting area image vBC (inner frustum).

[0055] Furthermore, the rendering engine 520 generates a background image vB as one frame of image data by synthesizing the outer frustum with the shooting area image vBC that dynamically changes in accordance with the movement of the camera 502. Then, the rendering engine 520 transmits the generated one frame of image data to the display controller 590.

[0056] The display controller 590 generates a divided video signal nD by dividing one frame of video data into video portions to be displayed on each LED panel 506, and transmits the divided video signal nD to each LED panel 506. At this time, the display controller 590 may perform calibration according to individual differences in color development and the like between display sections / manufacturing errors and the like. Note that these processes may be performed by the rendering engine 520 without providing the display controller 590. In other words, the rendering engine 520 may generate the divided video signal nD, perform calibration, and transmit the divided video signal nD to each LED panel 506.

[0057] Each LED processor 570 drives the LED panel 506 based on the divided video signal nD received, whereby the entire background image vB is displayed on the LED wall 505. The background image vB includes the shooting area image vBC that is rendered according to the position of the camera 502 at that time, etc.

[0058] In this way, camera 502 can capture the performance of performer 510, including the background image vB displayed on LED wall 505. The image captured by camera 502 is recorded on a recording medium inside camera 502 or in an external recording device (not shown), and is also supplied in real time to output monitor 503 and displayed as monitor image vM.

[0059] An operation image vOP for controlling the rendering engine 520 is displayed on the operation monitor 550. The engineer 511 can perform necessary settings and operations related to the rendering of the background image vB while viewing the operation image vOP.

[0060] The lighting controller 581 controls the light emission intensity, light emission color, irradiation direction, etc. of the light 580. The lighting controller 581 may control the light 580 asynchronously with the rendering engine 520, for example, or may control the light 580 synchronously with the shooting information and rendering processing. Therefore, the lighting controller 581 may control the light emission in response to an instruction from the rendering engine 520 or a master controller (not shown). The light 580 may also be controlled by the rendering engine 520.

[0061] FIG. 6 shows an example of processing by the rendering engine 520 in the imaging system 500 having such a configuration.

[0062] In step S10, the rendering engine 520 reads the 3D background data to be used this time from the asset server 530, and renders it in an internal work area. At this stage, an image to be used as the outer frustum may be generated.

[0063] Thereafter, the rendering engine 520 repeats the processes of steps S30 to S60 until it is determined in step S20 that the display of the background image vB based on the read 3D background data has ended.

[0064] In step S30, the rendering engine 520 acquires shooting information from the camera tracker 560 and the camera 502. This allows the position and state of the camera 502 to be reflected in the current frame to be confirmed.

[0065] In step S40, the rendering engine 520 performs rendering based on the shooting information. That is, rendering is performed by identifying the viewpoint position with respect to the 3D background data based on the position, shooting direction, or angle of view of the camera 502 to be reflected in the current frame. At this time, image processing reflecting the focal length, F-number, shutter speed, lens information, etc. can also be performed. This rendering makes it possible to obtain image data as the shooting area image vBC (inner frustum). The outer frustum may be generated in advance as a fixed image in step S10, or may be generated for each frame in step S40.

[0066] In step S50, the rendering engine 520 performs a process of synthesizing the outer frustum, which is the entire background image, with an image reflecting the viewpoint position of the camera 502, i.e., the shooting area image vBC. For example, this is a process of synthesizing an image generated by reflecting the viewpoint of the camera 502 with an image of the entire background rendered from a specific reference viewpoint. As a result, one frame of background image vB to be displayed on the LED wall 505, i.e., the background image vB including the shooting area image vBC, is generated.

[0067] The process of step S60 is performed by the rendering engine 520 or the display controller 590. In step S60, the rendering engine 520 or the display controller 590 generates divided video signals nD by dividing one frame of background video vB into videos to be displayed on the individual LED panels 506. Calibration may also be performed. Then, each divided video signal nD is sent to each LED processor 570.

[0068] Through the above process, the background image vB including the shooting area image vBC captured by the camera 502 is displayed on the LED wall 505 at each frame timing.

[0069] Although Fig. 5 shows only one camera 502, multiple cameras 502 can also be used for shooting. Fig. 7 shows a configuration example in which multiple cameras 502a and 502b are used. The cameras 502a and 502b are each capable of independently shooting the performance area 501. The cameras 502a and 502b and the LED processors 570 are kept synchronized by a sync generator 540.

[0070] Output monitors 503a and 503b are provided corresponding to the cameras 502a and 502b, and are configured to display the images captured by the corresponding cameras 502a and 502b as monitor images vMa and vMb, respectively.

[0071] Further, camera trackers 560a and 560b are provided corresponding to the cameras 502a and 502b, and detect the positions and shooting directions of the corresponding cameras 502a and 502b. Shooting information from the camera 502a and the camera tracker 560a and shooting information from the camera 502b and the camera tracker 560b are transmitted to the rendering engine 520.

[0072] The rendering engine 520 can perform rendering to obtain the background image vB of each frame using the shooting information from either or both of the camera 502a and the camera 502b.

[0073] Although FIG. 7 shows an example in which two cameras 502a and 502b are used, it is also possible to use three or more cameras 502 for shooting. However, when multiple cameras 502 are used and the shooting information of each is used to render and display the shooting area images vBC (inner frustum) corresponding to each camera 502, the shooting area images vBC interfere with each other. For example, in an example using two cameras 502a and 502b as shown in FIG. 7, the shooting area image vBC corresponding to the camera 502a is shown, but when the image of the camera 502b is used, the shooting area image vBC corresponding to the camera 502b is also required. In that case, if the shooting area images vBC corresponding to each camera 502a and 502b are simply displayed, they will interfere with each other. For this reason, some ingenuity is required regarding the display of the shooting area images vBC.

[0074] <2. Configuration of information processing device> Next, a configuration example of an information processing device 70 that can be used in the asset creation ST1, production ST2, and post-production ST3 will be described with reference to FIG. The information processing device 70 is a device capable of information processing, particularly image processing, such as a computer device. Specific examples of the information processing device 70 include personal computers, workstations, mobile terminal devices such as smartphones and tablets, video editing devices, etc. The information processing device 70 may also be a computer device configured as a server device or a computing device in cloud computing.

[0075] In the case of this embodiment, specifically, the information processing device 70 can function as a 3D model creation device that creates a 3D model in the asset creation ST1. The information processing device 70 can also function as a rendering engine 520 or an asset server 530 that constitutes the imaging system 500 used in production ST2. The information processing device 70 can also function as a video editing device that performs various types of video processing in post-production ST3.

[0076] Furthermore, the information processing device 70 can also function as an information processing device (rendering engine 31) that generates a simulation video vSM in previsualization, which will be described later with reference to FIG.

[0077] 8 executes various processes according to programs stored in a ROM 72 or a non-volatile memory unit 74 such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes, as appropriate.

[0078] The video processing unit 85 is configured as a processor that performs various types of video processing, such as 3D model generation processing, rendering, DB processing, video processing including color and brightness adjustment processing, video editing processing, video analysis and detection processing, or a combination of these.

[0079] This image processing unit 85 can be realized, for example, by a CPU separate from the CPU 71, a graphics processing unit (GPU), general-purpose computing on graphics processing units (GPGPU), an artificial intelligence (AI) processor, or the like. The video processing unit 85 may be provided as a function within the CPU 71 .

[0080] The CPU 71, ROM 72, RAM 73, non-volatile memory unit 74, and video processing unit 85 are interconnected via a bus 83. To this bus 83, the input / output interface 75 is also connected.

[0081] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, a key, a trackball, a dial, a touch panel, a touch pad, a grating panel, a remote controller, or the like. An operation by the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71. A microphone may also be used as the input unit 76. Voice uttered by the user may also be input as operation information.

[0082] Further, the input / output interface 75 is connected, either integrally or separately, to a display unit 77 formed of an LCD (Liquid Crystal Display) or an organic EL (electro-luminescence) panel, or the like, and an audio output unit 78 formed of a speaker, or the like. The display unit 77 is a display unit that performs various displays, and is configured, for example, by a display device provided in the housing of the information processing device 70, a separate display device connected to the information processing device 70, or the like. The display unit 77 displays various images, operation menus, icons, messages, etc., on the display screen based on instructions from the CPU 71, that is, displays as a GUI (Graphical User Interface).

[0083] The input / output interface 75 may be connected to a storage unit 79 and a communication unit 80 that are configured using a hard disk drive (HDD) or solid-state memory.

[0084] The storage unit 79 can store various data and programs. A DB can also be configured in the storage unit 79. For example, when the information processing device 70 functions as an asset server 530 or a rendering engine 31 described below, the storage unit 79 can be used to create a DB for storing a group of 3D background data.

[0085] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices such as an external DB, an editing device, and an information processing device via wired / wireless communication, bus communication, and the like. For example, when the information processing device 70 functions as a rendering engine 520 or a rendering engine 31 described below, the communication unit 80 can access a DB serving as an asset server 530 and receive shooting information from the camera 502 and the camera tracker 560. Also, in the case of the information processing device 70 used in post-production ST3, the communication unit 80 can access the DB serving as the asset server 530.

[0086] A drive 81 is also connected to the input / output interface 75 as required, and a removable recording medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. The drive 81 allows video data, various computer programs, and the like to be read from the removable recording medium 82. The read data is stored in the storage unit 79, and the video and audio contained in the data are output on the display unit 77 and the audio output unit 78. In addition, the computer programs, etc. read from the removable recording medium 82 are installed in the storage unit 79 as necessary.

[0087] In this information processing device 70, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via a removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.

[0088] <3. Configuration with artifact determination unit> An example of the configuration of an information processing device 70 including a determination unit that determines the occurrence of artifacts such as moire will be described below. Such an information processing device 70 may be an information processing device 70 such as the rendering engine 520 in the imaging system 500 shown in FIGS. 5 and 7, or may be an information processing device 70 used in a previsualization system 700 described in FIG.

[0089] In the imaging system 500, when an LED wall 505 is imaged by a camera 502, interference between the pixels of the image sensor of the camera 502 and the pixels of the LED panel 506 causes artifacts such as moire (moire, grid, color shift, etc.).

[0090] To avoid moire and the like, the camera 502 needs to blur the LED panel 506 rather than focus on it. However, changing the positions of the camera 502 and the performer 510 and adjusting the lenses at the actual shooting site reduces the efficiency of video production. Therefore, in this embodiment, the image capture system 500 and the pre-visualization system 700 use an information processing device 70 that includes a determination unit that performs processing to determine the occurrence of moire or the like.

[0091] First, the position of previsualization in the video production process is explained using Figure 9. Figure 9 shows the flow of asset creation (ST1), production (ST2), and post-production (ST3) previously explained in Figure 4. Production (ST2) is the stage where filming actually takes place in the studio, and is divided into pre-filming and filming.

[0092] It is expected that previsualization will be carried out mainly at a stage prior to the preparation for filming. Although the chronological order is not necessarily limited, previsualization may be carried out at a stage after the creation of a 3D model for rendering the background video vB in asset creation (ST1). In that sense, previsualization can be seen as taking place at the final stage of asset creation (ST1), or at the stage before filming preparation in production (ST2).

[0093] Previsualization involves processes such as image confirmation and moiré determination. Image confirmation is a process that allows the director and staff to check the shot image vC obtained by shooting prior to shooting. It mainly enables confirmation of the shot image vC, including its color tone, using simulation images.

[0094] Moire determination is a process that is performed, for example, when a staff member virtually operates the camera position while checking the image. If moire determination predicts that moire or the like will occur, a warning can be issued. This makes it possible to avoid artifacts occurring during subsequent shooting.

[0095] In the stage of preparation for shooting, color calibration is performed along with various settings of the LED, camera, etc. in the shooting system 500. Specifically, a calibration LUT (Look Up Table) is generated to cancel color changes that occur when the LED wall 505 is shot by the camera 502. The calibration LUT is, for example, a 3D-LUT. Note that instead of the 3D-LUT, a combination of a matrix and a 1D-LUT or preparation for calibration in other video signal processing may be performed.

[0096] Even at the stage of preparation for shooting, it is possible to perform moire detection and corresponding moire alert. That is, at the stage of preparation for shooting, the actual models of LED panel 506 and camera 502 are specified, and the position and direction of camera 502 are set, so that moire detection can be performed.

[0097] During the shooting stage, moiré detection, calibration LUT, and on-set camera settings are performed. The moire determination in this case refers to a moire determination by the rendering engine 520. A moire alert can also be output to the cameraman 512 or the like in real time depending on the determination.

[0098] The calibration LUT is a process for canceling color changes related to the background image display by the LED wall 505, the camera 502, etc., by applying an LUT generated during color calibration during preparation for shooting. On-set camera setting is a process in which the camera settings set in previsualization are obtained from, for example, a cloud server, and reflected on the camera 502 in the studio.

[0099] An example of the configuration according to the embodiment will be described with reference to Fig. 10. In Fig. 10, the configuration of the imaging system 500 and the configuration for previsualization are shown side by side. The upper part of the drawing shows a simplified configuration of the imaging system 500 described in FIG. 5 etc., and the lower part of the drawing shows a configuration used for previsualization (previsualization system 700).

[0100] 10, the imaging system 500 includes a rendering engine 520, an LED processor 570, an LED wall 505 formed of a plurality of LED panels 506, a camera 502, a camera signal processing unit 515, and an operation monitor 550 or an output monitor 503.

[0101] In this figure, the operation monitor 550 or the output monitor 503 collectively indicates a monitor that displays the shot video vC or the monitor video vM in the studio, which is the shooting site. For the sake of explanation, hereinafter, these will be abbreviated to "monitor 550, etc."

[0102] The rendering engine 520 is shown to include a rendering section 520a, a CG camera setting section 520c, and a determination section 520d.

[0103] As described above, the rendering unit 520a has the function of rendering the outer frustum and the inner frustum (image of the shooting area) vBC according to the position and shooting direction of the camera 502, and generating the background image vB to be displayed on the LED wall 505. That is, the rendering unit 520a has the function of performing the processing of FIG. The CG camera setting section 520c and the determination section 520d in the rendering engine 520 will be described later.

[0104] The rendering engine 520 may have a color calibration function. The color calibration function is a function for performing color conversion to cancel color changes caused by the processing of the LED processor 570, the LED light emission characteristics of the LED panel 506 constituting the LED wall 505, the light receiving characteristics and processing of the camera, etc.

[0105] That is, the rendering unit 520a may perform color conversion on the rendered background image vB using an LUT generated by color calibration at the shooting preparation stage. When performing color conversion using a calibration LUT in the rendering engine 520 in this manner, it is possible to prevent color change from occurring on the display side of the background image vB, for example. The color change referred to here is a color change that occurs due to the processing and light emission characteristics until the rendered background image vB is displayed on the LED wall 505, the light receiving characteristics and processing of the camera, and the like.

[0106] If the rendering engine 520 does not have a color calibration function, or if the function is provided but not enabled, there may be changes in the color of the rendered 3D model and the color of the displayed LED wall 505, such as changes in color tone or RGB balance.

[0107] 10 shows a camera signal processing unit 515 that processes an image signal captured by the camera 502. Although omitted in FIGS. 5 and 7, the camera signal processing unit 515 may be formed by a processor or the like in the camera 502, or may be provided as a device separate from the camera 502.

[0108] The video signal captured by the camera 502 is subjected to development processing including brightness processing, color processing, and resizing processing in a camera signal processing unit 515, and is output as a captured video vC or a monitor video vM.

[0109] The captured video vC is used as a main line video and is recorded on a recording medium, or is transmitted to other devices for broadcast or distribution. The captured video vC is also used as a monitor video vM and is displayed on a monitor 550 or the like.

[0110] For example, a previsualization system 700 is configured as a system outside the studio in relation to the above-described shooting system 500. The previsualization system 700 includes at least an information processing device 70 functioning as the rendering engine 31, and a monitor 40. The rendering engine 31 is configured by, for example, an information processing device 70 separate from the rendering engine 520 .

[0111] The rendering engine 31 includes a rendering unit 32 and a video processing unit 33 . The rendering unit 32 has a rendering function similar to that of the rendering unit 520a in the rendering engine 520 described above, and can generate the background image vBP based on a 3D model.

[0112] This rendering unit 32 can render the background image vB using at least the 3D model created in the asset creation (ST1) process. In other words, the rendering unit 32 obtains the same 3D model as that used in the production (ST2) process from, for example, the asset server 530 (see Figs. 5 and 7), and uses it to perform rendering.

[0113] The rendering unit 32 then uses the 3D model to render, for example, an inner frustum. When generating an inner frustum (a shooting area image vBC), any camera position and direction are input, and rendering can be performed at a viewpoint position for the 3D model corresponding to the input. These background images generated by the rendering unit 32 are referred to as "background images vBP" in order to distinguish them from the shooting area images vBC generated by the rendering engine 520 during shooting for the sake of explanation.

[0114] The video processing unit 33 has the function of performing various types of video processing on the background video vBP. As one type of image processing, the image processor 33 performs an actual shooting imaging process. The actual shooting imaging process is a process for generating a simulation image vSM that imitates the camera 502 that is actually used for shooting. In this actual shooting imaging process, the simulation image vSM is generated for the background image vBP using processing parameters that realize at least the luminance or color characteristics at the time of shooting by the camera 502 used in the shooting system 500.

[0115] By setting the brightness or color parameters to be the same as those of the camera 502 used during shooting, a simulation image vSM having the same color tone as the shot image vC is generated, and the simulation image vSM is displayed on the monitor 40.

[0116] The monitor 40 indicates a monitor device that displays an image output from the information processing device 70 serving as the rendering engine 31, and may be, for example, a monitor device located in a different place from the photography studio, or may be a monitor device located in the photography studio. The monitor 40 may be in any form as long as it displays the simulation image vSM at least at the previsualization stage.

[0117] In the rendering engine 31, the rendering unit 32 generates the background video vBP, and the video processing unit 33 performs actual shooting video processing to generate the simulation video vSM. By displaying this on the monitor 40, the staff can check the simulation video vSM, which has the same color tone as the background video included in the shot video vC at the time of shooting. This allows the color to be checked prior to shooting, and color adjustments can be made in the camera 502 and camera signal processing unit 515 in the preparation stage for shooting as necessary.

[0118] In the case of a shooting system 500 using a virtual background image vB such as the virtual production of this embodiment, the color tone of the shot image vC actually shot by the camera 502 may differ from what the director and staff had envisioned. In such a case, it becomes necessary to adjust the color tone by changing various settings at the shooting site, such as a studio, which is an extremely troublesome and time-consuming task in reality. This reduces the efficiency of video production. Therefore, it is extremely useful to be able to check an image with the same color tone as the actual shot image vC using the pre-visualization system 700 prior to shooting.

[0119] In the actual shooting imaging process of the image processing unit 33, at least the color tone of the background image vBP rendered by the rendering unit 32 is made equivalent to that of the background image contained in the actually shot image vC. To this end, specifically, the image processor 33 adjusts parameters such as RGB gain, white balance value, gamma value, offset value, etc., to those of the camera 502 .

[0120] For example, the image processor 33 performs a process of applying R, G, and B gains to the background image vBP that are the same as the gain values ​​for the pixel signals of each color read out from the image sensor in the camera 502 . Furthermore, the image processor 33 performs white balance processing on the background image vBP in the actual shooting imaging process using the same white balance value as that used when the camera 502 shot the image. Furthermore, parameters such as the spectral transmittance of optical components placed in the optical path, such as lenses, color conversion filters, ND filters, optical low-pass filters, color separation prisms, etc., the spectral transmittance of on-chip color filters and on-chip lenses of the image sensor, the spectral sensitivity characteristics of the photodiodes of the image sensor, etc. are also taken into consideration. Specifically, the parameters are calculated by integrating the product of the spectral transmittance and the spectral sensitivity characteristics of the LEDs in the LED panel 506, if a diffusion plate, anti-reflection film, protective film, etc. is attached to the LED panel, and the spectral transmittance of the LEDs in the LED panel 506, by wavelength. Then, color processing according to the parameters is performed. These operations enable the simulation image vSM to be obtained, which has a background of the same color as the background in the captured image vC. This allows the monitor 40 to confirm an image of the same color as when the captured image vC is viewed on the monitor 550 or the like during shooting.

[0121] Furthermore, as an actual shooting imaging process, the image processor 33 may perform a process for making the angle of view, based on the cropping range from the image sensor of the camera 502 and the zoom magnification setting, the same for the background image vBP.

[0122] Furthermore, the image processor 33 may perform a process for making the background image vBP into an actual captured image by reflecting the lens distortion characteristics of the camera 502. This makes the simulation image vSM closer to the captured image vC.

[0123] Incidentally, when color changes up to the LED wall 505 are cancelled by color calibration processing in the rendering engine 520 during shooting, the actual shooting imaging processing may be processing that reflects color changes on the camera side. This is because it is considered that the background image vB displayed on the LED wall 505 does not change in color from the rendered state. On the other hand, if the above-mentioned color calibration is not performed, color changes may already occur in the background image vB displayed on the LED wall 505. In such a case, the actual shooting imaging process in the image processing unit 33 also performs processing that reflects the luminance (intensity) characteristics of each color on the LED side. This makes it possible to make the simulation image vSM closer to the shot image vC.

[0124] As a premise for the above-mentioned actually captured imaging process, the camera settings such as the frame rate, shutter speed, and exposure value of the virtual camera assumed when the rendering engine 31 renders the simulation video vSM are set to be the same as those of the actual camera 502. That is, the video processing unit 33 generates the simulation video vSM that mimics the color tone of the actually captured video vC by performing color processing such as RGB gain and white balance value on the background video vBP so that the RGB values ​​correspond to various camera settings such as the frame rate, shutter speed, exposure value, and ND (Neutral Density) setting. It is also advisable to reflect the brightness of the LED wall 505 in the same way as the camera settings.

[0125] The rendering engine 31 includes an adjustment operation unit 35 that can be used by a user such as a staff member to perform color adjustment operations. For example, a GUI using the display unit 77 and input unit 76 of the information processing device 70, which is the rendering engine 31, is provided to the user. This allows the user to manipulate the color tone of the image while watching the monitor 40. That is, when the staff looks at the simulation image vSM obtained by the actual shooting imaging process and feels that the color tone is different from what they had in mind, they can make color adjustments to obtain the desired color.

[0126] In addition, the content of the adjustment is provided as color conversion information Ic to the camera 502 and the camera signal processing unit 515 via the cloud server 41, for example, so that the adjusted color can be reflected at the time of shooting. The color conversion information Ic is, for example, an LUT or a CDL (Color Decision List). The video processing unit 33 may transmit the color conversion information Ic to the camera 502 or the camera signal processing unit 515 without passing through the cloud server 41.

[0127] Also, the user may be able to change and set various camera settings using the adjustment operation unit 35. The video processing unit 33 performs a process of changing the camera settings to be reflected in the simulation video vSM in response to an input, and is able to output camera setting information CS.

[0128] The camera setting information CS here refers to, for example, the read size setting from the image sensor of the camera 502, the shutter speed setting, the exposure setting, the ND setting, the lens characteristics, the focus / iris / zoom operation settings, the white balance value, the RGB gain, and the like. In the above-mentioned actual shooting imaging process, these setting values ​​are basically adjusted to match the settings of the camera 502 used for shooting, thereby simulating the actual shot video vC.

[0129] In addition, by adjusting these camera settings, various camera settings can be prepared in advance. The video processor 33 can store and transmit camera setting information CS indicating the camera settings changed in response to the operation. For example, the camera setting information CS corresponding to the adjustment operation by the operator can be uploaded to the cloud server 41 so that it can be reflected in the camera 502 etc. at a later time when shooting.

[0130] Furthermore, the rendering engine 31 includes a determination unit 36, a position operation unit 37, and a communication unit 38, and is capable of performing moire determination and moire alert processing.

[0131] The position operation unit 37 is an interface function that enables the user to arbitrarily operate the virtual camera position.

[0132] The determination unit 36 ​​is a function that uses a specific algorithm to determine whether or not moire occurs in a set camera position. This algorithm, which will be described in detail later, is an algorithm that determines whether artifacts such as moire and scan lines will occur when actually photographing.

[0133] The determination unit 36 ​​performs the determination process using various information acquired by the communication unit 38, information on the virtual camera position operated by the position operation unit 37, and information input by the user from the adjustment operation unit 35. Specifically, the determination unit 36 ​​determines the occurrence of artifacts such as moire using the spacing between the LEDs, which are light-emitting elements of the LED panel 506 used in shooting (the spacing between adjacent LEDs), the pixel spacing (the spacing between adjacent pixels) of the image sensor of the camera 502 used in shooting, the characteristics of the optical LPF, the focal length of the lens, the F-number, the focusing distance, and the distance between the LED panel 506 and the camera 502. In the pre-visualization stage, since the actual LED panel 506 and camera 502 do not exist, the distance between the LED panel 506 and the camera 502 is a virtual value according to the operation of the position operation unit 37 .

[0134] For this reason, communication unit 38 communicates with, for example, LED panel 506, camera 502, etc. actually used for shooting, acquires values ​​such as LED spacing, image sensor pixel spacing, optical LPF characteristics, etc., and provides these to determination unit 36. Regarding the LED spacing, image sensor pixel spacing, optical LPF characteristics, etc., communication unit 38 may communicate with LED panel 506 and camera 502 to acquire information such as model and model number, and determination unit 36 ​​may refer to an internal DB to acquire the above-mentioned values ​​according to the model. Also, the determination unit 36 ​​may acquire each of the above values ​​in response to a specific value or model input by the user. Regarding the focal length, F-number, and focusing distance, the determination unit 36 ​​acquires them in accordance with the camera settings and operations of the virtual camera by the user, and uses them in the determination process. Furthermore, the determination unit 36 ​​can obtain the value of the distance between the LED panel 506 and the camera 502 according to the camera position according to the operation of the position operation unit 37, and use this for the determination.

[0135] When the determination unit 36 ​​predicts the occurrence of moire in the determination process, it notifies the image processing unit 33 of information on the occurrence of the moire, information on the position where the moire has occurred, and the like. The image processor 33 causes the monitor 40 to display, for example, a moire warning or an image showing the position of the moire.

[0136] For example, FIG. 11 is an image including a simulation image vSM of a background image vBP, and further includes a virtual LED wall 54 and a virtual camera 51. In FIG. The video processing unit 33 not only displays the simulation video vSM of the background video vBP, but also makes it possible to display, for example, a video in which the viewpoint position is set at a certain position in the studio.

[0137] In the image of FIG. 11, position information 52 of an LED wall 54 and position information 52 of a camera are further displayed. The origin information 50 indicates the origin position that serves as the reference for each position in space and the three axial directions of X, Y, and Z.

[0138] FIG. 12 shows an example in which a moire alert 53 is displayed and a moire position 57 is also shown. For example, suppose that the position of the camera 51 has been changed to that shown in Figure 11, and moire occurrence is predicted under this condition. In this case, a moire alert 53 is displayed, and the position where moire will occur is displayed as a moire position 57.

[0139] The position of camera 51 on this image can be changed arbitrarily by the user. For example, when the user performs an operation to specify camera 51, movement axes 55 in the X, Y, and Z directions are displayed as shown in Fig. 13. When the user performs an operation such as dragging while aligning cursor 56 with a certain movement axis 55, the position of camera 51 in the virtual space is changed. For example, the above-mentioned user interface is provided by the function of position operation unit 37.

[0140] Therefore, the user can move the virtual camera 51 as desired and check whether or not moire occurs. This makes it possible to set the camera position and camera path during actual shooting so that moire is not generated.

[0141] FIG. 14 shows the process of determining moire and the like by the rendering engine 31. In step S401, the rendering engine 31 monitors whether or not it is a decision opportunity. For example, it monitors whether or not the camera position has been manipulated. When the user operates the camera position using the function of the position operation unit 37, the rendering engine 31 proceeds to step S402 and performs a determination process using the information on the changed camera position and shooting direction. That is, the determination process is performed using the values ​​of the LED interval, image sensor pixel interval, optical LPF characteristics, focal length, F value, focusing distance, and distance between the LED panel 506 and the camera 502 acquired at that time. The distance between the LED panel 506 and the camera 502 is calculated by calculating a virtual positional relationship according to the user's camera position operation, and the distance is found. In addition, the determination opportunity monitored in step S401 is not limited to the camera position operation. For example, the process may proceed to step S402 when a zoom operation, a focus operation, an exposure adjustment operation, or the like is performed as a virtual camera operation. In other words, the elements used in the judgment process are the LED spacing, the image sensor pixel spacing, and the characteristics of the optical LPF, which do not normally change, but the judgment opportunity may be an opportunity where any of the focal length, F-number, focusing distance, and the distance between the LED panel 506 and the camera 502 change.

[0142] In step S403, the rendering engine 31 checks the result of the determination process, and if it is determined that moire or the like will occur, in step S404, a process of outputting a moire alert is performed.

[0143] As a result of the above, a moire alert 53 as shown in Fig. 12 is displayed, which allows staff to know about the occurrence of moire, etc. before shooting begins. Therefore, they can consider camera positions and camera paths that will prevent moire, etc. from occurring before the actual shooting begins.

[0144] Incidentally, in FIG. 10, the rendering engine 520 includes the CG camera setting unit 520c and the determination unit 520d in addition to the rendering unit 520a. This allows the rendering engine 520 to perform moire determination during preparation for shooting or during actual shooting.

[0145] The CG camera setting unit 520c receives shooting information from the camera tracker 560, such as the position of the camera 502, the shooting direction, the angle of view, the focal length, the F-number, the shutter speed, lens information, and the like, and causes the rendering unit 520a to perform rendering according to that information.

[0146] The determination unit 520d performs a process of determining whether or not moire or the like is currently occurring in the rendered shooting area image vBC by using the information obtained by the CG camera setting unit 520c. In other words, the determination unit 520d determines the current occurrence of moire or the like by using the same algorithm as the above-mentioned determination unit 36. If it is determined that moire or the like has occurred, then, for example, that information is transmitted to the camera 502, and a moire alert is displayed on the viewfinder or the like.

[0147] An example of such a process for real-time moire alert is shown in Fig. 15. In Fig. 15, in addition to steps S10 to S60 described in Fig. 6, steps S70 and after are also performed. As described above, the rendering engine 520 performs the processes from step S30 to step S60 for each frame timing, thereby displaying the background image vB on the LED wall 505.

[0148] The rendering engine 520 monitors whether it is a chance to make a decision in step S70. This is similar to step S401 in Fig. 14, but in this case, for example, the position of the camera 502 and shooting information obtained from the actual camera tracker 560 are obtained to check whether it is a chance to make a decision. If it is an opportunity for judgment, the flow proceeds to step S71 where judgment processing is performed. It should be noted that the check in step S70 may not be performed, and the determination process may be performed for each frame.

[0149] In step S72, the rendering engine 520 checks the result of the determination process, and if it is determined that moire or the like has occurred, it performs a process of outputting a moire alert in step S73. For example, the rendering engine 520 transmits information about the moire alert to the camera 502, so that the moire alert is displayed in a viewfinder or the like. Alternatively, the moire alert may be displayed on the monitor 550 or the like.

[0150] As a result, the cameraman 512 can recognize that moire is currently occurring during shooting and deal with it by, for example, moving the camera position or adjusting the focus.

[0151] The above is the determination process during shooting, but the same process is possible during preparation for shooting in the shooting system 500. In addition, even during a period when the background image vB is not being rendered, such as during preparation for shooting, in other words, during a period when the background image vB is not being displayed on the LED wall 505, it is also possible to determine whether or not moire or the like will occur depending on the position of the camera 502. This is because the LED spacing of the LED panel 506, the pixel spacing of the image sensor of the camera 502, and the characteristics of the optical LPF can be acquired in advance, and the focal length, F-number, and focusing distance of the lens of the camera 502, and the distance between the LED panel 506 and the camera 502 can be acquired as shooting information from the camera tracker 560.

[0152] <4. Judgment process> The above-described determination process executed by the determination unit 36 ​​and the determination unit 520d will be described in detail. First, the calculation of the size of the image formed on the image sensor of the camera will be described.

[0153] 16 shows the relationship between the optical axis of the lens 21, a subject AA', and a real image BB'. The lens 21 conceptually represents one or more lenses of the camera 502, for example.

[0154] The light that enters lens 21 parallel to the optical axis (line segment A'O' in the figure) passes through focal point C of lens 21. On the other hand, the light that passes through center O of lens 21 (line segment A'O in the figure) continues traveling straight. A real image is formed at point B' where these two lines intersect. The distance between the lens 21 and the subject is "s", the focal length of the lens 21 is "f", and the distance between the lens 21 and the real image is "i". Since △AOA' and △BOB' are similar, the following (Equation 1) holds. Note that "△" represents a triangle. For example, "△AOA'" is a triangle with vertices A, O, and A' in the figure.

[0155]

number

[0156] Also, since △OCO' and △BCB' are similar, (Equation 2) holds.

[0157]

number

[0158] And since line segment AO and line A'O' are parallel, AA'=OO' If we eliminate i from the above, we get:

[0159]

number

[0160] Therefore, the magnification of the image formed on the image sensor of the camera 502 can be calculated from the distance s from the lens 21 to the subject and the focal length f of the lens 21 using the above formula (3).

[0161] In addition, the lens formula (Equation 4) can be derived from the similarity relations between (Equation 1) and (Equation 2).

[0162]

number

[0163] Note that when s < f, a real image is not formed. Since i → ∞ as s → f, s > f due to physical constraints.

[0164] Next, the image blur will be described. Generally, in a camera using a solid-state imaging device, an optical LPF is inserted between the lens and the image sensor to suppress aliasing distortion caused by sampling at the pixels of the image sensor. FIG. 17 shows a state where an optical LPF 22 is disposed between a lens 21 and an image sensor 20.

[0165] The optical LPF 22 uses a material such as calcite or quartz whose refractive index varies depending on polarization, and blurs the image by doubling it through birefringence. FIG. 18 shows the birefringence by the optical LPF 22. The width indicated by "b" is the amount by which the image is shifted (the amount of blur) due to the characteristics of the optical LPF 22, and is denoted as "shift amount b" for explanation. Assuming that the image of the subject becomes a double image with a shift amount b as shown in FIG. 18, its frequency characteristics are as follows after Fourier transform.

[0166]

Equation

[0167] Since the amplitude characteristic is √{(real part)^2 + (imaginary part)^2}, it becomes as follows. Note that "^" represents exponentiation, and "^2" means squared. ω is the angular frequency and is 2π × spatial frequency.

[0168]

Equation

[0169] Generally, the shift amount b is set to 1 / 2 the pixel width in order to insert a zero point in the pixel sampling frequency, but some cameras narrow or widen the bandwidth. In addition, optical LPFs 22 may be overlapped to provide zero points at two frequencies. For example, when two optical LPFs 22 with shift amounts b and b' are overlapped, the amplitude characteristics are in the form of a product, as shown below.

[0170]

number

[0171] In order to increase sensitivity, the photodiode of the image sensor 20 is focused by an on-chip lens. If we can ignore the vignetting caused by wiring and the like and assume that all the light incident on the pixel 20P with width Ws as shown in Figure 19 is focused on the photodiode, the frequency characteristics are Fourier transformed as follows:

[0172]

number

[0173] Here, assume that the image is blurred so that the light passing through the lens 21 does not converge to a single point but spreads out in a circle with a diameter δ. FIG. 20 shows the state of image blurring together with the lens 21 and the aperture 23. Suppose the subject is at position A and its real image is formed at position B (solid line). As shown in the figure, the distance from lens 21 to the subject is "s", the distance from lens 21 to the real image is "i", the focal length of lens 21 is "f", and the aperture of lens 21 narrowed by aperture 23 is "d" (length of OO').

[0174] As shown by the dashed dotted line, when the focus is set at a position Af a distance Δsf behind the subject, the real image can be placed at a position Bf a distance Δif forward of position B, and at position B it is blurred to a diameter δ. Furthermore, when the focus is set at a position An a distance Δsn in front of the subject as shown by the dashed line, the real image can be placed at a position Bn a distance Δin behind position B, and at position B it is blurred to a diameter δ.

[0175] The relationship between the distance to the subject, the distance to the real image, and the focal length in each of the cases shown by the solid line, dashed line, and broken line above is as follows, based on the above (Equation 4).

[0176]

number

[0177] And since △OBfO' and △B'BfB” are similar, we get the following.

[0178]

number

[0179] Also, since △OBnO' and △B'BnB” are similar, we get the following.

[0180]

number

[0181] If the F-number of the lens 21 is "F", then the following holds true from the definition.

[0182]

number

[0183] Therefore, by eliminating i, Δif, Δin, and d from the above, the diameter δ can be expressed as follows.

[0184]

number

[0185] Considering the focal length l of the lens 21, it can be rewritten as follows. The focal length is also called the shooting distance, but it is the distance between the lens and the subject when the subject is in focus. In the following, we will use the term focal length.

[0186]

number

[0187] The diameter δ of the blur when the focal length of the lens 21 is shifted to "l" relative to the actual distance s to the subject can be calculated from the focal length f of the lens 21 and the F-number F of the lens 21 in addition to the distance s and the focal length l, as shown in the above equation (14). As described in the explanation of equation (4), s>f and l>f, and the F-number F does not become zero, so if f>0, the denominator of equation (14) will not become zero.

[0188] The frequency characteristics when the image is blurred in a circular shape with a diameter δ can be calculated by Fourier transform as follows:

[0189]

number

[0190] However, since it is difficult to solve this integral, if we approximate it with a regular dodecagon as shown by the dashed line in Figure 21, it can be calculated using the partial integration formula as follows.

[0191]

number

[0192] Here, the calculation is made assuming that the inside of a circle with a diameter δ is blurred with uniform brightness, but in reality, the light beam may be vignetted inside the lens and may not be round. However, many expensive lenses used to take artistic works produce beautifully round blurs, so this formula will be applied.

[0193] The LED panel 506 has a high contrast ratio by making the LED area as small as possible and widening the area of ​​the black background. For this reason, when the LED panel 506 is photographed, it appears as shown in Fig. 22. The figure shows an area where six LEDs are lined up as the image of the LEDs on the image sensor 20 when the LED panel 506 is photographed by the camera 502. The white squares in the figure are the images of the LEDs.

[0194] Now, assuming that the actual spacing between the LEDs on LED panel 506 is "Wp", the distance between camera 502 and LED panel 506 is "p", and the focal length of lens 21 is "f", the spacing between the LED images on image sensor 20 of camera 502 can be expressed as the following (Equation 17) based on the above (Equation 3).

[0195]

number

[0196] The images with such an interval become a double image separated by the shift amount b by the optical LPF 22. However, since actual pixels are arranged two-dimensionally, they are actually divided vertically and horizontally, resulting in a quadruple image, top, bottom, left and right, as shown in FIG.

[0197] When this is blurred by a diameter δ, the square LED image becomes shaped like a barrel. Depending on how many of the blurred LED images overlap, uneven brightness occurs as shown in Figure 24, which causes moiré and other artifacts.

[0198] This is captured by an image sensor 20 with pixels spaced apart by Ws, as shown in Figure 25. The integral value of the light that falls within the range of a pixel becomes the output of the pixel. The result is the difference in brightness perceived by each pixel, as shown in Figure 26. Since the brightness changes slightly depending on the relative positions of the LED image and the image sensor pixels, a low-frequency moire pattern longer than the spacing between the LED images is observed.

[0199] Let us consider to what extent this LED image is suppressed by the optical LPF 22, the integration effect at the pixels, and the blur at the lens 21.

[0200] Since the repetition rate of the LED images is the reciprocal of the spacing between the LED images, the angular frequency ω can be calculated using the following formula:

[0201]

number

[0202] Considering the distance (shift amount b) between the double images formed by the optical LPF 22, the amplitude Af of the optical LPF 22 can be calculated from the above-mentioned (Equation 6) by the following (Equation 19).

[0203]

number

[0204] The amplitude Ap due to the integral effect at a pixel can be calculated from the above (Equation 8) by the pixel interval Ws of the image sensor 20 using the following (Equation 20).

[0205]

number

[0206] The amplitude Ab due to the blur of the lens 21 can be calculated approximately by the following formula using the F-number F of the lens 21 and the focal length l of the lens 21, based on the above (Formula 14) and (Formula 15).

[0207]

number

[0208] The pixel interval Ws and the shift amount b are constants determined by the model of the camera 502, and the LED interval Wp of the LED panel 506 is a constant determined by the model of the LED panel 506. Therefore, the amplitude Af of the optical LPF 22 and the amplitude Ap due to the integration effect at the pixel change depending on the distance p between the camera 502 and the LED panel 506 and the focal length f of the lens 21.

[0209] The amplitude Ab due to the blur of the lens 21 is a function of ωδ (ω: angular frequency, δ: diameter of the blur). From the above (Equation 18) and (Equation 21), ωδ can be calculated by the following equation.

[0210]

number

[0211] Therefore, the amplitude Ab changes depending on the F-number F of the lens 21 and the focal length l of the lens 21 in addition to the distance p between the camera 502 and the LED panel 506 and the focal length f.

[0212] The amplitude A of the LED image can be calculated as follows:

[0213]

number

[0214] If this amplitude A is sufficiently attenuated and becomes smaller than a certain threshold, it can be determined that no artifacts such as moire will occur.

[0215] The amplitude Ab due to blur can be expressed as a function of (ωδ) as follows, from the above (Equation 21) and (Equation 22).

[0216]

number

[0217] If we plot a graph with ωδ on the horizontal axis and amplitude Ab on the vertical axis, we will get a graph with multiple maximum and minimum values, rather than a monotonically decreasing value as in Figure 27. When trying to remove artifacts such as moire by shifting the focus of the lens 21 at the shooting site, it can be confusing if the amplitude Ab decreases and increases (repeated minimum and maximum values) as the focal distance l of the lens 21 changes, for example, by turning the focus ring. Also, as mentioned above, the calculation formula for the amplitude Ab is an approximation and not an exact calculation.

[0218] Therefore, it is easier to use in the field if we replace it with a function in which the amplitude Ab decays monotonically according to ωδ. Since a larger amplitude estimate provides more leeway for use in warning of the occurrence of moire and other artifacts, we consider approximating the amplitude with an envelope curve that traces the maximum value, as shown by the solid line in Figure 28.

[0219] Specifically, we use the following approximate formula for each range from ωδ<6.487887 to 86.34829≦ωδ. Note that each range from 6.487887?ωδ<17.240423 to 63.501086≦ωδ<86.34829 is approximated by a linear formula.

[0220]

number

[0221] Furthermore, from past experience, moire and the like becomes almost invisible if suppressed to 0.5% or less, so it is generally considered acceptable to stop at this level.

[0222] Incidentally, when considering the relative positional relationship between the camera 502 and the LED panel 506, it is rare for the panel surface and the image sensor surface to be directly opposed and parallel to each other, and in reality, the camera 502 often captures the LED panel 506 from an oblique direction as shown in Fig. 29. In such cases, the distance p between the camera 502 and the LED panel 506 changes depending on the location within the panel.

[0223] Since the focus is usually set on a subject (such as performer 510) in front of LED panel 506, the focal distance l of lens 21 is shorter than distance p. Therefore, it is expected that the amplitude Ab due to blur will be maximum at the shortest distance p within the range captured by camera 502.

[0224] However, when a wide-angle lens is used at a shallow angle, it is possible that l>p, and the amplitudes Af and Ap do not decrease monotonically as the distance p increases, so the amplitude A of the LED image does not necessarily reach its maximum at the shortest distance p.

[0225] Therefore, when camera 502 captures an image of LED panel 506 from an oblique direction as shown in FIG. 29, the amplitude A of the LED image is calculated by finely dividing the area between the shortest distance p of the area captured by camera 502 and the longest distance p of the area captured by camera 502, and the maximum value thereof is used to determine whether or not artifacts such as moire will occur.

[0226] Even if LED panel 506 is not flat but curved, the amplitude A of the LED image is calculated by finely dividing the distance between the shortest distance p of the range captured by camera 502 and the longest distance p of the range captured by camera 502, and the maximum value is used to determine whether or not artifacts such as moire occur.

[0227] Fig. 30 shows an image through the lens 21, roughly similar to Fig. 16. The range r captured by the camera on the subject side is given by the focal length l of the lens 21, the focal length f of the lens 21, and the width w of the imaging surface of the image sensor 20, using the above-mentioned (Equation 3), as shown in the following (Equation 26).

[0228]

number

[0229] Therefore, the angle θ in FIG. 21 can be calculated using the following formula. The range captured by the camera 502 is within an angle range of ±θ from the optical axis of the lens 21 .

[0230]

number

[0231] As shown in FIG. 31 , when the range captured by camera 502 is at an angle of ±θ with respect to the optical axis and LED panel 506 is tilted at an angle Φ (≦90°) with respect to the optical axis, the relational equation between the distance AD ​​between camera 502 and LED panel 506 along the optical axis and the distances AB and AC of the ends of the range captured by camera 502 is as follows:

[0232]

number

[0233] If the foot of the perpendicular line drawn from the camera 502 to the LED panel 506 is "H", the shortest distance between the camera 502 and the LED panel 506 is AH.

[0234]

number

[0235] When Φ≧90°−θ, the foot H of the perpendicular line falls within the range captured by the camera 502, and therefore the shortest distance between the camera 502 and the LED panel is the distance AH. In calculating the diameter δ of the blur, the distance on the optical axis is used as in Figures 16 and 20, so the distance AG from the foot H of the perpendicular line to the foot G of the perpendicular line dropped to the optical axis is used. The distance AG is calculated as follows:

[0236]

number

[0237] When Φ<90°-θ, foot H of the perpendicular line is outside the range captured by camera 502, so the shortest distance between camera 502 and LED panel 506 is distance AB. Since the distance on the optical axis is used in calculating the diameter δ of the blur, it is the distance AE from point B to foot E of the perpendicular line dropped to the optical axis.

[0238]

number

[0239] The maximum distance between camera 502 and LED panel 506 is distance AC. The distance on the optical axis is used in calculating the diameter δ of the blur, so it is distance AF from point C to foot F of the perpendicular line dropped to the optical axis.

[0240]

number

[0241] Distances on the image sensor 20 can be measured from the image if the pixel dimensions are known. As shown in Figure 32, the angle θ between the optical axis and a line (dashed line in the figure) as viewed from lens 21 of a point located at a distance ds from the optical axis on image sensor 20 is given by the following (Equation 33) using the above (Equation 3) from the focal length l and focal length f of lens 21.

[0242]

number

[0243] Here, it is assumed that an auxiliary camera 502S for distance measurement is provided in addition to the camera 502 for photographing. The auxiliary camera 502S may be an additional imaging optical system fixed to or built into the camera 502, or may be a separate camera.

[0244] As shown in Figure 33, when the same point P is imaged by camera 502 and auxiliary camera 502S, whose optical axes are parallel and separated by a distance dc, the angles formed by lines AP and BP connecting each camera (502, 502S) to point P and the optical axis of each camera are α and β. The distance PH from point P to the foot H of the perpendicular line drawn to the line AB connecting the two cameras (502, 502S) can be calculated as follows.

[0245]

number

[0246] The distance AP between the camera 502 and point P, and the distance BP between the auxiliary camera 502S and point P can be calculated as follows.

[0247]

number

[0248] In this way, using two cameras, it is possible to measure the distance to any point from the captured image.

[0249] When point P is located outside the optical axis of camera 502 as in FIG. 34, the angle is in the opposite direction to that in FIG. 33, so calculation should be performed assuming a negative value.

[0250] Although the above examples show distance measurement using two cameras, it is also possible to use a distance measurement sensor. It is also possible to obtain distance values ​​from information from the camera tracker 560. Furthermore, if the position of the camera 502 is fixed, the distance can be actually measured with a tape measure or the like at the shooting site and input.

[0251] So far, the concept of the process for determining the occurrence of moire etc. has been explained. Fig. 35 shows the flow of the determination process executed by the information processing device 70 based on the above concept. The information processing device 70 here refers to an information processing device having a function as, for example, the determination unit 36 ​​or the determination unit 520d in Fig. 10. Specifically, in the example of Fig. 10, the information processing device 70 is the rendering engine 31 or the information processing device 70 is the rendering engine 520. Other information processing devices may also be used.

[0252] The process in FIG. 35 is an example of a process executed by the CPU 71 of the information processing device 70 in accordance with a program for determining moire and the like.

[0253] In step S501, the CPU 71 acquires the LED spacing Wp of the LED panel 506. The LED spacing Wp is a fixed value depending on the model. The CPU 71 can acquire information on the model of the LED panel 506 by communication with other devices or operator input, and acquire the LED spacing Wp by referring to the DB depending on the model. For this reason, for example, the non-volatile memory unit 74 or the storage unit 79 is provided with a DB that stores the LED spacing Wp depending on the model of the LED panel 506. Alternatively, the CPU 71 may directly acquire the value of the interval Wp through input by an operator or through communication between devices.

[0254] In step S502, the CPU 71 acquires the pixel interval Ws of the image sensor 20 of the camera 502 and the shift amount b of the optical LPF. These values ​​are fixed depending on the model of camera 502. CPU 71 can obtain information on the model of camera 502 through communication with other devices, operator input, or the like, and can acquire pixel interval Ws and shift amount b by referring to a DB according to the model. For this reason, for example, non-volatile memory unit 74, storage unit 79, or the like is provided with a DB that stores LED interval Wp according to the model of camera 502. Alternatively, the CPU 71 may directly acquire values ​​of the pixel interval Ws and the shift amount b through input by an operator or communication between devices.

[0255] In step S 503 , the CPU 71 acquires the focal length f, the F-number F, and the in-focus distance l of the lens 21 of the camera 502 . For example, the CPU 71 may be able to acquire these values ​​through communication with the camera 502 (lens barrel), or the values ​​may be acquired through operator input of numerical values ​​displayed on the barrel.

[0256] In step S504, the CPU 71 acquires the distance p between the camera 502 and the LED panel 506. For example, the above-mentioned shortest distance pmin and longest distance pmax are acquired.

[0257] The steps for measuring the distance along the optical axis of camera 502 using auxiliary camera 502S, whose optical axis is parallel to camera 502 and installed a distance dc away from it, as shown in Figure 33 above, are shown below as (i) to (v). In FIG. 33, when α=0, the length of the line segment AP becomes the distance along the optical axis of the camera 502.

[0258] (i) The pixel values ​​of several pixels near the center of the optical axis of the camera 502 are acquired.

[0259] (ii) The pixel values ​​of all pixels of the auxiliary camera 502S are obtained. The focal length of the lens 21 of the camera 502 is "fmain", the in-focus distance is "lmain", and the pixel size of the image sensor 20 is "Wsmain". The focal length of the lens 21 of the auxiliary camera 502S is "fsub", the in-focus distance is "lsub", and the pixel size of the image sensor 20 is "Wssub".

[0260] From the above (Equation 3), the image of the camera 502 has actual dimensions fmain / (lmain-fmain) times larger, and is sampled at intervals of pixel size Wsmain. The image of the auxiliary camera 502S has actual dimensions fsub / (lsub-fsub) times larger than the actual dimensions according to (Equation 3), and is sampled at intervals of the pixel size Wssub. Therefore, the image acquired from the auxiliary camera 502S must be enlarged or reduced by the following magnification:

[0261]

number

[0262] Conversely, the partial image acquired in (i) above may be enlarged or reduced to match the auxiliary camera 502S. In that case, the reciprocal of the above magnification is used, and the focal length, focusing distance, and pixel dimensions used in the calculations from (iii) onwards are those of the auxiliary camera 502S.

[0263] (iii) A cross-correlation function is calculated between the pixel values ​​of auxiliary camera 502S and the pixel values ​​near the center of the optical axis obtained from camera 502 from the upper left to the lower right of the screen, and the coordinates where the correlation function is maximum are searched for.

[0264] (iv) The distance between the pixel position at the center of the optical axis of the auxiliary camera 502S and the coordinates at which the cross-correlation function is maximized is calculated, and the distance ds is calculated by multiplying it by the pixel size Wsmain of the image sensor 20.

[0265] (v) In the above (Equation 34) and (Equation 35), α and β are substituted as follows:

[0266]

number

[0267] The above α and β are substituted into the equation for calculating the line segment AP in (Equation 35), and the distance pa to the LED panel 506 along the optical axis of the camera 502 is calculated by the following equation.

[0268]

number

[0269] Similarly, in the steps (i) to (v), a distance pb to a point on LED panel 506 that is visible at as large an angle θ′ as possible from the optical axis of camera 502 is obtained within the range visible from both camera 502 and auxiliary camera 502S. The distance pb is determined by the angle Φ between the optical axis of the LED panel 506 and the optical axis of the camera 502 as follows:

[0270]

number

[0271] Therefore, the angle Φ can be calculated using the following formula:

[0272]

number

[0273] If the range visible from camera 502 is within an angle range of ±θ with respect to the optical axis, the shortest distance pin and the longest distance pmax between camera 502 and LED panel 506 can be calculated by the following formula. Although angle Φ is 90° or less in Fig. 31, the maximum and minimum values ​​can be calculated by the same formula even if it exceeds 90°. However, taking into consideration the possibility of measuring the angle in the wrong direction, the calculation is performed as follows.

[0274]

number

[0275] For example, in the above process, the CPU 71 acquires the shortest distance pmin and the longest distance pmax between the camera 502 and the LED panel 506 in step S504 in FIG. However, the minimum distance pmin and the maximum distance pmax may be detected by a distance measuring sensor, or may be obtained based on information from the camera tracker 560. Furthermore, they may be input by an operator.

[0276] In step S505, the CPU 71 substitutes the shortest distance pmin for the distance p, which is a variable used in the calculation, and also substitutes "0" for the maximum amplitude value Amax.

[0277] In steps S506 to S510, the CPU 71 performs processing for calculating the amplitude characteristics while gradually increasing the value of the distance p until the distance p exceeds the maximum distance pmax.

[0278] First, in step S506, the CPU 71 calculates the amplitude characteristic (amplitude A). For this reason, the CPU 71 calculates the angular frequency ω using the above-mentioned (Equation 18).

[0279] Further, the CPU 71 calculates the amplitude characteristic (amplitude Af) due to the optical LPF 22 and the amplitude characteristic (amplitude Ap) due to the integration effect in the pixel using the above-mentioned formulas (19) and (20), respectively.

[0280] Moreover, the CPU 71 calculates the diameter δ of the blur caused by defocusing using the above-mentioned formula (14).

[0281] Then, the CPU 71 calculates the amplitude Ab due to blur according to the value of (ωδ) using the above-mentioned (Equation 25).

[0282] The amplitude characteristic (amplitude A) is the product of these (A = Af Ap Ab).

[0283] Initially, since the distance p=the shortest distance pmin, the amplitude A at the shortest distance pmin is found. If the calculated amplitude A is greater than the maximum amplitude Amax at that time in step S507, the CPU 71 substitutes the currently calculated amplitude A for the maximum amplitude Amax in step S508.

[0284] In step S509, the CPU 71 gradually increases the distance p. In step S510, the CPU 71 determines whether the distance p has exceeded the maximum distance pmax, and if not, returns to step S506, and calculates the amplitude A with the gradually increased distance p. If it is determined in step S510 that the distance p exceeds the maximum distance pmax, the process proceeds from step S510 to step S511.

[0285] Therefore, in steps S506 to S510, the amplitude A is calculated for each distance of a certain interval in the range from the shortest distance pmin to the longest distance pmax, and the maximum value among them is set as the maximum amplitude Amax.

[0286] In step S511, the CPU 71 compares the maximum amplitude value Amax with a threshold value Ath. If the maximum amplitude value Amax does not exceed the threshold value Ath, the CPU 71 determines that no moire or the like occurs and ends the determination process.

[0287] If the maximum amplitude value Amax exceeds the threshold value Ath, the CPU 71 determines that moire or the like occurs, and performs a moire occurrence determination process in step S512. For example, information such as data for determining whether moiré occurs, a flag, etc., is set. This information may be stored in a storage medium in association with video data such as the captured video vC and the simulation video vSM, camera setting information CS, etc., or may be transmitted to, for example, a cloud server 41, etc. Also, it may be added as metadata to the video data as the captured video vC.

[0288] For example, by performing the process of FIG. 35 in step S402 in FIG. 14 or step S71 in FIG. 15, moire alert control may be performed thereafter in step S404 in FIG. 14 or step S73 in FIG.

[0289] The threshold value Ath used in step S511 in Fig. 35 may be determined after on-site adjustment. Contrast may be strengthened or weakened during editing, and may also change depending on the environment in which the finished work is viewed. The threshold value Ath should be determined so that moire and other issues are not noticeable in the final form.

[0290] Incidentally, the LED wall 505 may appear to be installed on a curved surface, but in reality, such an LED wall 505 is installed by combining small flat LED panels 506 as shown in FIG. Therefore, by performing the above calculation for each LED panel 506, it is possible to predict the occurrence of artifacts such as moire.

[0291] Although moire only interferes with viewing purposes, as shown in Figure 37, the moire makes the slight misalignment of the LEDs at the joints of the LED panels 506 look magnified. By utilizing this, it is very useful for adjusting the positions of the joints of the LED panels 506 when setting up the LED wall 505.

[0292] That is, when setting up, an image of the LED wall 505 is captured by a camera 502 as shown in the figure, and observed, for example, on an output monitor 503. Then, the misalignment of the LED panel 506 becomes clearly visible due to moire. This is because, if there is a subtle difference between the spacing of the LED images formed on the image sensor 20 and the spacing of the pixels of the image sensor 20, a minute misalignment appears magnified, similar to the relationship between the main and secondary scales of a caliper. When considering such a use, it is preferable to carry out the above-mentioned determination process in order to find the conditions for deliberately causing moire to occur.

[0293] FIG. 38 shows another example of the determination process. The process in Fig. 35 above determines the maximum amplitude Amax between the shortest distance pmin and the longest distance pmax, and compares this with the threshold Ath, so it is simply a determination of whether or not moire occurs. In contrast, Fig. 38 shows an example of determining in which distance range, within the range from the shortest distance pmin to the longest distance pmax, moire occurs. This also determines in which range within the plane of the captured image vC moire etc. occurs.

[0294] In FIG. 38, steps S501 to S504 are similar to those in FIG. In step S520 of FIG. 38, the CPU 71 substitutes the shortest distance pmin for the distance p, which is a variable for calculation. Furthermore, the CPU 71 assigns "0" to a flag FM which indicates the occurrence of moire. The flag FM=0 indicates that moire or the like does not occur, and the flag FM=1 indicates that moire or the like occurs. The CPU 71 also assigns the maximum distance pmax to the variable PMmin and assigns the minimum distance pmin to the variable PMmax. The variable PMmin is a variable indicating the minimum distance of the range where moire occurs, and the initial value of this variable is set to the maximum distance pmax. The variable PMmax is a variable indicating the maximum distance of the range where moire occurs, and the initial value of this variable is set to the minimum distance pmin.

[0295] After setting the initial values ​​as described above, in steps S521 to S529, the CPU 71 performs processing for calculating the amplitude characteristics while gradually increasing the value of the distance p until the distance p exceeds the maximum distance pmax.

[0296] First, in step S521, the CPU 71 calculates the amplitude characteristic (amplitude A) at the current distance p. The calculation of the amplitude A is similar to that in step S506 in FIG.

[0297] In step S522, the CPU 71 compares the calculated amplitude A with a threshold value Ath. This threshold value Ath is the threshold value Ath used in step S511 in Fig. 35, and is a threshold value that is used to determine that moire or the like will occur if it is exceeded.

[0298] If the amplitude A does not exceed the threshold Ath, the CPU 71 gradually increases the distance p in step S528, and then determines in step S529 whether the distance p has exceeded the maximum distance pmax, and if not, returns to step S521.

[0299] If it is determined in step S522 that the amplitude A exceeds the threshold value Ath, the CPU 71 proceeds to step S523 and sets the flag FM=1.

[0300] In step S524, the CPU 71 determines whether the distance p is equal to or smaller than the variable PMmin. If p≦PMmin, the CPU 71 assigns the distance p to the variable PMmin in step S525. Since the initial value of the variable PMmin is the maximum distance pmax and the distance p gradually increases from the minimum distance pmin, the distance p when A>Ath is first set is assigned to the variable PMmin. Then, the process proceeds to step S526. If p≦PMmin is not satisfied, the CPU 71 skips step S525 and proceeds to step S526.

[0301] In step S526, the CPU 71 determines whether or not the distance p is equal to or greater than the variable PMmax. If p≧PMmax, the CPU 71 assigns the distance p to the variable PMmax in step S527. Since the initial value of the variable PMmax is the shortest distance pmin, and the distance p is gradually increased from the shortest distance pmin, after A>Ath is first set, the variable PMmax is updated to the distance p at each point in time while the distance p is gradually increased and the state of A>Ath continues. Then, the process proceeds to step S528. If p≧PMmax is not satisfied, the CPU 71 skips step S527 and proceeds to step S528.

[0302] After repeating the above process until P>Pmax, the CPU 71 advances from step S529 to step S530. If the flag FM=0 at this point, it is determined that no moire or the like occurs.

[0303] On the other hand, if the flag FM=1, it is determined that moire or the like occurs. In this case, the CPU 71 proceeds to step S531 to perform a moire occurrence determination process. For example, information such as data and flags indicating the moire occurrence determination and occurrence range are set. The range in which moire occurs is the range from the distance indicated by the variable PMmin to the distance indicated by the variable PMmax. In other words, moire and the like will occur in the area within the image plane that corresponds to this distance range.

[0304] The information on the moiré occurrence determination and the occurrence range may be transmitted to, for example, the cloud server 41 in association with the video data, the camera setting information CS, etc. Also, the information may be added as metadata to the video data such as the captured video vC and the simulation video vSM.

[0305] <5. Summary and variations> According to the above embodiment, the following effects can be obtained.

[0306] The information processing device 70 functioning as the rendering engine 31, 520 in the embodiment includes a judgment unit (36, 520d) that performs a judgment process of whether or not artifacts occur in the captured image vC using the spacing Wp of the LEDs, which are the light-emitting elements of the LED panel 506, the pixel spacing Ws of the image sensor 20 of the camera 502, the characteristics (shift amount b) of the optical LPF 22, the focal length f, F-number F, and focusing distance l of the lens 21, and the distance p between the LED panel 506 and the camera 502. This makes it possible to predict whether artifacts such as moire will occur based on obtainable numerical values ​​at the shooting site or during the pre-shooting review stage. By being able to predict artifacts at the pre-shooting review stage or the preparation stage at the shooting site, work during shooting can be made more efficient. As for artifacts such as moire, moire, grid, color shift, etc. can be judged. Also, the LED panel 506 element for this determination is the LED spacing, and is not related to the image content. Therefore, even if no image is displayed on the LED wall 505, moire determination is possible according to, for example, the camera position.

[0307] In the embodiment, the judgment unit (36, 520d) determines whether or not an artifact has occurred by determining the amplitude A of the image of the light-emitting element on the image sensor 20 and comparing the determined amplitude A with a threshold value Ath (see Figures 35 and 38). For example, the number of overlapping blurred images of the subject causes uneven brightness, which can lead to artifacts such as moire. Therefore, the amplitude of the image on the image sensor 20 of the actual LEDs of the LED panel 506 is calculated. If this amplitude is large, it can be determined that moire or the like will occur.

[0308] In the embodiment, an example was given in which the judgment unit (36, 520d) acquires the shortest distance pmin and the longest distance pmax between the LED panel 506 and the camera 502, and performs judgment processing using the shortest distance pmin and the longest distance pmax (see Figures 31, 35, and 38). The LED panel 506 and the camera 502 are not necessarily positioned directly opposite each other, but rather are often positioned diagonally opposite each other. In such cases, there will be a minimum distance and a maximum distance between the camera 502 and the display surface of the LED wall 505 (LED panel 506). By performing moiré determination within this range, the occurrence of moiré, etc. can be appropriately determined regardless of the position or direction of the camera 502.

[0309] In the embodiment, an example has been given in which the determining unit (36, 520d) determines the range in which artifacts occur in the captured video vC (see FIG. 38). That is, it is determined where moire occurs within the range of the plane of the captured image vC. Such determination results can also be used for various setting changes and shooting preparations. According to the process shown in FIG. 38, it is possible to determine in what range between the shortest distance pmin and the longest distance pmax the moire will occur.

[0310] In the embodiment, an example has been given in which the determining unit (36, 520d) performs a warning process in response to determining that an artifact has occurred (see FIGS. 14 and 15). For example, by warning staff watching at the previsualization stage, it becomes possible to take measures to prevent moire and the like from occurring in advance. The same can be achieved by warning staff at the filming preparation stage. Furthermore, if it is determined that moire has occurred during shooting, a warning is displayed to the cameraman 512, etc., so that the cameraman 512 can immediately take measures such as blurring the focus.

[0311] In the embodiment, an example has been given in which the judgment unit (36, 520d) performs processing to associate the judgment result information of the judgment processing with a project. In this case, the project is a video content production project. In other words, it is a video production project in which video is produced by shooting in the shooting system 500. Note that associating the judgment result information with a project includes making it information corresponding to the project, making it information associated with the video of the project, and the like. Associating with a video includes associating the judgment result information as management information, etc. corresponding to the entire video content file, and associating the judgment result information as information corresponding to a frame of the video. The process of associating the data includes a process of storing the data in an associated state on a storage medium, and a process of transmitting the data to an external device. For example, data, flags, etc. indicating the determination result of moiré occurrence are stored in a storage medium in association with the video. Alternatively, data, flags, etc. indicating the determination result of moiré occurrence are transmitted to and stored in the cloud server 41 in accordance with the camera settings. This allows the moiré determination result at the pre-visualization stage to be referred to when preparing for shooting, for example.

[0312] In the embodiment, an example has been described in which the determination unit (36, 520d) acquires all or part of the values ​​of the LED spacing Wp, the pixel spacing Ws of the image sensor 20, the shift amount b of the optical LPF 22, the focal length f, the F-number F, the focusing distance l, and the distance p based on information received through communication with other devices. For example, the determination unit 36, 520d acquires values ​​necessary for these moire determinations through communication with the LED panel 506, the camera 502, or an information processing device in the system. Alternatively, the determination unit 36, 520d may acquire the model numbers of the LED panel 506 or the camera 502 through communication, and obtain each value according to the model numbers. With these, the determination unit 36, 520d can automatically obtain the necessary numerical values ​​to perform the moire determination.

[0313] In the embodiment, the judgment unit (36, 520d) compares the maximum amplitude Amax of the LED image on the image sensor 20 with a threshold Ath for multiple distance values ​​from the shortest distance pmin to the longest distance pmax, and judges that an artifact has occurred if the maximum amplitude Amax exceeds the threshold Ath (see FIG. 35). This makes it possible to determine the presence or absence of artifacts in actual distance relationships through relatively simple processing.

[0314] The imaging system 500 according to the embodiment is a system that uses a camera 502 to capture an image of an LED panel 506 that displays a virtual image (background image vB) obtained by rendering using a 3D model. The rendering engine 31, which is the information processing device 70 according to the embodiment, includes a rendering unit 32 and an image processing unit 33 in addition to the determination unit 36. The rendering unit 32 performs rendering using a 3D model used in the imaging system 500 to generate a virtual image. The image processing unit 33 generates a simulation image vSM for the virtual image generated by the rendering unit 32, using processing parameters that realize the luminance or color characteristics of the camera 502 used in the imaging system 500 at the time of imaging. That is, this is a configuration in which the present technology is applied to the rendering engine 31 in the previsualization system 700, and the rendering engine 31 is configured to include a determination unit 36 ​​in addition to a rendering unit 32 and an image processing unit 33. This makes it possible to determine, at the previsualization stage, whether or not moiré or the like will occur in an image under the same conditions as in the actual shooting.

[0315] Moreover, the rendering engine 520, which is the information processing device 70 according to the embodiment, includes a rendering unit 520a. That is, this is a configuration in which the present technology is applied to the rendering engine 520 in the imaging system 500, and the rendering engine 520 is provided with a determination unit 520d. This makes it possible to determine whether or not moire or the like occurs at the stage of preparation for imaging or imaging. In the imaging system 500, a determination unit such as the determination unit 520d that determines moire or the like may be provided in an information processing device other than the rendering engine 520. For example, the determination unit may be provided in the camera 502 in FIG. 7, in the display controller 590, in a control unit on the operation monitor 550 side, or in various control and arithmetic units not shown.

[0316] The program of the embodiment is a program that causes a processor such as a CPU or a DSP, or a device including these, to execute the processes shown in the above-mentioned FIGS. 14, 15, 35, and 38. That is, the program of the embodiment is a program that causes the information processing device 70 to execute a process of determining whether or not artifacts occur in the captured image vC, using the spacing Wp of the LEDs, which are light-emitting elements, the pixel spacing Ws of the image sensor 20 of the camera 502, the shift amount b of the optical LPF 22, the focal length f of the lens 21, the F-number F, the focusing distance l, and the distance p between the LED panel 506 and the camera 502.

[0317] By using such a program, the information processing device 70 that performs the determination process of the embodiment can be realized by various types of computer devices.

[0318] Such a program can be pre-recorded in a HDD as a recording medium built into a device such as a computer device, or in a ROM in a microcomputer having a CPU. Also, such a program can be temporarily or permanently stored (recorded) in a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such a removable recording medium can be provided as a so-called package software. Furthermore, such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0319] Furthermore, such a program is suitable for providing a wide range of information processing devices 70 that perform determination processing, such as the information processing devices 70 that are the rendering engines 31 and 520 of the embodiments. For example, by downloading the program to personal computers, communication devices, mobile terminal devices such as smartphones and tablets, mobile phones, game devices, video devices, PDAs (Personal Digital Assistants), and the like, these devices can function as the information processing devices 70 of the present disclosure.

[0320] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0321] The present technology can also be configured as follows. (1) The imaging system includes a camera that captures an image of a display, and a determination unit that performs a process of determining whether or not artifacts occur in the image captured by the camera using the interval between light-emitting elements of the display, the pixel interval of the image sensor of the camera, the characteristics of the optical low-pass filter of the camera, the focal length, F-number, and focusing distance of the lens of the camera, and the distance between the display and the camera. Information processing device. (2) In the determination process, the determination unit determines whether an artifact occurs by obtaining an amplitude of an image of the light-emitting element on an image sensor and comparing the obtained amplitude with a threshold value. The information processing device according to (1) above. (3) The determination unit obtains the shortest distance and the longest distance between the display and the camera, and performs the determination process using the shortest distance and the longest distance. The information processing device according to (1) or (2) above. (4) In the determination process, the determination unit determines a range in which artifacts occur in the captured video. An information processing device according to any one of (1) to (3) above. (5) The determination unit performs a warning process in response to determining that an artifact has occurred. An information processing device according to any one of (1) to (4) above. (6) The judgment unit performs a process of associating judgment result information of the judgment process with a project. An information processing device according to any one of (1) to (5) above. (7) The judgment unit performs a process of storing the judgment result information in a storage medium in association with a project. An information processing device according to any one of (1) to (6) above. (8) The judgment unit performs a process of associating the judgment result information with a project and transmitting the information to an external device. An information processing device according to any one of (1) to (7) above. (9) The determination unit acquires all or a part of values ​​such as the interval between light-emitting elements of the display, the pixel interval of the image sensor of the camera, the characteristics of the optical low-pass filter of the camera, the focal length of the lens of the camera, the F-number, the focusing distance, and the distance between the display and the camera, based on information received through communication with another device. An information processing device according to any one of (1) to (8) above. (10) The determination unit is Obtaining the minimum and maximum distances between the display and the camera; For a plurality of distance values ​​from the shortest distance to the longest distance, the maximum value of the amplitude of the image of the light emitting element on the image sensor is compared with a threshold value, and if the maximum value exceeds the threshold value, it is determined that an artifact has occurred. An information processing device according to any one of (1) to (9) above. (11) The imaging system is a system for imaging an image of a display that displays a virtual image obtained by rendering using a 3D model, using a camera; A rendering unit that performs rendering using the 3D model to generate a virtual image; and an image processing unit that generates a simulation image using processing parameters that realize the luminance or color characteristics at the time of shooting of the camera used in the shooting system for the virtual image generated by the rendering unit. An information processing device according to any one of (1) to (10) above. (12) The imaging system is a system for imaging an image of a display that displays a virtual image obtained by rendering using a 3D model, using a camera; a rendering unit that performs rendering using the 3D model to generate a virtual image to be displayed on the display; An information processing device according to any one of (1) to (10) above. (13) An information processing device, In an imaging system in which an image of a display is captured by a camera, a process of determining whether or not artifacts occur in the image captured by the camera is performed using the interval between light-emitting elements of the display, the pixel interval of the image sensor of the camera, the characteristics of the optical low-pass filter of the camera, the focal length, F-number, and focusing distance of the lens of the camera, and the distance between the display and the camera. Information processing methods. (14) a process for determining whether or not artifacts occur in an image captured by a camera using an interval between light-emitting elements of a display in an imaging system that captures an image of the display with a camera, an interval between pixels of an image sensor of the camera, characteristics of an optical low-pass filter of the camera, a focal length, an F-number, and a focusing distance of a lens of the camera, and a distance between the display and the camera; A program to be executed by an information processing device. [Explanation of symbols]

[0322] 20 Image Sensor 21 Lens 22 Optical LPF 31 Rendering Engine 32 Rendering Department 33 Video Processing Section 35 Adjustment operation section 36 Judgment section 37 Position operation section 38 Communications Department 70 Information processing device, 71 CPU 500 Imaging System 502 Camera 502S Auxiliary Camera 505 LED Wall 506 LED Panel 520 Rendering Engine 520d Judging section

Claims

1. A photographing system for photographing an image of a display with a camera includes a determination unit that performs a process of determining whether or not moire occurs in an image photographed by the camera, using an interval between light-emitting elements of the display, an interval between pixels of an image sensor of the camera, characteristics of an optical low-pass filter of the camera, a focal length, an F-number, and a focusing distance of a lens of the camera, and a distance between the display and the camera. Information processing device.

2. In the determination process, the determination unit determines the amplitude of the image of the light-emitting element on the image sensor and compares the determined amplitude with a threshold value to determine whether or not moire occurs. The information processing device according to claim 1 .

3. The determination unit obtains the shortest distance and the longest distance between the display and the camera, and performs the determination process using the shortest distance and the longest distance. The information processing device according to claim 1 .

4. In the determination process, the determination unit determines a range in which moire occurs in the captured image. The information processing device according to claim 1 .

5. The determination unit performs a warning process in response to determining that moire has occurred. The information processing device according to claim 1 .

6. The judgment unit performs a process of associating judgment result information of the judgment process with a project. The information processing device according to claim 1 .

7. The judgment unit performs a process of storing the judgment result information in a storage medium in association with a project. The information processing device according to claim 6.

8. The judgment unit performs a process of associating the judgment result information with a project and transmitting the information to an external device. The information processing device according to claim 6.

9. The determination unit acquires all or a part of values ​​such as the interval between light-emitting elements of the display, the pixel interval of the image sensor of the camera, the characteristics of the optical low-pass filter of the camera, the focal length of the lens of the camera, the F-number, the focusing distance, and the distance between the display and the camera, based on information received through communication with another device. The information processing device according to claim 1 .

10. The determination unit is Obtaining the minimum and maximum distances between the display and the camera; At a plurality of distance values ​​from the shortest distance to the longest distance, the maximum value of the amplitude of the image of the light emitting element on the image sensor is compared with a threshold value, and if the maximum value exceeds the threshold value, it is determined that moire has occurred. The information processing device according to claim 1 .

11. The imaging system is a system for imaging an image of a display that displays a virtual image obtained by rendering using a 3D model, using a camera; A rendering unit that performs rendering using the 3D model to generate a virtual image; and an image processing unit that generates a simulation image using processing parameters that realize the luminance or color characteristics at the time of shooting of the camera used in the shooting system for the virtual image generated by the rendering unit. The information processing device according to claim 1 .

12. The imaging system is a system for imaging an image of a display that displays a virtual image obtained by rendering using a 3D model, using a camera; a rendering unit that performs rendering using the 3D model to generate a virtual image to be displayed on the display; The information processing device according to claim 1 .

13. An information processing device, In an imaging system in which an image of a display is captured by a camera, a determination process is performed as to whether or not moire occurs in the captured image of the camera using the interval between light-emitting elements of the display, the interval between pixels of an image sensor of the camera, the characteristics of an optical low-pass filter of the camera, the focal length, F-number, and focusing distance of the lens of the camera, and the distance between the display and the camera. Information processing methods.

14. A process for determining whether or not moire occurs in an image captured by a camera using an interval between light-emitting elements of the display in an image capturing system in which the image of the display is captured by the camera, an interval between pixels of an image sensor of the camera, characteristics of an optical low-pass filter of the camera, a focal length, an F-number, and a focusing distance of a lens of the camera, and a distance between the display and the camera A program to be executed by an information processing device.

Images