Information processing apparatus, information processing method, and program

Abstract

An information processing apparatus includes a determination unit that performs a determination process, in an imaging system that captures a video of a display using a camera, a determination process is performed by using an interval between light-emitting elements of the display, a pixel interval between image sensors of the camera, characteristics of an optical low-pass filter of the camera, a focal length, an F-value, and a focus distance of a lens of the camera, and a distance between the display and the camera. And determining whether artifacts appear in the shot video of the camera.

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method, and a program, and to a technology that can be used in video production using virtual video, for example. Background Art

[0002] As an imaging method for producing video contents such as movies, there is known a technique in which performers perform actions with a so-called green screen and then a background video is composited.

[0003] Furthermore, in recent years, instead of green screen imaging, an imaging system has been developed in which a background video is displayed on a display device and performers perform actions in front of the background video in a studio equipped with a large display device, thereby enabling imaging of the performers and the background, and this imaging system is referred to as so-called virtual production, in-camera VFX, or LED wall virtual production.

[0004] Patent Document 1 listed below discloses a system technology for causing a performer to act in front of a background video.

[0005] In addition, Patent Document 2 listed below discloses a technique for disposing an optical member in the form of a film or the like in order to prevent moiré fringes generated when imaging a large display device.

[0006] Citation List

[0007] Patent Literature

[0008] Patent Document 1: U.S. Patent Application Publication No. 2020 / 0145644A

[0009] Patent Document 2: JP 2014-202816A Summary of the invention

[0010] Problems to be solved by the present invention

[0011] When a background video is displayed on a large monitor and then the performers and the background video are shot with a camera, there is no need to synthesize the background video separately after shooting, and the performers and staff can visually understand the scene and act, or determine whether the performance is good or bad, etc., which is more advantageous than green screen imaging.

[0012] However, to capture video of a display, artifacts such as moiré occur due to interference between the pixels of the camera's image sensor and the pixels of the display panel. To avoid this, the camera needs to blur the display instead of focusing on it.

[0013] However, since the conditions for the appearance of moire fringes and the like are unknown, it is actually necessary to change the position of the camera and the position of the performer at the imaging location, or adjust the lens by trial and error. This reduces the efficiency of video production.

[0014] Therefore, the present disclosure proposes a technique capable of more accurately determining the occurrence of artifacts such as moire fringes.

[0015] Solutions to the Problem

[0016] An information processing device according to the present technology includes a determination unit that performs determination processing for determining whether artifacts appear in a video captured by a camera by using the intervals between light-emitting elements of the display, the pixel intervals between image sensors of the camera, the characteristics of an optical low-pass filter of the camera, the focal length, F value and focusing distance of a lens of the camera, and the distance between the display and the camera in an imaging system that uses a camera to capture a video of a display.

[0017] That is, the information processing device determines whether an artifact such as moire fringes appears in a captured video when a video of the display is captured by a camera before or during the capturing, by using the value of the interval between the above-mentioned light emitting elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is an explanatory diagram of an imaging system according to an embodiment of the present technology.

[0019] Figure 2 is an explanatory diagram of a background video of the camera position of the imaging system according to the present embodiment.

[0020] Figure 3 is an explanatory diagram of a background video of the camera position of the imaging system according to the present embodiment.

[0021] Figure 4 It is an explanatory diagram of the video content production steps of this embodiment.

[0022] Figure 5 is a block diagram of the imaging system of this embodiment.

[0023] Figure 6 4 is a flowchart of background video generation of the imaging system of this embodiment.

[0024] Figure 7 is a block diagram of an imaging system using a plurality of cameras of this embodiment.

[0025] Figure 8 is a block diagram of the information processing device of this embodiment.

[0026] Fig. 9 It is an explanatory diagram of a preliminary visualization of an embodiment.

[0027] Fig.10 is a block diagram of a configuration example including a determination unit according to an embodiment.

[0028] Fig.11 is an explanatory diagram of a camera position operation video according to an embodiment.

[0029] Fig.12 It is an explanatory diagram of a moire warning according to an embodiment.

[0030] Fig.13 1 is an explanatory diagram of operations in a camera position operation video according to an embodiment.

[0031] Fig.14 This is a flowchart of the moire fringe warning process of this embodiment.

[0032] Fig.15 is a flowchart of the moire warning process at the time of imaging according to this embodiment.

[0033] Fig.16 This is a diagram illustrating an image on an image sensor.

[0034] Fig.17 This is an explanatory diagram of an optical low-pass filter (LPF).

[0035] Fig.18 This is a diagram for explaining the characteristics of the optical LPF.

[0036] Fig.19 This is a diagram to explain the width of pixels.

[0037] Fig. 20 This diagram explains image blur on an image sensor.

[0038] Fig.21 This is an approximate illustration of a regular dodecagon.

[0039] Fig. 22 This is an explanatory diagram of an image of a light emitting diode (LED).

[0040] Fig.23 This is an explanatory diagram of a state where an LED image is divided into four by an optical LPF.

[0041] Fig.24 This is an explanatory diagram showing a state in which the LED image is further blurred by a diameter δ.

[0042] Fig.25 The diagram is an explanatory diagram of pixels of an image sensor.

[0043] Fig.26 An illustration of the brightness perceived by each pixel of an image sensor.

[0044] Fig. 27 This is a diagram for explaining the amplitude characteristics.

[0045] Fig.28 This is a diagram illustrating the amplitude characteristics using envelope approximation.

[0046] Fig.29 This is an explanatory diagram of a state in which a camera images a board surface from an oblique direction.

[0047] Fig.30 This is an illustration of the range that appears on the camera.

[0048] Fig.31 This is a diagram to explain the shortest and longest distances in the range that appears on the camera.

[0049] Fig.32 This is a diagram illustrating the distance on the image sensor.

[0050] Fig.33 This is an illustration of distance calculation using a sub-camera.

[0051] Fig.34 This is a diagram illustrating distance calculation using a sub-camera.

[0052] Fig.35 : is a flowchart of the moire fringe determination process of this embodiment.

[0053] Fig.36 It is an explanatory diagram of the LED wall 505.

[0054] Fig.37 This is an explanatory diagram of an arrangement for highlighting moiré fringes.

[0055] Fig.38 is a flowchart of another example of the moire fringe determination process according to this embodiment. DETAILED DESCRIPTION

[0056] Hereinafter, the embodiments will be described in the following order.

[0057] <1. Imaging system and content production>

[0058] <2. Configuration of information processing equipment>

[0059] <3. Configuration including artifact determination unit>

[0060] <4. Confirm the processing>

[0061] <5. Overview and Modification Examples>

[0062] Note that in the present disclosure, "video" or "image" includes both still images and moving images. In addition, "video" refers not only to the state in which video data is displayed on the display, but also video data in the state in which video data is not displayed on the display can be collectively referred to as "video".

[0063] For example, in an embodiment, the background video before being displayed on the display, the video shot by the camera, etc. are not the actual video being displayed, but video data. For the sake of convenience, such video data are referred to as "background video", "shot video", etc.

[0064] <1. Imaging system and video content production>

[0065] An imaging system and production of video content to which the techniques of this disclosure may be applied will be described.

[0066] Figure 1 An imaging system 500 is schematically shown. The imaging system 500 is a system for performing imaging for virtual production, and the figure shows a part of the equipment arranged in an imaging studio.

[0067] In the imaging room, a performance area 501 is set where a performer 510 performs a performance such as an action. A large display device is set on at least the back, left and right sides, and the upper surface of the performance area 501. Although the device type of the display device is not limited, the accompanying drawings show an example in which an LED wall 505 is used as an example of the large display device.

[0068] One LED wall 505 forms a large panel by vertically and horizontally connecting and arranging a plurality of LED panels 506. The size of the LED wall 505 is not particularly limited, but only needs to be a size necessary or sufficient as a size for displaying a background when imaging the performer 510.

[0069] A necessary number of lamps 580 are arranged at necessary positions such as above or to the side of the performance area 501 so as to illuminate the performance area 501 .

[0070] Near the performance area 501, for example, a camera 502 for shooting video content such as a movie is arranged. The camera operator 512 can move the position of the camera 502, and can operate the imaging direction, the viewing angle, etc. Of course, it is also conceivable to perform the movement of the camera 502, the viewing angle operation, etc. by remote control. In addition, the camera 502 can automatically or autonomously move or change the viewing angle. For this purpose, the camera 502 can be installed on a camera platform or a mobile body.

[0071] Camera 502 jointly images performer 510 in performance area 501 and video displayed on LED wall 505. For example, by displaying a scene as background video vB on LED wall 505, a video similar to the situation where performer 510 actually exists at the place of the scene and performs can be captured.

[0072] The output monitor 503 is arranged near the performance area 501. The video shot by the camera 502 is displayed in real time as a monitor video vM on the output monitor 503. Therefore, the director and the staff who produce the video content can confirm the shot video.

[0073] As described above, imaging system 500 for imaging a performer's 510 performance against the backdrop of an LED wall 505 in an imaging studio has various advantages over green screen imaging.

[0074] For example, in the case of green screen imaging, it is difficult for the performer to imagine the background and situation of the scene, which may affect the action. By displaying the background video vB, the performer 510 can easily perform the action, and the quality of the action is improved. In addition, the director and other staff can easily determine whether the action of the performer 510 matches the background or the situation of the scene.

[0075] In addition, post-production after imaging is more efficient than in the case of green screen imaging. This is because so-called chroma key compositing may not be necessary, or color correction or reflection compositing may not be necessary. In addition, even in the case where chroma key compositing is required at the time of imaging, only green or blue video needs to be displayed, so since there is no need to add a physical background screen, improved efficiency is also helpful.

[0076] In the case of green screen imaging, a green tint is added to the performer's body, clothes, and objects, so it is necessary to correct it. In addition, in the case of green screen imaging, in the presence of an object in which the surrounding scene is reflected, such as glass, a mirror, or a snowball, it is necessary to generate and synthesize the reflected image, but this is troublesome work.

[0077] On the other hand, in Figure 1 In the case of imaging by the imaging system 500 in , the green hue does not increase, so correction is unnecessary. In addition, by displaying the background video vB, the reflection on the actual object such as glass is naturally obtained and photographed, so it is also unnecessary to synthesize the reflection video.

[0078] Here, we will refer to Figure 2 and 3Describing the background video vB, even if the background video vB is displayed on the LED wall 505 and filmed with the performer 510, the background of the filmed video becomes unnatural by simply displaying the background video vB. This is because a three-dimensional background having depth is actually used as the background video vB in a flat manner.

[0079] For example, camera 502 can shoot performer 510 in performance area 501 from various directions, and can also perform zoom operation. Performer 510 does not stay in one place. Then, the actual appearance of the background of performer 510 should change with the position, imaging direction, viewing angle, etc. of camera 502, but such change cannot be obtained in background video vB which is a plane video. Therefore, background video vB is changed so that the background is similar to the actual appearance including parallax.

[0080] Figure 2 The figure shows a state where the camera 502 is imaging the performer 510 from the position on the left side of the figure. Figure 3 It shows the state that the camera 502 images the performer 510 from the position on the right side of the figure. In each figure, the imaging area video vBC is shown in the background video vB.

[0081] Note that the portion of the background video vB that does not include the imaging area video vBC is referred to as an "outer frustum", and the imaging area video vBC is referred to as an "inner frustum".

[0082] The background video vB described here means the entire video displayed as the background including the imaging area video vBC (inner frustum).

[0083] The range of the imaging area video vBC (inner frustum) corresponds to the range actually imaged by the camera 502 in the display surface of the LED wall 505. Then, the imaging area video vBC is a video representing the scene actually viewed when the position of the camera 502 is set as the viewpoint, based on the position of the camera 502, the imaging direction, the viewing angle, and the like.

[0084] Specifically, 3D background data of a three-dimensional (3D) model as a background is prepared, and the imaging area video vBC is sequentially rendered in real time based on the viewpoint position of the camera 502 relative to the 3D background data.

[0085] Note that the range of the imaging area video vBC is actually a slightly wider range than the range imaged by the camera 502 at that point in time. This is to prevent the video of the outer frustum from being reflected due to rendering delay, and to avoid the influence of diffracted light from the video of the outer frustum when the imaging range is slightly changed by panning, tilting, zooming, etc. of the camera 502.

[0086] The video of the imaging area video vBC presented in real time in this way is combined with the video of the outer frustum. The video of the outer frustum used in the background video vB can be rendered in advance based on the 3D background data, or it can be rendered in real time at each frame or each intermittent frame, merging the video of the imaging area video vBC (inner frustum) into a part of the video of the outer frustum to generate the entire background video vB.

[0087] Note that there is a case where the video of the outer frustum is also rendered for each frame similar to the inner frustum, but here, a static video is taken as an example, and in the following description, the case of only rendering the head frame of the video of the outer frustum will be mainly described as an example.

[0088] Therefore, even when camera 502 moves forward and backward, or left and right, or performs a zoom operation, the background of the range imaged together with performer 510 is captured as a video corresponding to the change in the viewpoint position or field of view (FOV) accompanying the actual movement of camera 502.

[0089] like Figure 2 and 3 As shown, the monitor video vM including the performer 510 and the background is displayed on the output monitor 503, and this is the captured video. The background of the monitor video vM is the imaging area video vBC. That is, the background included in the captured video is a video rendered in real time.

[0090] As described above, in the imaging system 500 of this embodiment, the background video vB is not simply displayed in a planar manner, but the background video vB including the imaging area video vBC is changed in real time so that the video can be shot in the same manner as when the scene is actually imaged.

[0091] Note that it is also possible to design to reduce the processing load of the system by rendering in real time only the imaging area video vBC as the range reflected by the camera 502 instead of the entire background video vB displayed on the LED wall 505 .

[0092] Here, the production steps of video content as a virtual production will be described, in which imaging is performed by the imaging system 500. Figure 4 As shown, the video content production steps are roughly divided into three stages. These stages are asset creation ST1, production ST2, and post-production ST3.

[0093] Asset creation ST1 is a step of producing 3D background data for displaying background video vB. As described above, background video vB is generated by rendering in real time using 3D background data at the time of imaging. For this purpose, 3D background data as a 3D model is produced in advance.

[0094] Examples of methods of producing 3D background data include full computer graphics (CG), point cloud data scanning, and photogrammetry.

[0095] Full CG is a method of making a 3D model using computer graphics. Among the three methods, this method requires the most man-hours and time, but is preferably used when a non-realistic video (a video that is difficult to shoot in reality, etc.) is expected to be used as the background video vB.

[0096] Point cloud data scanning is a method of generating a 3D model based on point cloud data by performing distance measurement from a certain position using, for example, LiDAR, taking a 360-degree image from the same position with a camera, and placing the color data taken by the camera on the points measured by LiDAR. Compared with full CG, a 3D model can be created in a short time. In addition, it is easy to make a 3D model with higher definition than photogrammetry.

[0097] Photogrammetry is a photogrammetry technique for analyzing parallax information from two-dimensional images obtained by imaging an object from multiple viewpoints to obtain size and shape. 3D model creation can be performed in a short time.

[0098] Note that the point cloud information acquired by LIDAR can be used in 3D data generation through photogrammetry.

[0099] In asset creation ST1, for example, a 3D model that will become 3D background data is produced by using these methods. Of course, the above methods can be used in combination. For example, a part of a 3D model produced by point cloud data scanning or photogrammetry is produced and synthesized by CG.

[0100] ST2 is made in Figure 1 The steps of performing imaging in the imaging studio shown, in this case the essential technologies include real-time rendering, background display, camera tracking, lighting control, etc.

[0101] Real-time rendering is as follows Figure 2 and Figure 3 The rendering process for obtaining the imaging area video vBC at each time point (each frame of the background video vB) is described. This is to render the 3D background data produced in the asset creation ST1 from the viewpoint corresponding to the position of the camera 502 or the like at each time point.

[0102] In this way, real-time rendering is performed to generate a background video vB including each frame of the imaging area video vBC, and the background video vB is displayed on the LED wall 505.

[0103] Camera tracking is performed to obtain imaging information using the camera 502, and to track the position information, imaging direction, and viewing angle of the camera 502 at each time point. By providing the imaging information including these to the rendering engine in association with each frame, real-time rendering according to the viewpoint position of the camera 502, etc. can be performed.

[0104] Imaging information is information that is linked or associated with the video as metadata.

[0105] It is assumed that the imaging information includes position information of the camera 502 at each frame timing, the direction of the camera, the angle of view, the focal length, the F value (aperture value), the shutter speed, and the lens information.

[0106] Illumination control is to control the illumination state in the imaging system 500, and specifically, to control the light amount, emission color, illumination direction, etc. of the light 580. For example, illumination control is performed according to the time setting, location setting, etc. of the scene to be imaged.

[0107] Post-production ST3 refers to various processing performed after imaging, such as video correction, video adjustment, clip editing, video effects, etc.

[0108] As video correction, color gamut conversion, color matching between camera and material, etc. can be performed.

[0109] As video adjustment, color adjustment, brightness adjustment, contrast adjustment, etc. can be performed.

[0110] As clip editing, cutting of clips, adjustment of order, adjustment of time length, and the like can be performed as clip editing.

[0111] As the video effect, synthesis of CG video, special effect video, or the like can be performed.

[0112] Next, the configuration of the imaging system 500 used in producing ST2 will be described.

[0113] Figure 5 It shows that its summary has been referenced Figure 1 , 2 3 is a block diagram of a configuration of an imaging system 500 .

[0114] Figure 5 The imaging system 500 shown in FIG. 5 includes the above-mentioned LED wall 505 including a plurality of LED panels 506, a camera 502, an output monitor 503, and a light 580. Then, as shown in FIG. Figure 5 As shown, the imaging system 500 further includes a rendering engine 520 , an asset server 530 , a synchronization generator 540 , an operation monitor 550 , a camera tracker 560 , an LED processor 570 , a lighting controller 581 , and a display controller 590 .

[0115] Each LED processor 570 is provided corresponding to one or more LED panels 506 , and performs video display driving of the corresponding one or more LED panels 506 .

[0116] The synchronization generator 540 generates a synchronization signal for synchronizing the frame timing of the display video of the LED panel 506 with the frame timing of the imaging of the camera 502 , and provides the synchronization signal to each LED processor 570 , the camera 502 , and the rendering engine 520 .

[0117] The camera tracker 560 generates imaging information from the camera 502 at each frame timing and provides the imaging information to the rendering engine 520. For example, the camera tracker 560 detects the position information of the camera 502 relative to the LED wall 505 or the predetermined reference position and the imaging direction of the camera 502 as one piece of imaging information and provides them to the rendering engine 520.

[0118] As a specific detection method of the camera tracker 560, there is a method of randomly arranging reflectors on the ceiling and detecting the position based on reflected light from infrared light emitted from the camera tracker 560 to the reflectors, and the camera tracker 560 is assembled to the camera 502. In addition, as a detection method, there is also a method of estimating the own position of the camera 502 through information of a gyroscope mounted on a camera platform of the camera 502 or a main body of the camera 502, or image recognition of a captured video of the camera 502.

[0119] In addition, the viewing angle, focal length, F value, shutter speed, lens information, etc. can be provided from the camera 502 to the rendering engine 520 as imaging information.

[0120] The asset server 530 is a server capable of storing the 3D model made in the asset creation ST1, that is, the 3D background data on the recording medium, and reading the 3D model as needed. That is, the asset server serves as a database (DB) of the 3D background data.

[0121] The rendering engine 520 performs a process of generating a background video vB to be displayed on the LED wall 505. To this end, the rendering engine 520 reads necessary 3D background data from the asset server 530. Then, the rendering engine 520 generates a video of an outer frustum used in the background video vB as a video obtained by rendering the 3D background data in a form viewed from pre-specified spatial coordinates.

[0122] Furthermore, the rendering engine 520 specifies a viewpoint position or the like relative to 3D background data by using imaging information provided from the camera tracker 560 or the camera 502 , and renders an imaging area video vBC (inner frustum).

[0123] In addition, the rendering engine 520 combines the imaging area video vBC dynamically changing according to the movement of the camera 502 with the outer frustum to generate the background video vB as video data of one frame. Then, the rendering engine 520 sends the generated video data of one frame to the display controller 590.

[0124] The display controller 590 generates divided video signals nD obtained by dividing the video data of one frame into video parts to be displayed on the respective LED panels 506, and transmits the divided video signals nD to the respective LED panels 506. At this time, the display controller 590 may perform calibration according to individual differences in color rendering, etc., manufacturing errors, etc. between display units.

[0125] Note that the display controller 590 may not be provided, and the rendering engine 520 may perform these processes. That is, the rendering engine 520 may generate the segmented video signal nD, perform calibration, and transmit the segmented video signal nD to the corresponding LED panel 506.

[0126] The LED processor 570 drives each LED panel 506 based on the respectively received divided video signals nD, thereby displaying the entire background video vB on the LED wall 505. The background video vB includes an imaging area video vBC rendered according to the position of the camera 502 at that point in time and the like.

[0127] The camera 502 can image the performance of the performer 510, including the background video vB displayed on the LED wall 505 in this manner. The video obtained by imaging with the camera 502 is recorded on a recording medium in the camera 502 or an external recording device (not shown), and is provided to the output monitor 503 in real time and displayed as a monitor video vM.

[0128] The operation monitor 550 displays an operation image vOP for controlling the rendering engine 520. The engineer 511 can perform necessary settings and operations for rendering the background video vB while viewing the operation image vOP.

[0129] The lighting controller 581 controls the emission intensity, emission color, irradiation direction, etc. of the lamp 580. For example, the lighting controller 581 can control the lamp 580 asynchronously with the rendering engine 520, or can perform control synchronously with the imaging information and rendering process. Therefore, the lighting controller 581 can perform light emission control according to instructions from the rendering engine 520, a main controller (not shown), etc. In addition, the rendering engine 520 can control the lamp 580.

[0130] Figure 6 An example of processing by the rendering engine 520 in the imaging system 500 having such a configuration is shown.

[0131] In step S10 , the rendering engine 520 reads the 3D background data to be used at this time from the asset server 530 , and expands the 3D background data in the internal work area.

[0132] At this stage, a video can be generated to be used as the outer frustum.

[0133] Thereafter, the rendering engine 520 repeats the process from step S30 to step S60 until it is determined in step S20 that the display of the background video vB is ended based on the read 3D background data.

[0134] In step S30, the rendering engine 520 acquires imaging information from the camera tracker 560 and the camera 502. Therefore, the position and state of the camera 502 to be reflected in the current frame are confirmed.

[0135] In step S40, the rendering engine 520 performs rendering based on the imaging information. That is, the viewpoint position relative to the 3D background data is specified based on the position, imaging direction, viewing angle, etc. of the camera 502 to be reflected in the current frame, and rendering is performed. At this time, video processing reflecting the focal length, F value, shutter speed, lens information, etc. can also be performed. Through this rendering, video data as the imaging area video vBC (inner frustum) can be obtained. The outer frustum is pre-generated as a fixed video in step S10, or the outer frustum can be generated for each frame in step S40.

[0136] In step S50, the rendering engine 520 performs a process of synthesizing the outer frustum of the entire background video and the video reflecting the viewpoint position of the camera 502, that is, the imaging area video vBC. For example, the process is to synthesize the video generated by reflecting the viewpoint of the camera 502 and the video of the entire background rendered at a specific reference viewpoint. Thus, a background video vB of one frame displayed on the LED wall 505 is generated, that is, the background video vB including the imaging area video vBC.

[0137] The processing in step S60 is performed by the rendering engine 520 or the display controller 590. In step S60, the rendering engine 520 or the display controller 590 generates a segmented video signal nD obtained by segmenting the background video vB of one frame into videos to be displayed on the respective LED panels 506. Calibration may be performed. Then, the segmented video signals nD are respectively sent to the LED processor 570.

[0138] In the above-described process, the background video vB including the imaging area video vBC captured by the camera 502 is displayed on the LED wall 505 at each frame timing.

[0139] By the way, in Figure 5Only one camera 502 is shown in FIG. 5 , but imaging may be performed by multiple cameras 502 . Figure 7 A configuration example is shown in the case of using a plurality of cameras 502a and 502b. The cameras 502a and 502b can independently perform imaging in the performance area 501. In addition, synchronization between the cameras 502a and 502b and the LED processor 570 is maintained by the synchronization generator 540.

[0140] The output monitors 503a and 503b are provided corresponding to the cameras 502a and 502b, respectively, and are configured to display the videos captured by the corresponding cameras 502a and 502b as monitor videos vMa and vMb, respectively.

[0141] In addition, camera trackers 560a and 560b are respectively provided corresponding to cameras 502a and 502b, and detect the position and imaging direction of each of the corresponding cameras 502a and 502b. Imaging information from camera 502a and camera tracker 560a and imaging information from camera 502b and camera tracker 560b are sent to rendering engine 520.

[0142] The rendering engine 520 may perform rendering for obtaining the background video vB of each frame by using imaging information of one or both of the camera 502 a side or the camera 502 b side.

[0143] Note that although Figure 7 An example using two cameras 502 a and 502 b is shown, but three or more cameras 502 may be used to perform imaging.

[0144] However, when a plurality of cameras 502 are used to render and display imaging area videos vBC (inner frustums) corresponding to the respective cameras 502 by using respective pieces of imaging information, there are cases where the imaging area videos vBC interfere with each other. Figure 7 In the example of using two cameras 502a and 502b, the imaging area video vBC corresponding to the camera 502a is shown, but in the case of using the video of the camera 502b, the imaging area video vBC corresponding to the camera 502b is also required. In this case, when the imaging area videos vBC corresponding to the respective cameras 502a and 502b are simply displayed, they interfere with each other. Therefore, it is necessary to design the display of the imaging area video vBC.

[0145] <2. Configuration of information processing equipment>

[0146] Next, we will refer to Figure 8 A configuration example of the information processing device 70 that can be used in asset creation ST1, production ST2, and post-production ST3 is described.

[0147] The information processing device 70 is a device capable of performing information processing, particularly video processing, such as a computer device. Specifically, as the information processing device 70, a personal computer, a workstation, a portable terminal device such as a smartphone and a tablet, a video editing device, etc. are assumed. In addition, the information processing device 70 may be a computer device configured as a server device or a computing device in cloud computing.

[0148] In the case of the present embodiment, specifically, the information processing device 70 can be used as a 3D model creation device that creates a 3D model in asset creation ST1.

[0149] Furthermore, the information processing apparatus 70 can also function as the rendering engine 520 and the asset server 530 constituting the imaging system 500 used in producing ST2.

[0150] Furthermore, the information processing device 70 can also function as a video editing device configured to perform various types of video processing in post-production ST3.

[0151] In addition, the information processing device 70 can also be used to generate a later reference Fig.10 The information processing device (rendering engine 31) functions as the simulated video vSM in the pre-visualization described above.

[0152] Figure 8 The CPU 71 of the illustrated information processing device 70 executes various processes according to a program stored in a nonvolatile memory unit 74 such as a ROM 72 or an electrically erasable programmable read-only memory (EEP-ROM), or a program loaded from a storage unit 79 to a RAM 73. The RAM 73 also appropriately stores data and the like necessary for the CPU 71 to execute various processes.

[0153] The video processing unit 85 is configured as a processor that performs various types of video processing. The processor is a processor that can perform any one or more of, for example, 3D model generation processing, rendering, DB processing, video processing including color / brightness adjustment processing, video editing processing, video analysis / detection processing, etc.

[0154] The video processing unit 85 may be implemented by, for example, a CPU separate from the CPU 71, a graphics processing unit (GPU), general purpose computing on a graphics processing unit (GPGPU), an artificial intelligence (AI) processor, or the like.

[0155] Note that the video processing unit 85 may be provided as a function in the CPU 71 .

[0156] The CPU 71, the ROM 72, the RAM 73, the nonvolatile memory unit 74, and the video processing unit 85 are connected to each other via a bus 83. An input / output interface 75 is also connected to the bus 83. An input unit 76 including an operating element or an operating device is connected to the input / output interface 75. As the input unit 76, for example, various operating elements or operating devices such as a keyboard, a mouse, a key, a trackball, a dial, a touch panel, a touch pad, a grating panel, a remote controller, and the like can be cited.

[0157] The input unit 76 detects an operation by a user, and the CPU 71 interprets a signal corresponding to the input operation.

[0158] It is also assumed that a microphone is the input unit 76. A voice uttered by a user may also be input as the operation information.

[0159] Furthermore, a display unit 77 including a liquid crystal display (LCD), an organic electroluminescence (EL) panel, or the like, and an audio output unit 78 including a speaker, or the like are integrally or individually connected to the input / output interface 75 .

[0160] The display unit 77 is a display unit that performs various displays, and includes, for example, a display device provided in a housing of the information processing device 70 , a separate display device connected to the information processing device 70 , and the like.

[0161] The display unit 77 performs display of various images, operation menus, icons, messages, and the like on a display screen, that is, display as a graphical user interface (GUI), based on instructions from the CPU 71 .

[0162] In some cases, a storage unit 79 including a hard disk drive (HDD), a solid-state memory, or the like, or a communication unit 80 is connected to the input / output interface 75 .

[0163] The storage unit 79 can store various data and programs. A DB can also be configured in the storage unit 79.

[0164] For example, in the case where the information processing device 70 functions as the asset server 530 or the rendering engine 31 described later, a DB storing a 3D background data group can be constructed by using the storage unit 79 .

[0165] The communication unit 80 performs communication processing via a transmission path such as the Internet, performs wired / wireless communication with various devices such as an external DB, an editing device, and an information processing device, performs bus communication, and the like.

[0166] For example, in the case where the information processing device 70 functions as the rendering engine 520 or the rendering engine 31 described later, it is possible to access the DB as the asset server 530 through the communication unit 80 and receive imaging information from the camera 502 or the camera tracker 560 .

[0167] Furthermore, also in the case of the information processing apparatus 70 used in the post-production ST3 , the communication unit 80 can access a DB or the like as the asset server 530 .

[0168] A drive 81 is also connected to the input / output interface 75 as necessary, and a removable recording medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted.

[0169] The drive 81 can read video data, various computer programs, and the like from the removable recording medium 82. The read data is stored in the storage unit 79, and the video and audio included in the data are output by the display unit 77 and the audio output unit 78. In addition, the computer program and the like read from the removable recording medium 82 are installed in the storage unit 79 as necessary.

[0170] In the information processing apparatus 70, for example, software for the processing in the present embodiment may be installed through network communication via the communication unit 80 or the removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0171] <3. Configuration including artifact determination unit>

[0172] Hereinafter, a configuration example of the information processing device 70 including a determination unit that determines the occurrence of an artifact such as moire fringes will be described.

[0173] Such information processing device 70 may be, for example, Figure 5 and 7 The information processing device 70 such as the rendering engine 520 in the imaging system 500 shown, or may be Fig.10 The information processing device 70 used in the pre-visualization system 700 described in FIG.

[0174] When the camera 502 in the imaging system 500 images the LED wall 505 , artifacts such as moiré fringes (moiré fringes, grids, color shifts, etc.) appear due to interference between pixels of the image sensor of the camera 502 and pixels of the LED panel 506 .

[0175] In order to avoid moire fringes, etc., the camera 502 needs to blur the LED panel 506 without focusing. However, changing the position of the camera 502 and the performer 510 or adjusting the lens at the actual imaging location reduces the efficiency of video production.

[0176] Therefore, in the present embodiment, the information processing apparatus 70 including the determination unit that performs the process of determining the occurrence of moire fringes or the like is used in the imaging system 500 and the pre-visualization system 700 .

[0177] First, refer to Fig. 9 Describe the position of pre-visualization in the video production process.

[0178] Fig. 9 The above reference shows Figure 4 Describes the process of asset creation (ST1), production (ST2), and post-production (ST3).

[0179] Production (ST2) is the stage where imaging is actually performed in a studio, but this is divided into imaging preparation and imaging.

[0180] It is assumed that pre-visualization is mainly performed at a stage before imaging preparation. Note that the order in the time series is not necessarily limited, but it is only required to perform pre-visualization at a stage after making a 3D model for rendering the background video vB in asset creation (ST1).

[0181] In this sense, pre-visualization can be considered to be performed at the final stage of asset creation (ST1), or it can be considered to be performed at a stage prior to imaging preparation in production (ST2).

[0182] In pre-visualization, video confirmation and moire fringe determination are performed. Video confirmation is a process for enabling a director or a crew member to confirm the captured video vC obtained by shooting before shooting. Confirmation including the color tone of the captured video vC is mainly achieved by analog video.

[0183] Moire determination is, for example, a process performed when a worker virtually operates the camera position while confirming a video. In the event that the occurrence of moire or the like is predicted by the moire determination, a warning can be issued. This makes it possible to avoid artifacts in subsequent imaging.

[0184] In the imaging preparation stage, color calibration is performed together with various settings for LEDs, cameras, etc. in imaging system 500. Specifically, a calibration lookup table (LUT) is generated that cancels color changes that occur when LED wall 505 is imaged by camera 502. The calibration LUT is, for example, a 3D-LUT.

[0185] Note that, instead of 3D-LUT, a combination of a matrix and 1D-LUT may be performed, or preparation for calibration performed in other video signal processing may be performed.

[0186] Even in the imaging preparation stage, moire fringe determination and moire fringe warning based on moire fringe determination can be performed. That is, moire fringe determination can be performed by specifying the actual model of the LED panel 506 and the camera 502 and setting the position and direction of the camera 502 in the imaging preparation stage.

[0187] During the imaging phase, moiré fringe determination, calibration LUT and on-set camera setup are performed.

[0188] The moire determination in this case is a moire determination performed by the rendering engine 520. Based on this determination, a moire warning can be output to the camera operator 512 or the like in real time.

[0189] The calibration LUT is a process of eliminating color variations associated with background video display by the LED wall 505, the camera 502, etc. by applying the LUT generated by the color calibration in preparation for imaging.

[0190] The on-site camera setting is a process of acquiring the camera setting set in the pre-visualization from, for example, a cloud server or the like and reflecting the camera setting in the camera 502 of the studio.

[0191] Reference Fig.10 Describes a configuration example as an embodiment. Note that Fig.10 , the configuration of the imaging system 500 and the configuration for pre-visualization are described in parallel.

[0192] The upper part of the figure shows Figure 5 et al., and the lower part of the figure shows a configuration used in pre-visualization (pre-visualization system 700).

[0193] exist Fig.10 , regarding the imaging system 500, a rendering engine 520, an LED processor 570, an LED wall 505 including a plurality of LED panels 506, a camera 502, a camera signal processing unit 515, an operation monitor 550 or an output monitor 503 are shown.

[0194] The operation monitor 550 or the output monitor 503 in the figure comprehensively represents a monitor that displays the shooting video vC and the monitor video vM in the studio as the imaging location. Hereinafter, for the sake of explanation, the monitor will be simply referred to as "monitor 550 etc."

[0195] The rendering engine 520 is illustrated as including a rendering unit 520a, a CG camera setting unit 520c, and a determining unit 520d.

[0196] As described above, the rendering unit 520a has a function of rendering the inner frustum (imaging area video) vBC according to the positions of the outer frustum and the camera 502 and the imaging direction and generating the background video vB to be displayed on the LED wall 505. That is, the rendering unit 520a is a processor that performs Figure 6 The processing function.

[0197] The CG camera setting unit 520c and the determination unit 520d in the rendering engine 520 will be described later.

[0198] Note that the rendering engine 520 may have a color calibration function. The color calibration function is a function of performing color conversion for eliminating color variations caused by processing by the LED processor 570, LED light emission characteristics of the LED panel 506 constituting the LED wall 505, light receiving characteristics of the camera, processing, and the like.

[0199] That is, the rendering unit 520a may perform color conversion on the rendered background video vB using the LUT generated through color calibration in the imaging preparation stage.

[0200] When color conversion using the correction LUT is performed in the rendering engine 520 in this manner, for example, it can be considered that no color change occurs on the display side of the background video vB.

[0201] Note that the color change here is a color change caused by the processing or light emission characteristics until the rendered background video vB is displayed on the LED wall 505, or the light receiving characteristics or processing of the camera, etc.

[0202] In the case where the rendering engine 520 does not have a color calibration function, or in the case where the function is not enabled even if the rendering engine has this function, there is a situation where changes occur due to color tone, that is, color shading, RGB balance, etc. between the color in the state of being rendered from the 3D model and the color in the state of being displayed on the LED wall 505.

[0203] Fig.10 The camera signal processing unit 515 is shown to perform signal processing on the captured video signal of the camera 502. Figure 5 and Figure 7 Although omitted in the figure, the camera signal processing unit 515 may be formed by a processor or the like in the camera 502 , or may be provided as a device having a unit separate from the camera 502 .

[0204] The video signal captured by the camera 502 is subjected to development processing including brightness processing, color processing, etc., resizing processing, etc. by the camera signal processing unit 515, and is output as a captured video vC or a monitoring video vM.

[0205] The captured video vC is recorded on a recording medium as a main line video, or is sent to another device for broadcast or distribution. In addition, the captured video vC is used as a monitor video vM and is displayed on a monitor 550 or the like.

[0206] For the imaging system 500 as described above, for example, the pre-visualization system 700 is configured as a system outside the studio. The pre-visualization system 700 includes at least an information processing device 70 serving as a rendering engine 31 and a monitor 40 .

[0207] The rendering engine 31 includes, for example, an information processing device 70 separate from the rendering engine 520 .

[0208] The rendering engine 31 includes a rendering unit 32 and a video processing unit 33 .

[0209] The rendering unit 32 has a rendering function similar to that of the rendering unit 520 a in the above-described rendering engine 520 , and can generate a background video vBP based on the 3D model.

[0210] The rendering unit 32 can render the background video vB by using at least the 3D model created in the asset creation (ST1) step. Figure 5 and 7 ) obtains a 3D model that is the same as the 3D model used in the production (ST2) step, and performs rendering using the 3D model.

[0211] Then, the rendering unit 32 renders the inner frustum using the 3D model, for example. Note that in the case of generating the inner frustum (imaging area video vBC), any camera position and direction are input, and rendering can be performed at a viewpoint position relative to the 3D model according to the input camera position and direction.

[0212] In the sense of being distinguished from the imaging area video vBC generated by the rendering engine 520 when imaging, the background video generated by the rendering unit 32 is referred to as “background video vBP” for the sake of explanation.

[0213] The video processing unit 33 represents a function of performing various video processing on the background video vBP.

[0214] The video processing unit 33 performs actual shot video conversion processing as one of the video processing types. The actual shot video conversion processing is a processing for generating a simulated video vSM that imitates the camera 502 actually used for imaging. In the actual shot video conversion processing, a simulated video vSM is generated for the background video vBP by using at least a processing parameter that realizes the characteristics of brightness or color when imaging is performed by the camera 502 used in the imaging system 500.

[0215] By setting parameters related to brightness or color to be the same as those of the camera 502 during imaging, a simulated video vSM having the same color tone as that of the captured video vC is generated and displayed on the monitor 40 .

[0216] The monitor 40 represents a monitor device that displays a video output from the information processing device 70 as the rendering engine 31, and may be, for example, a monitor device at a place different from the imaging studio or a monitor device in the imaging studio. The monitor 40 may have any form as long as it displays the simulation video vSM at least in the pre-visualization stage.

[0217] In the rendering engine 31, the rendering unit 32 generates a background video vBP, and the video processing unit 33 performs actual shot video conversion processing to generate a simulated video vSM. By displaying it on the monitor 40, the staff can confirm the simulated video vSM having the same color tone as the background video included in the shot video vC at the time of imaging. Therefore, the color can be confirmed before imaging, and the color can be adjusted in the camera 502 or the camera signal processing unit 515 as needed in the imaging preparation stage.

[0218] In the case of an imaging system 500 that uses a virtual background video vB as in the virtual production of the present embodiment, the color tone and the like of the captured video vC actually captured by the camera 502 may be different from what the director or the staff envisioned. In this case, it is necessary to adjust the color tone by making various setting changes and the like in a studio or the like as the imaging location, which is an extremely troublesome and time-consuming work in reality. This reduces the efficiency of video production. Therefore, it is very useful to enable the pre-visualization system 700 to confirm a video having the same color tone as the actually captured video vC before imaging.

[0219] As the actual shot video conversion processing of the video processing unit 33, at least the color tone of the background video vBP rendered by the rendering unit 32 is set to be equivalent to the video of the background included in the shot video vC at the time of actual imaging.

[0220] To this end, specifically, the video processing unit 33 adjusts parameters such as RGB gain, white balance value, gamma value, and offset value to parameters of the camera 502 .

[0221] For example, the video processing unit 33 performs processing of giving R, G, and B gains, which are similar to the gain values ​​of the pixel signals of the respective colors read from the image sensor in the camera 502, to the background video vBP.

[0222] In addition, the video processing unit 33 performs white balance processing on the background video vBP, and its white balance value is the same as the white balance value when imaged by the camera 502 in the actual shooting video conversion process.

[0223] In addition, parameters such as the spectral transmittance of optical components placed on the optical path, the spectral transmittance of on-chip color filters, on-chip lenses, etc. of the image sensor, and the spectral sensitivity characteristics of the photodiode of the image sensor are also considered. Specifically, the parameter is obtained by integrating the spectral characteristics of the LEDs of the LED panel 506, the spectral transmittance when a diffusion plate, an anti-reflection film, a protective film, etc. are attached to the LED panel, and the product of the spectral transmittance and the spectral sensitivity characteristics over the wavelength. Then, color processing according to the parameter is performed.

[0224] As a result, a simulated video vSM is obtained as a background having the same hue as the background in the captured video vC. As a result, on the monitor 40, a video having the same hue as when the captured video vC is viewed on the monitor 550 or the like during imaging can be confirmed.

[0225] Furthermore, as actual shot video conversion processing, the video processing unit 33 may perform processing to make the angle of view of the cutout range, zoom factor setting, etc. from the image sensor of the camera 502 the same on the background video vBP.

[0226] Furthermore, as actual shot video conversion processing, the video processing unit 33 may perform processing on the background video vBP that reflects the lens distortion characteristics of the camera 502. As a result, the simulation video vSM is made closer to the shot video vC.

[0227] Meanwhile, in the case where the color change of the LED wall 505 is eliminated by the color calibration processing in the rendering engine 520 at the time of imaging, the actual shot video conversion processing may be processing that reflects the color change, etc. on the camera side. This is because it is assumed that the background video vB displayed on the LED wall 505 has no color change compared to the rendered state.

[0228] On the other hand, without performing the color calibration as described above, sometimes a color change has occurred in the background video vB displayed on the LED wall 505. In this case, in the actual shot video conversion process in the video processing unit 33, a process of reflecting the brightness (intensity) characteristics of each color on the LED side and the like is also performed. As a result, the simulated video vSM can be made close to the shot video vC.

[0229] Note that, as a premise for the above-mentioned actual shot video conversion processing, the camera settings such as the frame rate, shutter speed, and exposure value of the virtual camera assumed when the rendering engine 31 renders the simulated video vSM are similar to those of the actual camera 502. That is, the video processing unit 33 generates a simulated video vSM simulating the color tone of the actual shot video vC by performing color processing such as RGB gain and white balance value after setting the background video vBP to have RGB values ​​according to various camera settings such as the frame rate, shutter speed, exposure value, ND (neutral density) setting.

[0230] Furthermore, the brightness of the LED wall 505 may be reflected similarly to the camera setting.

[0231] The rendering engine 31 includes an adjustment operation unit 35 that can be used for color adjustment operations by a user such as a worker.

[0232] For example, a GUI or the like using the display unit 77 or the input unit 76 of the information processing apparatus 70 serving as the rendering engine 31 is provided to the user. As a result, the user can operate the color tone of the video while viewing the monitor 40.

[0233] That is, when the staff watches the simulated video vSM through the actual shooting video conversion process and feels that the color tone is different from the desired color tone, the staff performs color adjustment to obtain the desired color.

[0234] Furthermore, for example, the adjusted contents are provided as color conversion information Ic to the camera 502 and the camera signal processing unit 515 via the cloud server 41 so that the adjusted colors can be reflected at the time of imaging. The color conversion information Ic is, for example, a LUT or a color decision list (CDL).

[0235] Note that the video processing unit 33 may send the color conversion information Ic to the camera 502 or the camera signal processing unit 515 without passing through the cloud server 41 .

[0236] Furthermore, the adjustment operation unit 35 can allow the user to perform operations of changing and setting various camera settings. The video processing unit 33 performs a change process of the camera setting reflected in the simulation video vSM according to the input, and can output the camera setting information CS.

[0237] The camera setting information CS mentioned here indicates, for example, a read size setting from the image sensor of the camera 502, a shutter speed setting, an exposure setting, an ND setting, lens characteristics, focus / aperture / zoom operation settings, a white balance value, RGB gain, and the like.

[0238] In the above-described actual shot video conversion process, basically, these setting values ​​are adjusted to the settings of the camera 502 used for imaging, thereby simulating the actual shot video vC.

[0239] Then, by adjusting these camera settings, you can prepare various camera setups in advance.

[0240] Then, the video processing unit 33 can store and transmit the camera setting information CS indicating the camera setting changed according to the operation. For example, the camera setting information CS according to the operator's adjustment operation is uploaded to the cloud server 41 and can be reflected in the camera 502 or the like when imaging later.

[0241] Furthermore, the rendering engine 31 includes a determination unit 36 ​​, a position operation unit 37 , and a communication unit 38 , and can perform moire determination and moire warning processing.

[0242] The position operation unit 37 is an interface function that enables the user to arbitrarily operate the virtual camera position.

[0243] The determination unit 36 ​​has a function of determining whether moire fringes appear by using a specific algorithm in a state where the camera position is set.

[0244] The algorithm which will be described in detail later is an algorithm for determining that artifacts such as moire fringes or scan lines appear when an image is actually captured.

[0245] The determination unit 36 ​​performs determination processing using various information acquired by the communication unit 38 , information on the virtual camera position operated by the position operation unit 37 , and information input by the user from the adjustment operation unit 35 .

[0246] Specifically, the determination unit 36 ​​determines the occurrence of artifacts such as moiré fringes by using the spacing of LEDs (the spacing between adjacent LEDs to each other), the pixel spacing of the image sensor of the camera 502 for imaging (the spacing between adjacent pixels to each other), the characteristics of the optical LPF, the focal length, F value, and focusing distance of the lens, and the value of the distance between the LED panel 506 and the camera 502, wherein the LED is a light-emitting element of the LED panel 506 for imaging.

[0247] Note that since the actual LED panel 506 and the camera 502 do not exist in the pre-visualization stage, the distance value between the LED panel 506 and the camera 502 is a virtual value corresponding to the operation of the position operating unit 37 .

[0248] Therefore, the communication unit 38 communicates with, for example, the LED panel 506, the camera 502, etc., which are actually used for imaging, acquires values ​​such as the LED interval, the pixel interval of the image sensor, and the characteristics of the optical LPF, and provides the acquired values ​​to the determination unit 36. Regarding the LED interval, the pixel interval of the image sensor, the characteristics of the optical LPF, etc., the communication unit 38 can communicate with the LED panel 506 and the camera 502 to obtain information such as the model and the model number, and the determination unit 36 ​​can refer to the internal DB and acquire the respective values ​​according to the model.

[0249] Furthermore, the determination unit 36 ​​may acquire each of the above values ​​in response to a user inputting a specific value or model number.

[0250] Furthermore, the determination unit 36 ​​acquires the focal length, F value, and focus distance according to the camera setting, operation, and the like of the virtual camera by the user, and performs determination processing using them.

[0251] Furthermore, the determination unit 36 ​​may obtain a value of the distance between the LED panel 506 and the camera 502 according to the camera position corresponding to the operation of the position operation unit 37 , and use the value for determination.

[0252] In the case where the occurrence of moire fringes is predicted in the determination process, the determination unit 36 ​​notifies the video processing unit 33 of information on the occurrence of moire fringes, information on the occurrence position, and the like.

[0253] The video processing unit 33 causes the monitor 40 to display a video indicating, for example, a moire warning or a moire position.

[0254] For example, Fig.11 A video of a simulation video vSM including a background video vBP is shown, and is an image further showing a virtual LED wall 54 and a virtual camera 51 .

[0255] The video processing unit 33 can display not only the simulation video vSM of the background video vBP but also a video in which the viewpoint position is set to a certain position in a studio, for example.

[0256] exist Fig.11 In the video, position information 52 about the LED wall 54 and position information 52 about the camera are further displayed.

[0257] Furthermore, as the origin information 50 , the origin position and the three axial directions of X, Y, and Z serving as a reference for each position in the space are shown.

[0258] Fig.12 1 is an example in which a moire warning 53 is displayed and a moire position 57 is shown.

[0259] For example, the position of camera 51 changes from Fig.11 The position in the image is changed, but it is assumed that the moire fringe generation prediction is performed in this state. In this case, the moire fringe warning 53 is displayed, and the position where the moire fringe occurs is displayed as the moire fringe position 57.

[0260] The user can arbitrarily change the position of the camera 51 on the video. For example, when the user performs an operation to specify the camera 51, such as Fig.13 The moving axes 55 in the X, Y, and Z directions are shown. When the user performs an operation such as dragging while aligning the cursor 56 with a certain moving axis 55, the position of the camera 51 in the virtual space is changed. For example, the user interface (or user interface) described above is provided by the function of the position operation unit 37.

[0261] Therefore, the user can confirm whether moire fringes appear while arbitrarily moving the virtual camera 51 .

[0262] Therefore, the camera position and the camera path can also be set at the time of actual imaging so as not to cause moire fringes.

[0263] Fig.14 Processing related to determination of moire fringes and the like by the rendering engine 31 is shown.

[0264] In step S401 , the rendering engine 31 monitors whether it is a determination opportunity.

[0265] For example, monitoring whether camera position manipulation is performed.

[0266] In the case where the user performs the camera position operation through the function of the position operation unit 37, the rendering engine 31 proceeds to step S402 and performs determination processing using information about the changed camera position and imaging direction. That is, the determination processing is performed using the LED interval, the pixel interval of the image sensor, the characteristics of the optical LPF, the focal length, the F value, the focus distance, and the value of the distance between the LED panel 506 and the camera 502 acquired at this time. The distance between the LED panel 506 and the camera 502 is obtained by calculating the virtual position relationship according to the camera position operation of the user.

[0267] In addition, the determination opportunity monitored in step S401 is not limited to the camera position operation. For example, even in the case where a zoom operation, a focus operation, an exposure adjustment operation, etc. are performed as a virtual camera operation, the process can proceed to step S402.

[0268] That is, as an element for determining processing, the following opportunity can be used as a determination opportunity, in which although the interval of LEDs, the pixel interval of the image sensor, and the characteristics of the optical LPF generally do not change, any one of the focal length, F value, focusing distance, the distance between the LED panel 506 and the camera 502, etc. changes.

[0269] In step S403 , the rendering engine 31 confirms the result of the determination processing, and in the case of determining that moire or the like occurs, performs a process of outputting a moire warning in step S404 .

[0270] As mentioned above, display Fig.12 The moire warning 53 is shown so that the staff can know the occurrence of moire etc. at the stage before imaging. Therefore, the camera position and camera path that do not cause moire etc. can be considered before actual imaging.

[0271] at the same time, Fig.10 On the rendering engine 520 side, a CG camera setting unit 520c and a determination unit 520d are shown in addition to the rendering unit 520a.

[0272] As a result, the rendering engine 520 can perform moiré fringe determination in preparation for imaging or in actual imaging.

[0273] The CG camera setting unit 520c receives imaging information such as the position, imaging direction, viewing angle, focal length, F value, shutter speed, lens information, etc. of the camera 502 from the camera tracker 560, and enables the rendering unit 520a to perform rendering corresponding thereto.

[0274] The determination unit 520d determines whether moire fringes, etc. are currently generated for the rendered imaging area video vBC by using the information obtained by the CG camera setting unit 520c. That is, the determination unit 520d determines the occurrence of current moire fringes, etc. using an algorithm similar to the algorithm of the above-mentioned determination unit 36.

[0275] Then, in the case where it is determined that moire or the like has been generated, for example, information about it is sent to the camera 502 , and it is caused to display a moire warning on a viewfinder or the like.

[0276] Fig.15 An example of a process for performing such a real-time moire warning is shown. Fig.15 In addition to Figure 6 In addition to steps S10 to S60 described in the above, step S70 and subsequent steps are also performed.

[0277] As described above, the rendering engine 520 performs the processing from step S30 to step S60 for each frame timing. As a result, the background video vB is displayed on the LED wall 505.

[0278] In step S70, the rendering engine 520 monitors whether it is a confirmed opportunity. Fig.14 In step S401 in this case, for example, the position and imaging information of the camera 502 acquired from the actual camera tracker 560 is acquired to check whether it is a determination opportunity.

[0279] Then, if it is a determination opportunity, the process proceeds to step S71, and a determination process is performed.

[0280] Note that the determination process may be performed every frame without performing the check in step S70 .

[0281] In step S72, the rendering engine 520 checks the result of the determination process, and in the case where it is determined that moire fringes or the like have occurred, a process of outputting a moire fringes warning is performed in step S73. For example, the rendering engine 520 sends information about the moire fringes warning to the camera 502 so that the moire fringes warning is displayed on the viewfinder, etc. Alternatively, the moire fringes warning may be displayed on the monitor 550, etc.

[0282] As described above, the camera operator 512 can recognize that moire fringes are currently generated during imaging and process them. For example, the situation can be processed by moving the camera position or adjusting the focus.

[0283] The above-described processing is a determination processing during imaging, but similar processing can be performed during imaging preparation in the imaging system 500. In addition, even in a period when the background video vB is not rendered, that is, in a period when the background video vB is not displayed on the LED wall 505 during imaging preparation and the like, it is possible to determine whether moire fringes or the like appear according to the position of the camera 502. This is because the LED interval of the LED panel 506, the pixel interval of the image sensor of the camera 502, and the characteristics of the optical LPF can be acquired in advance, and the focal length of the lens of the camera 502, the F value, the focusing distance, and the distance between the LED panel 506 and the camera 502 can be acquired from the camera tracker 560 as imaging information.

[0284] <4. Confirm the processing>

[0285] The determination processing performed by the determination unit 36 ​​and the determination unit 520d as described above will be described in detail.

[0286] First, calculation of the size of an image formed on the image sensor of a camera will be described.

[0287] Fig.16 The relationship among the optical axis of the lens 21, the object AA', and the real image BB' is shown. The lens 21 conceptually represents one or more lenses of the camera 502, for example.

[0288] Light incident on the lens 21 parallel to the optical axis (line segment A'O' in the figure) passes through the focus C of the lens 21. On the other hand, light passing through the center O of the lens 21 (line segment A'O in the figure) directly propagates in a straight line, and a real image is formed at the point B' where the two intersect.

[0289] The distance between the lens 21 and the object is represented by “s”, the focal length of the lens 21 is represented by “f”, and the distance between the lens 21 and the real image is represented by “i”.

[0290] Since ΔAOA' and ΔBOB' are similar, the following (Math. 1) holds. Note that "Δ" represents a triangle, and for example "ΔAOA'" is a triangle having points A, O, and A' in the figure as vertices.

[0291] [Mathematical formula 1]

[0292]

[0293] Furthermore, since ΔOCO′ and ΔBCB′ are similar to each other, (Mathematical Formula 2) holds true.

[0294] [Mathematical formula 2]

[0295]

[0296] Next, since line segment AO and line A'O' are parallel to each other,

[0297] AA'=OO'

[0298] When i is deleted from above, the following formula is obtained.

[0299] [Mathematical formula 3]

[0300]

[0301] Therefore, the magnification of the image formed on the image sensor of the camera 502 can be calculated by the above (Math. 3) according to the distance s from the lens 21 to the object and the focal length f of the lens 21 .

[0302] Furthermore, (Math. 4), which is a lens formula, is derived from a relational expression of similarity between (Math. 1) and (Math. 2).

[0303] [Formula 4]

[0304]

[0305] Note that in the case of s<f, no real image is formed. Since i→∞ is satisfied in s→f, s>f is satisfied due to physical constraints.

[0306] Next, image blur will be described.

[0307] Generally, in a camera using a solid-state imaging element, an optical LPF is inserted between a lens and an image sensor to suppress aliasing distortion due to sampling in pixels of the image sensor.

[0308] Fig.17 A state in which the optical LPF 22 is arranged between the lens 21 and the image sensor 20 is shown.

[0309] The optical LPF 22 uses a substance having a refractive index that differs depending on polarized light, such as calcite or crystal, to blur an image by double birefringence.

[0310] Fig.18 The birefringence of the optical LPF 22 is shown. The width indicated by "b" is the shift amount (blur amount) of the image due to the characteristics of the optical LPF 22, and is described as "shift amount b" for the sake of description.

[0311] Assume that the image of the subject is Fig.18 The frequency characteristics of the double image shown with the shift amount b are as follows by performing Fourier transform.

[0312] [Mathematical formula 5]

[0313]

[0314] Since the amplitude characteristic is √{(real part)^2+(imaginary part)^2}, the following mathematical formula is obtained. Note that "^" represents power and "^2" represents a square.

[0315] ω is the angular frequency and 2π × the spatial frequency.

[0316] [Mathematical formula 6]

[0317]

[0318] Generally, the offset b is set to 1 / 2 of the pixel width so that a zero point is added to the sampling frequency of the pixel, but some cameras narrow or widen the bandwidth. In addition, the optical LPF 22 may overlap to provide a zero point for two frequencies. For example, the amplitude characteristic in the case where two optical LPFs 22 of the shift amount b and the shift amount b' overlap each other is in the form of a product, and is as follows.

[0319] [Formula 7]

[0320]

[0321] The photodiode of the image sensor 20 uses an on-chip lens to focus light in order to obtain sensitivity. Fig.19 All light on the pixel 20P having a width Ws shown is collected in the photodiode and is ignored in the wiring and the like, and the frequency characteristics are as follows through Fourier transform.

[0322] [Mathematical formula 8]

[0323]

[0324] Here, it is assumed that the image is blurred and the light passing through the lens 21 is diffused in a circular shape having a diameter δ without being condensed at one point.

[0325] Fig. 20 It is shown how the image is blurred together with the lens 21 and the aperture 23 .

[0326] Assume that the subject is located at position A, and the real image may be located at position B (solid line).

[0327] Furthermore, as shown in the figure, the distance from the lens 21 to the subject is "s", the distance from the lens 21 to the real image is "i", the focal length of the lens 21 is "f", and the aperture of the lens 21 narrowed by the aperture 23 is "d" (length of OO').

[0328] As indicated by the one-dot chain line, when the focus is at position Af behind the subject by a distance Δsf, the real image may be at position Bf in front of position B by a distance Δif and blurred to a diameter δ at position B.

[0329] Furthermore, as indicated by the dotted line, when the focus is at position An a distance Δsn in front of the subject, the real image may be located at position Bn a distance Δin behind position B and blurred to a diameter δ at position B.

[0330] Based on the above (Math. 4), the relational expression between the distance to the subject, the distance to the real image, and the focal length in each case represented by the solid line, the one-dot chain line, and the dotted line is as follows.

[0331] [Mathematical formula 9]

[0332]

[0333] Then, since ΔOBfO' and ΔB'BfB" are similar to each other, the following results are obtained.

[0334] [Formula 10]

[0335]

[0336] Furthermore, since ΔOBnO' and ΔB'BnB" are similar to each other, the following results are obtained.

[0337] [Mathematical formula 11]

[0338]

[0339] When the F value of the lens 21 is "F", the following formula is obtained from the definition.

[0340] [Mathematical formula 12]

[0341]

[0342] Therefore, when i, Δif, Δin, and d are deleted from the above, the diameter δ is expressed as follows.

[0343] [Mathematical formula 13]

[0344]

[0345] The focusing distance l of the lens 21 is rewritten as follows. Note that the focusing distance is the distance between the lens and the subject when the focus is on the subject, although it is also called an imaging distance. In the following, the focusing distance is unified.

[0346] [Formula 14]

[0347]

[0348] In addition to the distance s and the focusing distance l as described above, the diameter δ of blur when the focus of the lens 21 is offset relative to the actual distance s to the subject and the focusing distance is set to "l" can be calculated according to the focal length f of the lens 21 and the F value F of the lens 21 (Math 14). As described in the description of (Math 4), s>f, l>f, and the F value F is not 0. Therefore, if f>0, the denominator of (Math 14) is not 0.

[0349] The frequency characteristics of a circular blur with a diameter of δ can be calculated as follows by Fourier transform.

[0350] [Mathematical formula 15]

[0351]

[0352] However, since it is difficult to solve this integral, if we use a regular dodecagon to perform an approximation, as Fig.21 As shown by the dotted line in , the calculation can be performed using the following partial integration formula.

[0353] [Formula 16]

[0354]

[0355] Here, it is calculated that the inside of a circle with a diameter of δ is blurred with uniform brightness, but in reality, the light beam may be vignetted inside the lens and may not be circular. However, since many expensive lenses for photographing works of art are neatly rounded, this calculation expression is applied.

[0356] The LED panel 506 obtains contrast by minimizing the area of ​​the LEDs to widen the area of ​​the black matrix. Therefore, when the LED panel 506 is imaged, it appears as follows: Fig. 22 This figure shows that six LEDs are arranged as part of an image of the LEDs on the image sensor 20 when the camera 502 images the LED panel 506. It is assumed that the white squares in the figure are images of the LEDs.

[0357] Now, assuming that the actual LED spacing of the LED panel 506 is "Wp", the distance between the camera 502 and the LED panel 506 is "p", the focal length of the lens 21 is "f", the spacing of the LED image formed on the image sensor 20 of the camera 502 is expressed as the following (Formula 17) based on the above (Formula 3).

[0358] [Mathematical formula 17]

[0359]

[0360] The image with such an interval becomes a double image separated by the optical LPF 22 by the offset b. However, since actual pixels are arranged two-dimensionally, the pixels are actually divided vertically and horizontally to form a double image as shown in FIG. Fig.23 Quadruple images of top, bottom, left and right shown.

[0361] When this is blurred by diameter δ, the rectangular LED image becomes a packed shape. Then, due to some overlap of the blurred images of the LEDs, such as Fig.24 As shown, brightness non-uniformity occurs, which results in artifacts such as moiré fringes.

[0362] This is imaged by the image sensor 20 at a pixel spacing Ws, as shown Fig.25 The integrated value of the light entering the pixel range is the output of the pixel.

[0363] The result is as follows Fig.26 The brightness difference felt by each pixel is shown. Since the brightness varies slightly depending on the positional relationship between the LED image and the pixel of the image sensor, low-frequency moiré fringes that are longer than the interval between the LED images are observed.

[0364] How much the image of the LED is suppressed by the optical LPF 22, the integration effects in the pixel and the blur in the lens 21 will be considered.

[0365] Since the repetition frequency of the LED image is the inverse of the interval between the LED images, the angular frequency ω can be calculated by the following expression.

[0366] [Mathematical formula 18]

[0367]

[0368] Taking into account the interval (shift amount b) of the double images of the optical LPF 22, the amplitude Af of the optical LPF 22 can be calculated by the following (Math. 19) from the above (Math. 6).

[0369] [Mathematical formula 19]

[0370]

[0371] By using the interval Ws between the pixels of the image sensor 20 , the amplitude Ap due to the integration effect in the pixel can be calculated by (Formula 20) following (Formula 8) described above.

[0372] [Mathematical formula 20]

[0373]

[0374] The amplitude Ab caused by the blur of the lens 21 can be approximately calculated by the following equation using the F value F of the lens 21 and the focusing distance l of the lens 21 from the above (Math. 14) (Math. 15).

[0375] [Mathematical formula 21]

[0376]

[0377] The pixel interval Ws and the offset b are constants determined by the model of the camera 502, and the LED interval Wp of the LED panel 506 is a constant determined by the model of the LED panel 506. Therefore, the amplitude Af of the optical LPF 22 and the amplitude Ap generated due to the integration effect in the pixel vary according to the distance p between the camera 502 and the LED panel 506 and the focal length f of the lens 21.

[0378] The amplitude Ab due to blurring of the lens 21 becomes a function of ωδ (ω: angular frequency, δ: blurring diameter). From the above (Math. 18) and (Math. 21), ωδ can be calculated by the following expression.

[0379] [Mathematical formula 22]

[0380]

[0381] Therefore, the amplitude Ab also varies depending on the F value F of the lens 21 and the focusing distance l of the lens 21 in addition to the distance p between the camera 502 and the LED panel 506 and the focal length f.

[0382] The amplitude A of the LED image can be calculated as follows.

[0383] [Mathematical formula 23]

[0384] A=A f ·A p ·A b

[0385] If the amplitude A is sufficiently attenuated and becomes smaller than a certain threshold, it can be determined that no artifact such as moire fringes occurs.

[0386] The amplitude Ab caused by blur can be expressed as a function of (ωδ) as described below based on the above (Math. 21) and (Math. 22).

[0387] [Mathematical formula 24]

[0388]

[0389] When a graph of ωδ is plotted on the horizontal axis and amplitude Ab is plotted on the vertical axis, the graph does not look like Fig. 27 It decreases monotonically as in , and there are many local maxima and local minima.

[0390] When artifacts such as moire fringes are eliminated by moving the focus of the lens 21 at the imaging position, for example, if the amplitude Ab decreases or increases (local minimum and local maximum values ​​are repeated) as the focus ring is turned and the focus distance l of the lens 21 changes, it becomes a source of confusion. In addition, the calculation expression of the amplitude Ab is an approximate expression as described above, and is not a strict calculation.

[0391] Therefore, it is easier to use a replacement in the field with a function whose amplitude Ab decays monotonically according to ωδ.

[0392] When the estimated amplitude is larger, there is more room for warning about the occurrence of artifacts such as moire fringes. Fig.28 The solid line in the figure approximates the amplitude by tracking the local maximum.

[0393] Specifically, it is obtained by an approximate expression for each range from the case of ωδ<6.487887 to the case of 86.34829≤ωδ, as described below. Note that each range from 6.487887≤ωδ<17.240423 to 63.501086≤ωδ<86.34829 is approximated by a linear expression.

[0394] [Mathematical formula 25]

[0395]

[0396] Note that, empirically, if moire or the like is suppressed to 0.5% or less, the moire or the like is hardly visible, and thus it is considered that approximation to this extent can be performed.

[0397] Incidentally, considering the relative positional relationship between the camera 502 and the LED panel 506, the panel surface and the image sensor surface rarely face each other in parallel, and in practice, as shown in FIG. Fig.29 As shown, the camera 502 often photographs the LED panel 506 from an oblique direction. In this case, the distance p between the camera 502 and the LED panel 506 varies depending on the position in the panel.

[0398] Generally, since the subject (performer 510, etc.) in front of the LED panel 506 is focused, the focusing distance l of the lens 21 is shorter than the distance p. Therefore, in the range appearing in the camera 502, the amplitude Ab due to blur is expected to be maximized at the shortest distance p.

[0399] However, in the case of using a wide angle lens at a shallow angle, it can be considered that l>p, and the amplitude Af and the amplitude Ap do not decrease monotonically with respect to the increase in distance p. Therefore, the amplitude A of the image of the LED is not necessarily maximized at the shortest distance p.

[0400] Therefore, when the camera 502 is Fig.29 In the case where the LED panel 506 is photographed in the oblique direction shown, the amplitude A of the LED image is calculated by finely engraving the distance between the shortest distance p of the range appearing in the camera 502 and the longest distance p of the range appearing in the camera 502, and the calculated amplitude A is used to determine whether artifacts such as moiré fringes appear.

[0401] Even if the LED panel 506 is not a flat surface but a curved surface, similarly, the amplitude A of the LED image is calculated by finely engraving the shortest distance p of the range appearing in the camera 502 and the longest distance p of the range appearing in the camera 502, and the maximum value is used to determine whether artifacts such as moiré fringes appear.

[0402] Fig.30 It shows that Fig.16 The image through the lens 21. According to the focusing distance l of the lens 21, the focal length f of the lens 21 and the width w of the imaging surface of the image sensor 20, the range r appearing in the camera on the subject side is expressed by the following (Math 26) using the above (Math 3).

[0403] [Mathematical formula 26]

[0404]

[0405] therefore, Fig.30 The angle θ in can be calculated by the following expression.

[0406] Then, the range appearing in the camera 502 is the range of angles of ±θ from the optical axis of the lens 21 .

[0407] [Mathematical formula 27]

[0408]

[0409] like Fig.31 As shown, when the range appearing in the camera 502 is tilted at an angle of ±θ relative to the optical axis and the LED panel 506 is tilted at an angle of Φ (≤90°) relative to the optical axis, the relationship expression between the distance AD ​​along the optical axis between the camera 502 and the LED panel 506 and the lengths of the distances AB and AC at the ends of the range appearing in the camera 502 is as follows.

[0410] [Mathematical formula 28]

[0411] AD=AE+ED=AB cosθ+AB sinθ·cotφ

[0412]

[0413] AD=AF-DF=AC cosθ+AC sinθ·cotΦ

[0414]

[0415] When the foot of a vertical line drawn from the camera 502 to the LED panel 506 is "H", the shortest distance between the camera 502 and the LED panel 506 is AH.

[0416] [Mathematical formula 29]

[0417] AH=AD sinφ

[0418] In the case of Φ≥90°-θ, since the foot of the vertical line H falls within the range appearing in the camera 502, the shortest distance between the camera 502 and the LED panel is the distance AH.

[0419] Because Fig.16 and 20 As shown, the distance on the optical axis is used when calculating the blur diameter δ, so the distance AG from the foot of the perpendicular line H to the foot of the perpendicular line G to the optical axis is used. The distance AG is as follows.

[0420] [Math 30]

[0421] AG=AH sinφ=AD sin 2 φ

[0422] In the case of Φ<90°-θ°, the foot of the vertical line H is outside the range appearing in the camera 502, so that the shortest distance between the camera 502 and the LED panel 506 is the distance AB. Since the distance on the optical axis is used when calculating the diameter δ of the blur, the distance is the distance AE from point B to the foot of the vertical line E drawn on the optical axis.

[0423] [Mathematical formula 31]

[0424]

[0425] Then, the longest distance between the camera 502 and the LED panel 506 is the distance AC. Since the distance on the optical axis is used when calculating the diameter δ of blur, the distance is the distance AF from the point C to the foot F of the vertical line drawn on the optical axis.

[0426] [Math 32]

[0427]

[0428] If the size of the pixels is known, the distances on the image sensor 20 can be measured from the image.

[0429] like Fig.32 As shown, the angle θ formed by the line (dashed line in the figure) obtained by observing the point appearing on the image sensor 20 at a distance ds from the optical axis from the lens 21 and the optical axis is calculated using the above (Formula 3) according to the focusing distance l and focal length f of the lens 21 as follows (Formula 33).

[0430] [Mathematical formula 33]

[0431]

[0432] Here, it is assumed that an auxiliary camera 502S for distance measurement is provided in addition to the camera 502 for imaging. The auxiliary camera 502S may be an additional imaging optical system fixed to or built in the camera 502, or may be a separate camera.

[0433] like Fig.33 As shown, when the same point P is imaged by camera 502 and auxiliary camera 502S whose optical axes are parallel to each other and separated by a distance dc, it is assumed that the angles formed by lines AP and BP connecting the cameras (502, 502S) and point P and the optical axes of the cameras are α and β.

[0434] The distance PH from the point P to the foot H of a perpendicular line drawn on the line AB connecting the two cameras (502, 502S) can be calculated as follows.

[0435] [Math 34]

[0436] d c =AH+BH=PH tanα+PH tanβ

[0437]

[0438] The distance AP between the camera 502 and the point P and the distance BP between the auxiliary camera 502S and the point P may be calculated as follows.

[0439] [Mathematical formula 35]

[0440]

[0441] In this way, the distance to any point from the captured image can be measured using two cameras.

[0442] At point P, Fig.34 In the case of the camera 502 shown outside the optical axis, the angle is at Fig.33 , so it can be calculated as a negative value.

[0443] Note that although an example of using two cameras to perform distance measurement is described above, it is also possible to use a distance measurement sensor for measurement. In addition, the distance value can also be obtained from the information of the camera tracker 560.

[0444] Furthermore, when the position of the camera 502 is fixed, the distance can be actually measured at the imaging location using a tape measure or the like and input.

[0445] The concept of the determination process of the occurrence of moire fringes or the like has been described above. Fig.35 The flow of determination processing performed by the information processing device 70 according to the above-described concept is shown.

[0446] The information processing device 70 here is, for example, Fig.10 Specifically, in Fig.10 In the example of FIG. 5 , the information processing device 70 as the rendering engine 31 and the information processing device 70 as the rendering engine 520 are used. Another information processing device may be used.

[0447] Then, Fig.35 The processing of is an example of processing executed by the CPU 71 of the information processing device 70 according to a program for determining moire fringes or the like.

[0448] In step S501, the CPU 71 acquires the LED interval Wp of the LED panel 506. The LED interval Wp has a fixed value depending on the model. The CPU 71 can obtain information about the model of the LED panel 506 by communicating with other devices, operator input, etc., and can acquire the LED interval Wp according to the model by referring to the DB. Therefore, for example, a DB for storing the LED interval Wp corresponding to the model of the LED panel 506 is provided in the nonvolatile memory unit 74, the storage unit 79, etc.

[0449] Alternatively, it is also conceivable that the CPU 71 directly acquires the value of the interval Wp through input by an operator or the like or communication between devices.

[0450] In step S502 , the CPU 71 acquires the pixel interval Ws of the image sensor 20 of the camera 502 and the shift amount b of the optical LPF.

[0451] These values ​​are fixed depending on the model of the camera 502. The CPU 71 can obtain information about the model of the camera 502 by communicating with another device, an operator input, etc., and can acquire the pixel interval Ws and the offset b by referring to the DB according to the model. Therefore, for example, a DB for storing the LED interval Wp corresponding to the model of the camera 502 is provided in the nonvolatile memory unit 74, the storage unit 79, etc.

[0452] Alternatively, it is also conceivable that the CPU 71 directly acquires the values ​​of the pixel interval Ws and the offset b through input by an operator or the like or communication between devices.

[0453] In step S503 , the CPU 71 acquires the focal length f, the F value F, and the focusing distance l of the lens 21 of the camera 502 .

[0454] For example, the CPU 71 can acquire these values ​​by communicating with the camera 502 (lens barrel), or can acquire these values ​​by operator input regarding numerical values ​​displayed on the lens barrel or the like.

[0455] In step S504, the CPU 71 acquires the distance p between the camera 502 and the LED panel 506. For example, the above-mentioned shortest distance pmin and longest distance pmax are acquired.

[0456] The following describes how to use Fig.33 The process of measuring the distance along the optical axis of the camera 502 by using the auxiliary camera 502S installed with its optical axis parallel to the camera 502 and separated from the camera by a distance dc is shown as (i) to (v).

[0457] exist Fig.33, when α=0, the length of line segment AP is the distance along the optical axis of camera 502.

[0458] (i)

[0459] Pixel values ​​of several pixels near the center of the optical axis are acquired from the camera 502 .

[0460] (ii)

[0461] The pixel values ​​of all pixels of the auxiliary camera 502S are obtained.

[0462] Assume that the focal length of the lens 21 of the camera 502 is “fmain”, the focusing distance is “lmain”, and the pixel size of the image sensor 20 is “Wsmain”.

[0463] Assume that the focal length of the lens 21 of the auxiliary camera 502S is “fsub”, the focusing distance is “lsub”, and the pixel size of the image sensor 20 is “Wssub”.

[0464] The image of the camera 502 is changed from the above (Equation 3) to fmain / (lmain-fmain) multiplied by the actual size, and is sampled at intervals of the pixel size Wsmain.

[0465] The image of the auxiliary camera 502S is converted from (Equation 3) to fsub / (lsub-fsub) multiplied by the actual size, and is sampled at intervals of the pixel size Wssub.

[0466] Therefore, it is necessary to magnify or reduce the image acquired from the auxiliary camera 502S by the next magnification.

[0467] [Mathematical formula 36]

[0468]

[0469] Conversely, the partial image acquired in (i) above may be enlarged or reduced according to the auxiliary camera 502S. In this case, the inverse of the above magnification is used, and the focal length, focus distance, and pixel size used for the following (iii) and subsequent calculations are those of the auxiliary camera 502S.

[0470] (iii)

[0471] With respect to the pixel values ​​of the auxiliary camera 502S, a cross-correlation function with the pixel values ​​near the center of the optical axis acquired from the camera 502 is calculated from the upper left to the lower right of the screen, and the maximum coordinates are searched.

[0472] (iv)

[0473] The distance between the pixel position of the optical axis center of the auxiliary camera 502S and the coordinates where the cross-correlation function is maximized is obtained, and the distance ds is obtained by multiplying by the pixel size Wsmain of the image sensor 20 .

[0474] (v)

[0475] Substitute α and β in the above (Equation 34) and (Equation 35) as follows.

[0476] [Mathematical formula 37]

[0477] α=0,

[0478] Substitute the above α and β into the calculation expression of the line segment AP in (Math. 35), and the distance pa from the optical axis of the camera 502 to the LED panel 506 is obtained by the following calculation formula.

[0479] [Math 38]

[0480]

[0481] Similarly, a distance pb from the optical axis of the camera 502 to a point on the LED panel 506 that is visible at the largest possible angle θ′ within a range visible from both the camera 502 and the auxiliary camera 502S is obtained in processes (i) to (v).

[0482] The distance pb is expressed as follows by the angle Φ formed by the optical axes of the LED panel 506 and the camera 502.

[0483] [Math 39]

[0484]

[0485] Therefore, the angle Φ can be calculated by the following expression.

[0486] [Math 40]

[0487]

[0488] Assuming that the range visible from the camera 502 is the range of angles ±θ relative to the optical axis, the shortest distance pin and the longest distance pmax between the camera 502 and the LED panel 506 can be calculated by the following expressions. Fig.31 In the embodiment, the angle Φ is 90° or less, but even in the case of an angle exceeding 90°, the maximum value and the minimum value can be calculated by the same formula. However, considering the possibility of error in the direction of measuring the angle, the angle is calculated as follows.

[0489] [Mathematical formula 41]

[0490]

[0491] For example, in the above process, Fig.35 In step S504 , the CPU 71 acquires the shortest distance pmin and the longest distance pmax between the camera 502 and the LED panel 506 .

[0492] However, the shortest distance pmin and the longest distance pmax may be detected by a distance measuring sensor, or may be acquired based on information of the camera tracker 560. In addition, it may be input by an operator.

[0493] In step S505, the CPU 71 substitutes the shortest distance pmin into the distance p as a variable for calculation. In addition, "0" is substituted into the maximum amplitude value Amax.

[0494] In steps S506 to S510 , the CPU 71 executes processing of calculating the amplitude characteristic while gradually increasing the value of the distance p until the distance p exceeds the longest distance pmax.

[0495] First, in step S506 , the CPU 71 calculates the amplitude characteristic (amplitude A).

[0496] Therefore, the CPU 71 calculates the angular frequency ω according to the above (Equation 18).

[0497] Furthermore, the CPU 71 calculates the amplitude characteristic (amplitude Af) obtained by the optical LPF 22 and the amplitude characteristic (amplitude Ap) obtained by the integration effect in the pixel by the above-mentioned expressions (Math. 19) and (Math. 20).

[0498] Furthermore, the CPU 71 calculates the diameter δ of blur due to focus blur by the above (Equation 14).

[0499] Then, the CPU 71 calculates the amplitude Ab due to blurring based on the value of (ωδ) in the above-mentioned (Math. 25).

[0500] The amplitude characteristic (amplitude A) is the product of the amplitudes (A=Af·Ap·Ab).

[0501] First, since the distance p is the shortest distance pmin, the amplitude A at the shortest distance pmin is obtained.

[0502] In step S507 , when the calculated amplitude A is larger than the maximum amplitude value Amax at this time, the CPU 71 replaces the maximum amplitude value Amax with the amplitude A calculated this time in step S508 .

[0503] In step S509, the CPU 71 gradually increases the distance p.

[0504] In step S510, the CPU 71 determines whether the distance p exceeds the longest distance pmax, and if not, returns to step S506. Then, the amplitude A is calculated by gradually increasing the distance p.

[0505] If the distance p exceeds the longest distance pmax in step S510, the process proceeds from step S510 to step S511.

[0506] Therefore, in steps S506 to S510 , for each distance having a certain width within the range from the shortest distance pmin to the longest distance pmax, the amplitude A is obtained, and the maximum value thereof is set as the maximum amplitude value Amax.

[0507] In step S511 , the CPU 71 compares the maximum amplitude value Amax with the threshold value Ath.

[0508] When the maximum amplitude value Amax does not exceed the threshold value Ath, the CPU 71 determines that moire fringes or the like do not occur, and ends the determination processing.

[0509] If the maximum amplitude value Amax exceeds the threshold value Ath, the CPU 71 determines that moire or the like occurs, and executes a moire occurrence determination process in step S512.

[0510] For example, information such as data and a logo is set as the occurrence of moire fringes. This information can be stored in a storage medium in association with video data such as the shot video vC and the simulated video vSM, camera setting information CS, etc., or can be transmitted to, for example, the cloud server 41, etc. In addition, the video data as the shot video vC can be added as metadata.

[0511] For example, in Fig.14 Step S402 or Fig.15 In step S71, execute Fig.35 processing, and thereafter, can be Fig.14 Step S404 or Fig.15 In step S73, moire fringe warning control is performed.

[0512] Note that it is conceivable that after field adjustments, Fig.35 The threshold value Ath used in step S511 of . The contrast may be increased or decreased when editing, and also varies depending on the environment in which the finished work is viewed. The threshold value Ath may be determined to be a value such that moire fringes or the like become inconspicuous in the final form.

[0513] Incidentally, the LED wall 505 may appear to be mounted on a curved surface, but such an LED wall 505 is actually also formed by combining Fig.36A small flat LED panel 506 is shown for installation.

[0514] Therefore, if the above calculations are performed for each LED panel 506, the occurrence of artifacts such as moire fringes can be predicted.

[0515] Furthermore, although moire fringes are only an obstacle for viewing applications, e.g. Fig.37 As shown, the slight deviation of the position of the LED in the joint of the LED panel 506 appears to be amplified by the moiré fringe. By utilizing this, when setting the LED wall 505, it is very useful for adjusting the joint position of the LED panel 506.

[0516] That is, when set, as shown in the figure, the LED wall 505 is imaged by the camera 502, for example, and observed through the output monitor 503. Then, due to the moiré fringes, the deviation of the LED panel 506 appears clearly. This is because, in the case where the intervals between the images of the LEDs formed on the image sensor 20 are slightly different from the intervals between the pixels of the image sensor 20, the slight deviation appears to be magnified, similar to the relationship between the main scale and the secondary scale of a caliper.

[0517] In consideration of such use, it is preferable to perform the above-described determination processing so as to purposefully search for conditions for generating moire fringes.

[0518] Fig.38 Another example of the determination process is shown.

[0519] In the above Fig.35 In the processing of , the maximum amplitude value Amax is obtained between the shortest distance pmin and the longest distance pmax, and it is compared with the threshold value Ath, thereby simply determining whether moire fringes appear. On the other hand, Fig.38 This is an example of determining in which distance range moire fringes appear in the range from the shortest distance pmin to the longest distance pmax. This also determines the range in which moire fringes or the like appear in the plane of the captured video vC.

[0520] exist Fig.38 In the step S501 to S504, Fig.35 The steps are similar to those in .

[0521] exist Fig.38 In step S520, the CPU 71 substitutes the shortest distance pmin into the distance p as a calculation variable.

[0522] Furthermore, the CPU 71 substitutes "0" into the flag FM indicating the occurrence of moire fringes. The flag FM=0 indicates that no moire fringes or the like occur, and the flag FM=1 indicates that moire fringes or the like occur.

[0523] In addition, the CPU 71 substitutes the longest distance pmax into the variable PMmin, and substitutes the shortest distance pmin into the variable PMmax. The variable PMmin is a variable representing the shortest distance within the range where the moire fringe appears, and the initial value is set to the longest distance pmax. In addition, the variable PMmax is a variable representing the longest distance within the range where the moire fringe appears, and its initial value is set to the shortest distance pmin.

[0524] After the initial value is set as described above, in steps S521 to S529, the CPU 71 executes processing for calculating the amplitude characteristic while gradually increasing the value of the distance p until the distance p exceeds the longest distance pmax.

[0525] First, in step S521, the CPU 71 calculates the amplitude characteristic (amplitude A) at the distance p at this time. The calculation of the amplitude A is similar to Fig.35 Step S506 in .

[0526] In step S522, the CPU 71 compares the calculated amplitude A with a threshold value Ath. The threshold value Ath is Fig.35 The threshold value Ath used in step S511 is used, and when the threshold value Ath is exceeded, it is determined that moire fringes or the like have occurred.

[0527] When the amplitude A does not exceed the threshold value Ath, the CPU 71 gradually increases the distance p in step S528, further determines whether the distance p exceeds the longest distance pmax in step S529, and returns to step S521 when the distance p does not exceed the longest distance pmax.

[0528] If the amplitude A exceeds the threshold value Ath in step S522 , the CPU 71 proceeds to step S523 , and sets the flag FM=1.

[0529] In step S524, the CPU 71 determines whether the distance p is equal to or smaller than a variable PMmin.

[0530] If p≤PMmin, then in step S525, the CPU 71 substitutes the distance p into the variable PMmin. Since the initial value of the variable PMmin is the longest distance pmax, and the distance p gradually increases from the shortest distance pmin, the distance p when A>Ath is initially satisfied is substituted into the variable PMmin. Then, the process proceeds to step S526.

[0531] If p≤PMmin is not satisfied, the CPU 71 skips step S525 and proceeds to step S526.

[0532] In step S526, the CPU 71 determines whether the distance p is equal to or greater than a variable PMmax.

[0533] If p ≥ PMmax, then in step S527, the CPU 71 substitutes the distance p into the variable PMmax. Since the initial value of the variable PMmax is the shortest distance pmin, and the distance p gradually increases from the shortest distance pmin, the variable PMmax is updated to the distance p at each time point in the period in which the distance p gradually increases, and after A>Ath is initially satisfied, the state of A>Ath continues. Then, the process proceeds to step S528.

[0534] If p ≥ PMmax is not satisfied, the CPU 71 skips step S527 and proceeds to step S528.

[0535] When the above-mentioned processing is repeated until P>Pmax, the CPU 71 proceeds from step S529 to step S530. At this time, if the flag FM=0, it is determined that moire fringes or the like do not occur.

[0536] On the other hand, when the flag FM=1, it is determined that moire or the like occurs. In this case, the CPU 71 proceeds to step S531 and executes a moire occurrence determination process.

[0537] For example, data indicating the occurrence determination and occurrence range of moire fringes and information such as flags are set. The range in which moire fringes occur is the range from the distance represented by the variable PMmin to the distance represented by the variable PMmax. In other words, moire fringes and the like appear in the in-plane area of ​​the image corresponding to the distance range.

[0538] The information of the moire occurrence determination and the occurrence range may be sent to, for example, the cloud server 41 or the like in association with the video data, the camera setting information CS, etc. Furthermore, video data such as the captured video vC and the simulation video vSM may be added as metadata.

[0539] <5. Conclusion and Modification>

[0540] According to the above-described embodiment, the following effects can be obtained.

[0541] The information processing device 70 used as the rendering engine 31 and 520 according to the present embodiment includes a determination unit (36, 520d), which performs a determination process for determining whether artifacts appear in the captured video vC by using the interval Wp of each LED serving as a light-emitting element of the LED panel 506, the pixel interval Ws of the image sensor 20 of the camera 502, the characteristics of the optical LPF 22 (offset b), the focal length f, F value F and focusing distance l of the lens 21, and the distance p between the LED panel 506 and the camera 502.

[0542] As a result, it is possible to predict whether an artifact such as moire fringes will occur based on the numerical values ​​that can be obtained at the imaging site or the inspection stage before imaging. Then, artifacts can be predicted at the preliminary inspection stage of imaging or the preparation stage of the imaging site, thereby making the work at the time of imaging efficient.

[0543] Because artifacts such as moire, rastering, color shifts, etc. can be determined.

[0544] In addition, the intervals between LEDs are used as an element for determination on the LED panel 506 side, and are independent of the video content. Therefore, even in a state where no image is displayed on the LED wall 505, for example, moiré determination based on the camera position can be performed.

[0545] In this embodiment, the determination unit (36, 520d) obtains the amplitude A of the image of the light emitting element on the image sensor 20, and compares the obtained amplitude A with the threshold value Ath to determine whether an artifact occurs (see Fig.35 and 38 ).

[0546] For example, when some blurred images of the subject overlap each other, uneven brightness occurs, which causes artifacts such as moire fringes. Therefore, the amplitude of the image of the actual LED of the LED panel 506 on the image sensor 20 is obtained. When the amplitude is large, it can be determined that moire fringes or the like have occurred.

[0547] In this embodiment, an example is described in which the determination unit (36, 520d) obtains the shortest distance pmin and the longest distance pmax between the LED panel 506 and the camera 502, and uses the shortest distance pmin and the longest distance pmax to perform determination processing (refer to Fig.31 , Fig.35 , Fig.38 ).

[0548] The LED panel 506 and the camera 502 are not necessarily in an opposing positional relationship, but face each other in an oblique direction in many cases. In this case, the shortest distance and the longest distance are generated as the distance from the camera 502 to the display surface of the LED wall 505 (LED panel 506). By performing moire fringe determination within this range, the occurrence of moire fringe and the like can be appropriately determined regardless of the state of the position or direction of the camera 502.

[0549] In this embodiment, an example is described in which the determination unit (36, 520d) determines the range in which the artifacts are generated in the captured video vC (refer to Fig.38 ).

[0550] That is, it is determined where moire fringes appear in the range within the plane of the captured video v C. Such a determination result can also be used for various setting changes, imaging preparations, and the like.

[0551] according to Fig.38 The processing shown can determine in which range between the shortest distance pmin and the longest distance pmax the moire fringes appear.

[0552] In this embodiment, an example has been described in which the determination unit (36, 520d) performs warning processing in response to determining that an artifact occurs (see Fig.14 and 15 ).

[0553] For example, by performing warning processing on a worker or the like who is watching at the pre-visualization stage, measures can be taken so that moire fringes or the like do not appear in advance preparation. Also, similarly, measures can be taken for a case where a warning is issued to a worker at the imaging preparation stage.

[0554] Furthermore, in the case where it is determined that moire fringes have occurred during imaging, by performing display or the like of issuing a warning to the camera operator 512 or the like, the camera operator 512 can immediately take measures such as blurring the focus.

[0555] In the present embodiment, an example has been described in which the determination unit (36, 520d) performs a process of associating the determination result information of the determination process with an item. In this case, the item is a video content production item. That is, it is a video production item generated by the imaging performed in the imaging system 500. Note that associating the determination result information with the item includes information corresponding to the item, information associated with the video of the item, etc. Associating with the video includes associating the determination result information as management information corresponding to the entire video content file, etc., and associating the determination result information as information corresponding to a frame of the video.

[0556] The association process includes a process of storing the data in the storage medium in an associated state and a process of transmitting the data to an external device.

[0557] For example, data, flags, etc. indicating the determination result of the occurrence of moire fringes are stored in a storage medium in association with the video. Alternatively, data, flags, etc. indicating the determination result of the occurrence of moire fringes are sent to the cloud server 41 and stored therein according to the camera setting. As a result, for example, the moire fringes determination result in the pre-visualization stage can be referred to at the time of imaging preparation, etc.

[0558] In this embodiment, a situation is described in which the determination unit (36, 520d) obtains all or part of the values ​​of the LED interval Wp, the pixel interval Ws of the image sensor 20, the offset b of the optical LPF 22, the focal length f, the F value F, the focusing distance l, and the distance p based on information received through communication with other devices.

[0559] For example, the determination units 36 and 520d acquire the values ​​required for the moire fringe determination by communicating with the LED panel 506, the camera 502, or an information processing device on the system. Alternatively, the determination units 36 and 520d may acquire the model of the LED panel 506 or the camera 502 through communication, and obtain each value according to the model.

[0560] As a result, the determination units 36 and 520d can automatically obtain necessary numerical values ​​and perform moire fringe determination.

[0561] In this embodiment, the determination unit (36, 520d) compares the maximum amplitude value Amax of the LED image on the image sensor 20 with the threshold value Ath among a plurality of distance values ​​between the shortest distance pmin and the longest distance pmax, and determines that an artifact has occurred when the maximum amplitude value Amax exceeds the threshold value Ath (see Fig.35 ).

[0562] As a result, relatively simple processing can be used to determine whether artifacts have occurred in the actual distance relationship.

[0563] The imaging system 500 of the present embodiment is a system that captures a video of an LED panel 506 that displays a virtual video (background video vB) obtained by rendering using a 3D model with a camera 502 .

[0564] Then, the rendering engine 31 as the information processing device 70 according to the present embodiment includes a rendering unit 32 and a video processing unit 33 in addition to the determination unit 36. The rendering unit 32 performs rendering to generate a virtual video using the 3D model used in the imaging system 500. The video processing unit 33 generates a simulated video vSM for the virtual video generated by the rendering unit 32 by using a processing parameter that realizes the characteristics of brightness or color at the time of imaging of the camera 502 used in the imaging system 500.

[0565] That is, this is a configuration in the case where the present technology is applied to the rendering engine 31 in the pre-visualization system 700 and the rendering engine 31 includes the determination unit 36 ​​in addition to the rendering unit 32 and the video processing unit 33. Therefore, in the pre-visualization stage, it is possible to determine whether moire fringes or the like appear in the video under the same conditions as in the case of actual imaging.

[0566] Furthermore, the rendering engine 520 as the information processing device 70 according to the present embodiment includes a rendering unit 520 a .

[0567] That is, this is a configuration in the case where the present technology is applied to the rendering engine 520 in the imaging system 500 and the rendering engine 520 includes the determination unit 520d. Therefore, it is possible to determine whether moire fringes or the like occur in the imaging preparation or imaging stage.

[0568] Note that in the imaging system 500 , a determination unit that determines moiré fringes, such as the determination unit 520 d , may be provided in an information processing device other than the rendering engine 520 .

[0569] For example, the determination unit may be provided in the camera 502, the display controller 590, a control unit on the operation monitor 550 side, or Figure 7 In various control and computing units (not shown).

[0570] The program of the embodiment is used to make a processor such as a CPU, a DSP, etc. or a device including a processor execute the above-mentioned Fig.14 , 15 , 35 and 38 as shown in the processing procedures.

[0571] That is, the program of the present embodiment is a program for causing the information processing device 70 to perform processing for determining whether artifacts appear in the captured video vC by using the interval Wp of the LEDs serving as light-emitting elements, the pixel interval Ws of the image sensor 20 of the camera 502, the offset b of the optical LPF 22, the focal length F, the F value F and the focusing distance l of the lens 21, and the distance p between the LED panel 506 and the camera 502.

[0572] Using such a program, the information processing apparatus 70 that executes the determination processing of the above-described embodiment can be realized by various computer means.

[0573] Such a program may be recorded in advance in a HDD as a recording medium built into a device such as a computer apparatus, a ROM in a microcomputer having a CPU, or the like. In addition, such a program may be temporarily or permanently stored (recorded) in a removable recording medium such as a floppy disk, a compact disk read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disk (DVD), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such a removable recording medium may be provided as a so-called software package.

[0574] Furthermore, such a program may be installed from a removable recording medium into a personal computer or the like, or may be downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0575] In addition, such a program is suitable for widely providing an information processing device 70 that performs a certain process, such as an information processing device 70 used as the rendering engine 31 and 520 according to the present embodiment. For example, by downloading the program to a personal computer, a communication device, a portable terminal device such as a smartphone or a tablet, a mobile phone, a game device, a video device, a personal digital assistant (PDA), etc., these devices can be used as the information processing device 70 of the present disclosure.

[0576] Note that the effects described in this specification are merely examples and not limitations, and other effects may be provided.

[0577] Note that the present technology may also have the following configurations. (1)

[0579] An information processing device, comprising:

[0580] A determination unit that performs determination processing for determining whether an artifact occurs in a video captured by a camera by using the intervals between light-emitting elements of the display, the pixel intervals between image sensors of the camera, the characteristics of the optical low-pass filter of the camera, the focal length, F value and focusing distance of the lens of the camera, and the distance between the display and the camera in an imaging system that captures a video of the display using a camera. (2)

[0582] The information processing device according to (1), wherein

[0583] In the determination process, the determination unit obtains the amplitude of the image of the light emitting element on the image sensor, and compares the obtained amplitude with a threshold value to determine whether an artifact has occurred. (3)

[0585] The information processing device according to (1) or (2), wherein

[0586] The determination unit acquires the shortest distance and the longest distance between the display and the camera, and performs determination processing using the shortest distance and the longest distance. (4)

[0588] The information processing device according to any one of (1) to (3), wherein

[0589] In the determination process, the determination unit determines a range in which an artifact appears in the captured video. (5)

[0591] The information processing device according to any one of (1) to (4), wherein

[0592] The determination unit performs a warning process in response to determining that the artifact has occurred. (6)

[0594] The information processing device according to any one of (1) to (5), wherein

[0595] The determination unit performs a process of associating determination result information of the determination process with the item. (7)

[0597] The information processing device according to any one of (1) to (6), wherein

[0598] The determination unit performs a process of storing the determination result information in the storage medium in association with the item. (8)

[0600] The information processing device according to any one of (1) to (7), wherein

[0601] The determination unit performs a process of transmitting the determination result information to an external device in association with the item. (9)

[0603] The information processing device according to any one of (1) to (8), wherein

[0604] The determination unit obtains all or some of the values ​​of the interval between light-emitting elements of a display, the pixel interval between image sensors of a camera, the characteristics of an optical low-pass filter of the camera, the focal length of a lens of the camera, the F value, the focusing distance, and the distance between the display and the camera based on information received through communication with another device. (10)

[0606] The information processing device according to any one of (1) to (9), wherein

[0607] The determining unit is configured to:

[0608] Get the shortest and longest distances between the display and the camera; and

[0609] The maximum value of the amplitude of the image of the light emitting element on the image sensor in a plurality of distance values ​​from the shortest distance to the longest distance is compared with a threshold value, and the occurrence of an artifact is determined if the maximum value exceeds the threshold value. (11)

[0611] The information processing device according to any one of (1) to (10), wherein

[0612] The imaging system is a system that uses a camera to capture a video of a display that displays a virtual video obtained by rendering using a 3D model, and

[0613] The imaging system comprises

[0614] a rendering unit that performs rendering using the 3D model to generate a virtual video, and

[0615] A video processing unit generates a simulated video for the virtual video generated by the rendering unit by using a processing parameter realizing a brightness or color characteristic of a camera used in the imaging system at the time of imaging. (12)

[0617] The information processing device according to any one of (1) to (10), wherein

[0618] An imaging system is a system that uses a camera to capture video of a display that displays a virtual video obtained by rendering using a 3D model, and

[0619] The imaging system includes a rendering unit that performs rendering using the 3D model and generates a virtual video to be displayed on a display.

[0620] (13) An information processing method that causes an information processing device to execute the following processing

[0621] A determination process is performed to determine whether artifacts appear in a video captured by a camera by using the intervals between light-emitting elements of the display, the pixel intervals between image sensors of the camera, the characteristics of the optical low-pass filter of the camera, the focal length, F value and focusing distance of the lens of the camera, and the distance between the display and the camera in an imaging system that uses a camera to capture a video of a display.

[0622] (14) A program that causes an information processing device to execute the following processing

[0623] In an imaging system that uses a camera to capture video of a display, it is determined whether artifacts appear in the video captured by the camera by using the spacing between the display's light-emitting elements, the pixel spacing between the camera's image sensors, the characteristics of the camera's optical low-pass filter, the focal length, F value and focusing distance of the camera's lens, and the distance between the display and the camera.

[0624] Reference numerals list

[0625] 20 Image Sensor

[0626] 21 lenses

[0627] 22 optical LPF

[0628] 31 Rendering Engine

[0629] 32 rendering units

[0630] 33 Video Processing Unit

[0631] 35 Adjustment operation unit

[0632] 36 Determine the unit

[0633] 37-position operating unit

[0634] 38 communication units

[0635] 70Information processing equipment

[0636] 71CPU

[0637] 500 Imaging System

[0638] 502 Camera

[0639] 502S auxiliary camera

[0640] 505 LED wall

[0641] 506 LED Panel

[0642] 520 Rendering Engine

[0643] 520D Determination Unit

Claims

1. An information processing device, comprising: A determination unit that performs determination processing for determining whether an artifact occurs in a video captured by a camera by using, in an imaging system that captures a video of a display using a camera, the intervals between light-emitting elements of the display, the intervals between pixels of an image sensor of the camera, the characteristics of an optical low-pass filter of the camera, the focal length, F value and focusing distance of a lens of the camera, and the distance between the display and the camera.

2. The information processing device according to claim 1, wherein In the determination process, the determination unit obtains the amplitude of the image of the light emitting element on the image sensor, and compares the obtained amplitude with a threshold value to determine whether an artifact has occurred.

3. The information processing device according to claim 1, wherein The determination unit acquires the shortest distance and the longest distance between the display and the camera, and performs determination processing using the shortest distance and the longest distance.

4. The information processing device according to claim 1, wherein In the determination process, the determination unit determines a range in which an artifact appears in the captured video.

5. The information processing device according to claim 1, wherein The determination unit performs a warning process in response to determining that the artifact has occurred.

6. The information processing device according to claim 1, wherein The determination unit performs a process of associating determination result information of the determination process with the item.

7. The information processing device according to claim 1, wherein The determination unit performs a process of storing the determination result information in the storage medium in association with the item.

8. The information processing device according to claim 1, wherein The determination unit performs a process of transmitting the determination result information to an external device in association with the item.

9. The information processing device according to claim 1, wherein The determination unit obtains all or some of the values ​​of the interval between light-emitting elements of a display, the pixel interval between image sensors of a camera, the characteristics of an optical low-pass filter of the camera, the focal length of a lens of the camera, the F value, the focusing distance, and the distance between the display and the camera based on information received through communication with another device.

10. The information processing device according to claim 1, wherein The determining unit is configured to: Get the shortest and longest distances between the display and the camera; and The maximum value of the amplitude of the image of the light emitting element on the image sensor in a plurality of distance values ​​from the shortest distance to the longest distance is compared with a threshold value, and the occurrence of an artifact is determined if the maximum value exceeds the threshold value.

11. The information processing device according to claim 1, wherein The imaging system is a system that uses a camera to capture a video of a display that displays a virtual video obtained by rendering using a 3D model, and The imaging system comprises a rendering unit that performs rendering using the 3D model to generate a virtual video, and A video processing unit generates a simulated video for the virtual video generated by the rendering unit by using a processing parameter realizing a brightness or color characteristic of a camera used in the imaging system at the time of imaging.

12. The information processing device according to claim 1, wherein An imaging system is a system that uses a camera to capture video of a display that displays a virtual video obtained by rendering using a 3D model, and The imaging system includes a rendering unit that performs rendering using the 3D model and generates a virtual video to be displayed on a display.

13. An information processing method causing an information processing device to perform the following processing A determination process is performed to determine whether artifacts appear in a video captured by a camera by using the intervals between light-emitting elements of the display, the pixel intervals between image sensors of the camera, the characteristics of the optical low-pass filter of the camera, the focal length, F value and focusing distance of the lens of the camera, and the distance between the display and the camera in an imaging system that uses a camera to capture a video of a display.

14. A program that causes an information processing device to execute the following processing In an imaging system that uses a camera to capture video of a display, it is determined whether artifacts appear in the video captured by the camera by using the spacing between the display's light-emitting elements, the pixel spacing between the camera's image sensors, the characteristics of the camera's optical low-pass filter, the focal length, F value and focusing distance of the camera's lens, and the distance between the display and the camera.

Images