Information processing device, information processing method, and recording medium

Abstract

The present technology relates to: an information processing device with which it is possible to easily prevent the generation of moiré; an information processing method; and a recording medium. The information processing device of the present technology comprises: a capture control unit that refers to a database which records, in association with each other, a reference capture parameter relating to capture by a camera for capturing an image of a display and information indicating the presence or absence of moiré in a captured-image captured by the camera, and that sets, as a current capture parameter of the camera, a reference capture parameter with which information indicating that there is no moiré is associated. The present technology is applicable to, for example, a capture system that is used for virtual production in which a subject is captured, with an image displayed on an LED display serving as a background.

Description

Information processing device, information processing method, and recording medium

[0001] The present technology relates to an information processing device, an information processing method, and a recording medium, and more particularly to an information processing device, an information processing method, and a recording medium that can easily avoid the occurrence of moire.

[0002] Recently, virtual production (in-camera VFX), a filming technique that uses large LED (Light Emitting Diode) displays, has become popular in filming movies and TV dramas.

[0003] In virtual production, an image displayed on an LED display is used as the background, and a subject placed in front of the LED display is used as the foreground. In virtual production, the background image is rendered based on the camera position, camera orientation, and lens profile (see, for example, Patent Document 1).

[0004] Special Publication No. 2022-554415

[0005] When capturing an image displayed on an LED display, moire can occur in the captured image due to a slight misalignment between the pixel pitch of the LED display and the pixel pitch of the camera. Conventionally, the photographer would manually adjust the shooting parameters when capturing the image to avoid moire.

[0006] There are multiple shooting parameters that can be adjusted to avoid moiré, but there are limitations to manually adjusting multiple shooting parameters. Also, because the adjusted shooting parameters are not recorded, even if you take a photo under the same shooting conditions at a later date, you will need to adjust the shooting parameters again to avoid moiré.

[0007] The present technology has been made in view of such circumstances, and makes it possible to easily avoid the occurrence of moire.

[0008] An information processing device according to a first aspect of the present technology includes a shooting control unit that refers to a database in which reference shooting parameters for shooting by a camera that shoots a display are recorded in association with information indicating the presence or absence of moiré in the captured image shot by the camera, and sets the reference shooting parameters associated with information indicating the absence of moiré as the current shooting parameters of the camera.

[0009] In an information processing method according to a first aspect of the present technology, an information processing device refers to a database in which reference shooting parameters relating to shooting by a camera that shoots a display are recorded in association with information indicating the presence or absence of moiré in an image captured by the camera, and sets the shooting parameters associated with information indicating the absence of moiré as the current shooting parameters of the camera.

[0010] The storage medium of the first aspect of the present technology records a program for executing a process of referring to a database in which reference shooting parameters for shooting by a camera that shoots a display are recorded in association with information indicating the presence or absence of moiré in the captured image captured by the camera, and setting the shooting parameters associated with information indicating the absence of moiré as the current shooting parameters of the camera.

[0011] In a first aspect of the present technology, the shooting parameters for photographing a display using a camera are set by referring to a database in which the shooting parameters are recorded in association with information indicating the presence or absence of moiré in the captured image photographed by the camera.

[0012] 1 is a diagram illustrating an overview of an imaging system to which the present technology is applied. FIG. 1 is a diagram illustrating an example of an image captured by a camera. FIG. 2 is a diagram illustrating an example of a state during imaging. FIG. 3 is a diagram illustrating the cause of moiré generation. FIG. 4 is a diagram illustrating a camera position. FIG. 5 is a diagram illustrating camera setting values. FIG. 6 is a diagram illustrating an example of focus adjustment to avoid the occurrence of moiré. FIG. 7 is a flowchart illustrating actual imaging processing performed by a conventional imaging system. FIG. 8 is a block diagram illustrating an example configuration of an imaging system. FIG. 9 is a flowchart illustrating imaging preparation processing performed by an imaging system of the present technology. FIG. 10 is a diagram illustrating an example list showing combinations of imaging parameters. FIG. 11 is a diagram illustrating an example of a grid pattern and moiré. FIG. 11 is a flowchart illustrating actual imaging processing performed by an imaging system of the present technology. FIG. 12 is a flowchart illustrating actual imaging processing when imaging preparation processing is not performed. FIG. 13 is a block diagram illustrating an example configuration of computer hardware.

[0013] Hereinafter, embodiments of the present technology will be described in the following order: 1. Overview of the imaging system 2. Configuration and operation of the imaging system 3. Modified examples

[0014] 1. Overview of Imaging System FIG. 1 is a diagram illustrating an overview of an imaging system 1 to which the present technology is applied.

[0015] 1 is a system used for shooting, for example, virtual production (in-camera VFX). The shooting system 1 is composed of a camera 11, a wall-mounted LED (Light Emitting Diode) display 12, an information processing device (not shown) that controls the camera 11, and a control device (not shown) that controls the LED display 12.

[0016] The LED display 12 is placed in a studio or the like. The LED display 12 displays, for example, an image of a virtual space created using CG (Computer Graphics). The photographer P1 uses the camera 11 to photograph the subject, the motorcycle M1, with the image displayed on the LED display 12 as the background. Hereinafter, the image displayed on the LED display 12 will be referred to as the background image.

[0017] FIG. 2 is a diagram showing an example of an image captured by the camera 11. As shown in FIG.

[0018] 2, the image captured by the camera 11 shows the motorcycle M1 as if it were present in a virtual space reflected in the background image. In this way, by using the photography system 1, the photographer P1 can take an image in a studio in which the space reflected in the background image appears to extend behind the motorcycle M1.

[0019] FIG. 3 is a diagram showing an example of a state during shooting.

[0020] As shown in FIG. 3, during shooting, for example, the entire background image is displayed on the entire LED display 12.

[0021] 3, when the camera 11 captures an image from the right side of the motorcycle M1 located in the center, a portion of the background image is displayed superimposed on the entire background image in the image capture area A1, which is an area on the LED display 12 included in the image capture range of the camera 11. In the image capture area A1, for example, an image of the portion of the background that would be displayed when an image is captured from the right side of the motorcycle M1 in the virtual space reflected in the background image is cut out from the entire background image and displayed.

[0022] 3, when the camera 11 captures an image from the left side of the motorcycle M1 located in the center, a portion of the background image is displayed in the image capture area A1 so that it is superimposed on the entire background image. For example, in the image capture area A1, an image of the portion of the background that would be displayed when the image is captured from the left side of the motorcycle M1 in the virtual space reflected in the background image is cut out from the entire background image.

[0023] In this way, when taking a photograph, the photographer P1 takes a photograph while successively changing the position, orientation, focal length, focus, etc. of the camera 11. The focal length, focus, etc. of the camera 11 are also changed by the information processing device.

[0024] The control device tracks the camera work (position and attitude) of the camera 11 and controls the position on the LED display 12 where the background image is displayed according to the camera work of the camera 11. Specifically, the control device detects the shooting area based on the position, attitude, and settings of the camera 11 and controls the background image to be displayed in the shooting area.

[0025] In FIG. 3, for ease of explanation, the imaging area A1 is shown surrounded by a thick line, but in reality, the line surrounding the imaging area is not displayed.

[0026] When a photograph is taken with the background image displayed on the LED display 12 as the background, moire may occur in the photographed image.

[0027] FIG. 4 is a diagram illustrating the cause of moire.

[0028] As shown in FIG. 4, the background image displayed on the LED display 12 is projected onto a virtual camera plane CP, and an image is taken using the background image as the background.

[0029] For example, when a uniform background image is displayed on the LED display 12, each pixel Pi1 constituting the LED display 12 emits light of the same color, as indicated by the tip of arrow #1 in Fig. 4. Hereinafter, the distance between the centers of adjacent pixels Pi will be referred to as the pixel pitch of the LED display 12.

[0030] Furthermore, the distance between the centers of adjacent pixels Pi11 among the multiple pixels Pi11 that make up the sensor provided in the camera 11 is referred to as the pixel pitch of the camera 11. If the pixel pitch of the LED display 12 differs from the pixel pitch of the camera 11, a slight misalignment may occur between the pixel pitch of the LED display 12 and the pixel pitch of the camera 11 depending on the shooting conditions.

[0031] If a slight misalignment occurs, for example, light emitted from one pixel Pi1 may not only be incident on one pixel Pi11 among the pixels Pi11 shown surrounded by a dashed line at the end of arrow #2 in Figure 4, but may also leak into the adjacent pixel Pi11.

[0032] If light emitted from one pixel Pi1 does not only enter one pixel Pi11 but also leaks into the adjacent pixel Pi11, the captured image will not be an image accurately projected by the LED display 12, and moire (deterioration) will occur in the captured image, as shown at the tip of arrow #3 in Figure 4.

[0033] The relationship between the pixel pitch of the LED display 12 and the pixel pitch of the camera 11 is affected by a number of shooting parameters. The shooting parameters include values ​​representing the camera position and camera setting values.

[0034] Fig. 5 is a diagram illustrating camera positions. Fig. 5 shows a plan view of a studio where virtual production shooting will take place. Area A11 in Fig. 5 is an area in which the camera 11 can be moved, and area A12 is an area in which the subject can be moved.

[0035] The value representing the camera position is a parameter that indicates the positional relationship of objects in the studio, such as the distance d between the camera 11 and the LED display 12, and the angle θ of the camera 11 with respect to the LED display 12. The value representing the camera position is obtained, for example, by camera tracking.

[0036] As shown in Fig. 5A, distance d indicates the distance between camera 11 and LED display 12 when the optical axis OA of camera 11 is perpendicular to the display surface of LED display 12. As shown in Fig. 5B, angle θ indicates the angle of the optical axis OA of camera 11 relative to the axis perpendicular to LED display 12, indicated by the dashed dotted line.

[0037] In FIG. 5, for ease of explanation, the display surface of the LED display 12 is shown as a flat surface, but the shape of the display surface of the LED display 12 is not limited to a flat surface.

[0038] Fig. 6 is a diagram for explaining camera setting values, showing how light emitted from pixel Pi1 on the LED display 12 passes through the lens and is imaged on pixel Pi11 on the sensor surface of the camera 11.

[0039] The camera setting values ​​are parameters that indicate the settings of the camera 11 itself, such as the focus s and the focal length F, and are parameters that are determined corresponding to values ​​that indicate the camera position, for example. The camera setting values ​​are acquired from the camera 11 itself, for example.

[0040] 6, focal length F indicates the focal length of the lens, and focus s indicates the extension position of the lens, in other words, the in-focus position. By adjusting focus s, front focus (a state in which the focus is shifted to the front side of the LED display 12 (toward the camera 11)) or rear focus (a state in which the focus is shifted to the rear side of the LED display 12 (toward the camera 11)) can be achieved.

[0041] 6, distance d indicates the distance from the display surface of the LED display 12 to the center of the lens when the focus is on the display surface of the LED display 12. Distance B indicates the distance from the center of the lens to the sensor surface when the focus is on the display surface of the LED display 12.

[0042] In order to avoid the occurrence of moire, it has been conventionally recommended to adjust the focus s as shown in Fig. 7 while keeping the optical axis of the camera 11 perpendicular to the display surface of the LED display 12. In the example of Fig. 7, the focus s is adjusted so that the image is not focused on the display surface of the LED display 12 but is in front focus.

[0043] The actual shooting process performed by a conventional shooting system will be described with reference to the flowchart in Fig. 8. The actual shooting process realizes shooting using virtual production.

[0044] In step S1, the LED display displays a background image and the camera starts taking pictures.

[0045] In step S2, the photographing system determines whether or not moiré has occurred. For example, the photographer looks at a monitor (monitoring display) that displays the photographed image taken by the camera, checks whether or not moiré has occurred in the photographed image, i.e., whether or not moiré exists in the photographed image, and operates the photographing system depending on whether or not moiré has occurred.

[0046] If it is determined in step S2 that moiré has occurred, the process proceeds to step S3. In step S3, the camera accepts a change in the shooting parameters by the photographer. For example, to avoid the occurrence of moiré, the photographer manually adjusts the focus s while keeping the optical axis of the camera 11 perpendicular to the display surface of the LED display 12. After the shooting parameters have been changed, the process proceeds to step S4.

[0047] On the other hand, if it is determined in step S2 that no moire has occurred, the process of step S3 is skipped and the process proceeds to step S4.

[0048] In step S4, the camera photographs the subject against the background image to obtain a photographed image.

[0049] In step S5, the imaging system determines whether or not to end imaging.

[0050] If it is determined in step S5 that the photographing should not be ended, the process returns to step S2, and the subsequent processes are carried out. On the other hand, if it is determined in step S5 that the photographing should be ended, the process ends.

[0051] As described above, in the past, in order to avoid the occurrence of moire, the photographer had to manually adjust the focus while keeping the optical axis of the camera perpendicular to the display surface of the LED display.

[0052] Adjusting the focus or limiting the camera angle relative to the LED display can lead to unintended reductions in resolution and limitations on the scenes you can capture. Furthermore, because the focus and camera position (for example, the distance between the camera and the LED display) are not recorded after adjustments, even if you take photos under the same conditions at a later date, you will need to adjust the focus and other settings again to avoid moire, which is time-consuming.

[0053] The present technology was conceived in light of the above points, and aims to automatically calculate shooting parameters suitable for avoiding moiré by referencing a database in which shooting parameters related to shooting by the camera 11 are recorded in association with information indicating the presence or absence of moiré in the captured image, and to set shooting parameters associated with information indicating the absence of moiré, thereby making it possible to improve the efficiency of shooting. In the present disclosure, the shooting parameters recorded in the database and referenced by the camera 11 may be referred to as reference shooting parameters. On the other hand, the shooting parameters currently set for the camera 11 may be referred to as current shooting parameters.

[0054] 2. Configuration and Operation of the Photography System Configuration of the Photography System FIG. 9 is a block diagram showing an example of the configuration of the photography system 1. As shown in FIG.

[0055] 9 performs preliminary photography before the actual photography (photographing of a subject) by photographing with various combinations of photography parameters to check for the presence or absence of moiré and generates a database containing the confirmed photography parameters as reference photography parameters. Then, during the actual photography, the photography system calculates reference photography parameters that can avoid the occurrence of moiré based on the database, and controls the current photography parameters to match the reference photography parameters, or presents information to enable the photographer to adjust the current photography parameters to match the reference photography parameters.

[0056] As shown in FIG. 9, the photography system 1 includes a camera 11 , a camera position tracking unit 51 , a background image display unit 52 , and a photography control unit 53 .

[0057] The camera 11 includes a photographing unit 61 , a camera setting value transmission unit 62 , and a camera setting value control unit 63 .

[0058] The photographing unit 61 photographs the subject against the background image displayed on the LED display 12 to obtain a photographed image. The photographing unit 61 transmits the photographed image to the photographing control unit 53.

[0059] The camera setting value transmission unit 62 acquires the current camera setting values ​​from the photographing unit 61 and transmits information indicating the current camera setting values ​​to the background image display unit 52 and the photographing control unit 53 .

[0060] The camera setting value control unit 63 controls the photographing unit 61 so that the camera setting values ​​calculated by the photographing control unit 53 (reference setting values ​​recorded in the database) are achieved.

[0061] The camera position tracking unit 51 includes a camera position estimation unit 71 and a camera position transmission unit 72 .

[0062] The camera position estimation unit 71 estimates the current position of the camera 11. In the present disclosure, the current position of the camera 11 may be simply referred to as the current value of the camera position. The current value of the camera position is estimated based on the results of tracking using, for example, an IR (Infrared) camera placed in the studio and a marker made of a retroreflective material. The camera position estimation unit 71 supplies information indicating the current value of the camera position to the camera position transmission unit 72.

[0063] The camera position transmitting unit 72 transmits the information indicating the current value of the camera position supplied from the camera position estimating unit 71 to the background image display unit 52 and the photography control unit 53 as current position information of the camera 11 .

[0064] Although not shown in the drawings, the camera position transmission unit 72 transmits and displays information indicating the camera position (reference value recorded in the database) calculated by the shooting control unit 53 based on the current position of the camera 11 to a display device such as a monitor provided on the camera 11 or a monitor located near the photographer. As will be described later, the displayed reference position of the camera 11 represents a relative position with respect to the display 12. In the present disclosure, these camera monitors / display devices may be simply referred to as camera monitors or camera display devices. The photographer can avoid the occurrence of moire by moving the camera 11 or changing the orientation of the camera 11 so that it matches the reference position displayed on the display device.

[0065] Note that a plurality of candidate camera positions may be displayed on the display device, allowing the photographer to select a camera position that provides a desired angle of view from among the candidate camera positions that can avoid the occurrence of moire.

[0066] The background image display unit 52 includes an LED display 12, a camera position acquisition unit 81, and a camera setting value acquisition unit 82.

[0067] The camera position acquisition unit 81 receives information indicating the current value of the camera position transmitted from the camera position tracking unit 51 and supplies it to the LED display 12 .

[0068] The camera setting value acquisition unit 82 receives information indicating the current camera setting values ​​transmitted from the camera 11 and supplies it to the LED display 12 .

[0069] The LED display 12 renders and displays a background image based on the current camera position and current camera settings.

[0070] The functions of the camera position acquisition unit 81 and the camera setting value acquisition unit 82, and the function of rendering a background image are realized by a control device such as a PC. In addition to these functions, the function of the camera position tracking unit 51 may also be realized by the control device.

[0071] The photographing control unit 53 includes a moire detection unit 91 , a recording unit 92 , and a photographing parameter calculation unit 93 .

[0072] The moire detection unit 91 detects moire in the image captured by the camera 11, and supplies information indicating the presence or absence of moire to the recording unit 92 as a moire detection result.

[0073] The recording unit 92 associates and records in a database the information indicating the current camera setting values ​​transmitted from the camera 11, the information indicating the current value of the camera position transmitted from the camera position tracking unit 51, and the moire detection result by the moire detection unit 91. In the present disclosure, the current camera setting values ​​and the current value of the camera position received by the shooting control unit 53 may be referred to as current shooting parameters.

[0074] The shooting parameter calculation unit 93 calculates reference shooting parameters that can avoid the occurrence of moiré, based on information indicating the current camera setting values ​​transmitted from the camera 11, information indicating the current value of the camera position transmitted from the camera position tracking unit 51, and the database recorded in the recording unit 92. For example, the shooting parameter calculation unit 93 searches for reference shooting parameters that can avoid the occurrence of moiré, based on the current value and reference value of the camera position.

[0075] The shooting parameter calculation unit 93 transmits the calculated (searched) reference shooting parameters as current shooting parameters to the camera 11. Specifically, the shooting parameter calculation unit 93 transmits information indicating camera setting values ​​(reference setting values) from among the calculated reference shooting parameters to the camera 11. In addition, the shooting parameter calculation unit 93 transmits information indicating the camera position (reference value) from among the calculated reference shooting parameters to the camera 11 via the camera position tracking unit 51.

[0076] The functions of the photography control unit 53 are realized by an information processing device such as a PC. In addition to the functions of the photography control unit 53, some of the functions of the camera position tracking unit 51 and the background image display unit 52 may also be realized by the information processing device.

[0077] - Operation of the shooting system 1 The processing performed by the shooting system 1 of the present technology includes a shooting preparation process as preliminary shooting, in which reference shooting parameters and moire detection results are recorded in a database, and a main shooting process as main shooting, in which the subject is actually photographed.

[0078] The photographing preparation process performed by the photographing system 1 of the present technology will be described with reference to the flowchart of FIG.

[0079] In step S21, the photography control unit 53 creates a list showing combinations of photography parameters based on, for example, the size of the studio that is the photography environment and the specifications of the camera 11.

[0080] 11 is a diagram showing an example of a list showing combinations of shooting parameters. There is a correspondence relationship between values ​​representing camera positions and camera setting values, for example, as shown in FIG.

[0081] 11, minθ, minθ+intervalθ, minθ+2×intervalθ, and minθ+3×intervalθ are given as the angle θ of the camera 11 relative to the LED display 12. minθ is the minimum angle between the camera 11 and the LED display 12 in the studio, and intervalθ is the angular interval at which the presence or absence of moiré is detected.

[0082] 11, mind, mind+intervald, mind+2×intervald, and mind+3×intervald are given as the distance d between the camera 11 and the LED display 12. mind is the shortest distance between the camera 11 and the LED display 12 in the studio, and intervald is the distance interval at which the presence or absence of moiré is detected.

[0083] 11, the focal length F is minF, minF+intervalF, minF+2×intervalF, and minF+3×intervalF. minF is the shortest focal length of the camera 11, and intervalF is the interval between focal lengths at which the presence or absence of moiré is detected.

[0084] 11, the focus s is expressed as mins, mins+intervals, mins+2×intervals, and mins+3×intervals. Mins is the minimum focus of the camera 11, and intervals is the focus interval for detecting the presence or absence of moiré.

[0085] As described above, for example, if four values ​​are given for each type of shooting parameter, the shooting parameters are combined in a round-robin manner to list 256 combinations. Note that the types of shooting parameters and the intervals at which moiré is detected can be changed as appropriate depending on the shooting environment and the specifications of the camera.

[0086] Returning to FIG. 10, in step S22, the LED display 12 displays, for example, a grid pattern shown in FIG. 12A.

[0087] In step S23, the photography system 1 changes the current camera position and camera setting values ​​according to the list. For example, the photography system 1 selects one combination from the listed combinations of photography parameters and changes the current camera position and camera setting values ​​to the selected camera position and camera setting values.

[0088] Specifically, the camera position transmitting unit 72 transmits information indicating the camera positions included in the combination selected from the list to the camera monitor for display, and the camera setting value control unit 63 controls the photographing unit 61 so that the camera setting values ​​are included in the combination selected from the list.

[0089] In step S24, the photographing unit 61 photographs the grid pattern displayed on the LED display 12 and acquires a photographed image.

[0090] In step S25, the moiré detection unit 91 checks the presence or absence of moiré based on the captured image. The moiré detection unit 91 detects moiré that has occurred in the captured image in which a grid pattern has been captured, as shown in B of Fig. 12 .

[0091] In step S26, the recording unit 92 records data in which the camera setting values, camera positions, and moire detection results are associated with each other in the database. For example, [1 or 0, θ ０ ,d ０ ,F ０ ,s ０ ], data indicating the moiré detection result and a combination of multiple types of shooting parameters is recorded in the database. The first item of the data indicates the moiré detection result, and if moiré is detected, the value of the item indicating the detection result is 1, and if moiré is not detected, the value of the item indicating the detection result is 0. In the present disclosure, a value indicating the position (relative position) of the camera 11 with respect to the display 12, which is recorded in the database as a reference shooting parameter, may be referred to as a reference value indicating a relative position. Note that a value indicating the position of the camera 11 may also be simply referred to as the position of the camera 11. On the other hand, a camera setting value recorded in the database as a reference shooting parameter may also be referred to as a reference setting value.

[0092] In step S27, the photography system 1 determines whether or not to end photography. For example, if the presence or absence of moire has been confirmed for all of the listed combinations, it is determined that photography should end.

[0093] If it is determined in step S27 that the photographing should not be ended, the process returns to step S23, and the subsequent processes are performed. For example, the presence or absence of moiré is checked for the other listed combinations, and [1 or 0, θ １ ,d １ ,F １ ,s １ ] is recorded in the database.

[0094] On the other hand, if it is determined in step S27 that the photographing is to be ended, the process proceeds to step S28. In step S28, the recording unit 92 classifies the data recorded in the database. Specifically, the recording unit 92 classifies the recorded data in the form of [1,θ ｉ ,d ｉ ,F ｉ ,s ｉ ] and a set of data with moiré such as [0,θｍ ,d ｍ ,F ｍ ,s ｍ ] and a moiré-free data group such as [i, m]. i indicates the index of the moiré-containing data group, and m indicates the index of the moiré-free data group.

[0095] After the data is classified in step S27, the photographing preparation process ends. As described above, in the photographing preparation process, the recording unit 92 repeatedly records the presence or absence of moiré in the photographed image in association with the photographing parameters while changing the photographing parameters, thereby generating a database.

[0096] Next, the main photographing process performed by the photographing system 1 of the present technology will be described with reference to the flowchart of FIG.

[0097] In step S41, the LED display 12 displays a background image, and the photographing unit 61 starts photographing.

[0098] In step S42, the shooting parameter calculation unit 93 acquires information indicating the current value of the camera position (current position information).

[0099] In step S43, the shooting parameter calculation unit 93 acquires information indicating the current camera setting values.

[0100] In step S44, the moire detection unit 91 determines whether or not moire has occurred in the photographed image, that is, whether or not moire exists in the photographed image.

[0101] If it is determined in step S44 that moiré has occurred, the process proceeds to step S45. In step S45, the shooting parameter calculation unit 93 refers to the database recorded in the recording unit 92 based on the current camera position and current camera setting values, and calculates shooting parameters that can avoid the occurrence of moiré.

[0102] Specifically, the photographing parameter calculation unit 93 calculates the moiré-free data set [0, θ ｍ ,d ｍ ,F ｍ ,s ｍ ], data indicating the current shooting parameters [1,θcurrent ,d current ,F current ,s current Here, the current imaging parameters are the imaging parameters determined by the photographer, that is, the manually input imaging parameters.

[0103] For example, first, the photographing parameter calculation unit 93 calculates |θ current -θ ｍ θ where | is smallest ｍ (This θ ｍ θ ｐ ) and calculate the moiré-free data set [0,θ ｍ ,d ｍ ,F ｍ ,s ｍ ], [0,θ ｐ ] including all data sets [0,θ ｐ ,d Ｋ ,F Ｋ ,s Ｋ ] (K≦(θ=θ ｐ Search for the number of data points for which .

[0104] Next, the photographing parameter calculation unit 93 calculates |d current -d Ｋ | is smallest d Ｋ (This d Ｋ d ｑ ) and calculate the data set [0,θ ｐ ,d Ｋ ,F Ｋ ,s Ｋ ], [0,θ ｐ ,d ｑ ] including all data sets [0,θ ｐ ,d ｑ ,F Ｍ ,s Ｍ ] (M≦(θ=θ ｐ and d = d ｑ Search for the number of data points for which .

[0105] Next, the imaging parameter calculation unit 93 calculates |F current -F Ｍ | is smallest F Ｍ (This F Ｍ F ｓ ) and calculate the data set [0,θｐ ,d ｑ ,F Ｍ ,s Ｍ ], [0,θ ｐ ,d ｑ ,F ｓ ] including all data sets [0,θ ｐ ,d ｑ ,F ｓ ,s Ｎ ] (N≦(θ=θ ｐ and d = d ｑ And F=F ｓ Search for the number of data points for which .

[0106] Finally, the photographing parameter calculation unit 93 calculates |s current -s Ｎ | is smallest s Ｎ (This s Ｎ s ｒ ) and calculate the data set [0,θ ｐ ,d ｑ ,F ｓ ,s Ｎ ], [0,θ ｐ ,d ｑ ,F ｓ ,s ｒ ] data. θ ｐ ,d ｑ ,F ｓ ,s ｒ represents a combination of multiple types of reference shooting parameters that can avoid the occurrence of moiré. In this way, the shooting parameter calculation unit 93 searches for a combination consisting of multiple types of reference shooting parameters based on, for example, current position information of the camera 11, and sets it as a combination of current shooting parameters. Note that the database may be considered to represent a correspondence between multiple combinations of reference shooting parameters and information indicating the presence or absence of moiré.

[0107] By repeatedly referencing the reference shooting parameters that have the smallest difference from the manually input shooting parameters in a predetermined order for each type of reference shooting parameter, data (one of the combinations of reference shooting parameters) to be set as the current shooting parameters is selected from the moiré-free data group.

[0108] The angle θ, distance d, focal length F, and focus s are searched in this order. ｐ ,d ｑ ,F ｓ ,s ｒ In the data, the angle θ of the shooting parameters is the parameter that is changed the least from the current value. The photographer can set the order of the shooting parameters to be searched in a desired order. For example, the photographer can set the order to search for the shooting parameters that are most important (the values ​​of which the photographer does not want to change).

[0109] After the reference shooting parameters that can avoid the occurrence of moiré are calculated, the shooting control unit 53 transmits the calculated combination of reference shooting parameters to the camera 11 to reflect the combination in shooting. Specifically, the camera position transmission unit 72 transmits information indicating the reference values ​​of the camera position included in the combination of shooting parameters that can avoid the occurrence of moiré to the camera monitor for display. Furthermore, the camera setting value control unit 63 controls the shooting unit 61 so that the reference setting values ​​of the camera 11 are included in the combination of shooting parameters that can avoid the occurrence of moiré.

[0110] On the other hand, if it is determined in step S44 that no moiré has occurred, the process proceeds to step S46. In step S46, the recording unit 92 additionally records data in which the current camera setting values, the current camera position values ​​(manually input shooting parameters), and the moiré detection results are associated with each other in the database. For example, m＋1 ,d m＋1 ,F m＋1 ,s m＋1 ] is recorded in the database. By recording the combination of shooting parameters that do not generate moiré in the database even during actual shooting, it is possible to improve the accuracy of calculating the reference shooting parameters that can avoid the generation of moiré.

[0111] After the current shooting parameters are changed in step S45, or after the data is recorded in step S46, the process proceeds to step S47. In step S47, the shooting unit 61 shoots the subject against the background image to obtain a shot image.

[0112] In step S48, the photographing system 1 determines whether or not to end photographing.

[0113] If it is determined in step S48 that the photographing should not be ended, the process returns to step S42, and the subsequent processes are carried out. On the other hand, if it is determined in step S48 that the photographing should be ended, the photographing process ends.

[0114] Through the above processing, the photography system 1 can automatically calculate photography parameters suitable for avoiding moiré. Since the photography system 1 automatically controls the photography unit 61 so that the camera setting values ​​are suitable for avoiding moiré, the photographer can easily avoid the occurrence of moiré by simply moving or changing the orientation of the camera 11 so that the camera 11 matches the camera position (reference position) displayed on a monitor or the like.

[0115] 3. Modifications In cases where it is difficult to perform preliminary photography, for example, when a studio is not available before the actual photography, it is also possible to record the photography parameters and moire detection results during the actual photography and build a database.

[0116] The actual photographing process when the photographing preparation process is not performed will be described with reference to the flowchart of FIG.

[0117] The processing in steps S101 to S105 is the same as the processing in steps S41 to S45 in Fig. 13. That is, a background image is displayed on the LED display 12, and reference shooting parameters that can avoid the occurrence of moire are calculated.

[0118] In step S106, the shooting control unit 53 determines whether the calculated reference shooting parameters are usable, i.e., whether the reference shooting parameters can be reflected in the camera 11. If the shooting preparation process is not performed, particularly immediately after the start of actual shooting, it is considered that the database is insufficient. Therefore, it is determined whether the calculated reference shooting parameters can be reflected in the camera 11.

[0119] For example, |θ current -θｐ |<error, |d current -d ｑ |<error, |F current -F ｓ |<error, and |s current -s ｒ If all the conditions of |<error are satisfied, it is determined that the calculated shooting parameters can be used. error indicates a predetermined threshold. The photographer can set the threshold error according to the actual shooting situation.

[0120] If it is determined in step S106 that the calculated reference shooting parameters cannot be used, the process proceeds to step S107. In step S107, the camera 11 accepts changes to the current shooting parameters made by the photographer. For example, the camera 11 accepts position information of the camera 11 moved by the photographer to avoid the occurrence of moiré and the value of the focus s manually adjusted as changed shooting parameters (adjusted shooting parameters), and transmits them to the recording unit 92. Note that if it is determined that the calculated reference shooting parameters cannot be used, the monitor of the camera 11 may present information to the photographer indicating that the shooting parameters cannot be calculated. After the current shooting parameters are changed and the occurrence of moiré is avoided, the process proceeds to step S108.

[0121] In step S108, the recording unit 92 records data in which the adjusted shooting parameters and the moire detection results are associated with each other in the database.

[0122] In step S108, the data is recorded in the database, and then the process proceeds to step S110. If it is determined in step S106 that the calculated reference shooting parameters are usable, steps S107 and S108 are skipped, and the photography system 1 transmits the calculated reference shooting parameters to the camera 11 as current shooting parameters to be reflected in photography. Then, the process proceeds to step S110.

[0123] If it is determined in step S104 that no moiré has occurred, the process proceeds to step S109. In step S109, the recording unit 92 records data in which the current camera setting values, the current camera position, and the moiré detection results are associated with each other in the database. Thereafter, the process proceeds to step S110.

[0124] In step S110, the photographing unit 61 photographs the subject against the background image to obtain a photographed image.

[0125] In step S111, the photographing system 1 determines whether or not to end photographing.

[0126] If it is determined in step S111 that the photographing should not be ended, the process returns to step S102, and the subsequent processes are carried out. On the other hand, if it is determined in step S111 that the photographing should be ended, the photographing process ends.

[0127] As described above, even when it is difficult to perform preliminary photography, the photography system 1 can record the photography parameters and moiré detection results during the actual photography, build a database, and automatically calculate reference photography parameters suitable for avoiding moiré.

[0128] Regarding the computer, the above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0129] 15 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes using a program. The imaging control unit 53 is, for example, configured by a PC having the same configuration as that shown in FIG.

[0130] A CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0131] An input / output interface 505 is also connected to the bus 504. An input unit 506 including a keyboard, a mouse, etc., and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. Also connected to the input / output interface 505 are a storage unit 508 including a hard disk, a nonvolatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 that drives removable media 511.

[0132] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0133] The program executed by the CPU 501 is installed in the storage unit 508 by being recorded on, for example, a removable medium 511 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0134] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0135] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0136] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0137] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0138] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0139] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0140] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0141] Example of configuration combinations The present technology can also be configured as follows.

[0142] (1) An information processing device comprising: an imaging control unit that references a database in which reference imaging parameters related to imaging by a camera that images a display are recorded in association with information indicating the presence or absence of moiré in an image captured by the camera, and sets the reference imaging parameters associated with the information indicating the absence of moiré as current imaging parameters of the camera. (2) The reference imaging parameters include at least one of a reference setting value of the camera and a reference value representing a relative position of the camera with respect to the display. (3) The information processing device described in (2), in which the reference value representing the relative position of the camera represents the distance between the camera and the display and the angle of the camera with respect to the display. (4) The information processing device described in (3), in which the reference setting value of the camera represents at least one of a focal length and a focus, and corresponds to the reference value representing the relative position of the camera. (5) The information processing device according to any one of (3) or (4), wherein the photography control unit acquires current position information of the camera, searches for the reference photography parameters based on the current position information of the camera and reference values ​​representing the relative position of the camera, and transmits the searched reference photography parameters to the camera as the current photography parameters. (6) The information processing device according to (5), wherein the photography control unit presents the set relative position of the camera on a monitor of the camera. (7) The information processing device according to (5) or (6), wherein the photography control unit transmits the set reference setting values ​​of the camera to the camera. (8) The information processing device according to any one of (5) to (7), wherein the photography control unit acquires manually input photography parameters from the camera, and sets, as the current photography parameters, the reference photography parameters that have the smallest difference from the manually input photography parameters, among the reference photography parameters associated in the database with information indicating the absence of moiré.(9) The information processing device according to any one of (8), wherein the photography control unit sets a combination of a plurality of types of the reference photography parameters as a combination of the current photography parameters based on current position information of the camera. (10) The information processing device according to any one of (8), wherein the database represents a correspondence relationship between a plurality of combinations of the reference photography parameters and information indicating the presence or absence of moiré. (11) The information processing device according to (10), wherein the photography control unit selects one combination associated with information indicating the absence of moiré from among a plurality of combinations of the reference photography parameters in setting the reference photography parameters having the smallest difference from the manually input photography parameters. (12) The information processing device according to any one of (5) to (11), wherein the photography control unit acquires manually input photography parameters from the camera, determines whether moiré is present in the photographed image at the current position of the camera, and, based on a determination that moiré is not present in the photographed image, associates the manually input photography parameters with the information indicating the absence of moiré and records them in the database. (13) The information processing device according to (12), wherein the photography control unit, when it is determined that the moire is present in the photographed image, determines whether the set reference photography parameters can be reflected in the camera based on a difference between the set reference photography parameters and the manually input photography parameters, and transmits the set reference photography parameters to the camera based on the determination that the set reference photography parameters can be reflected in the camera. (14) The information processing device according to (13), wherein the photography control unit, when it is determined that the moire is present in the photographed image, determines that the set reference photography parameters can be reflected in the camera when a difference between the manually input photography parameters and the set reference photography parameters is smaller than a predetermined threshold.(15) The information processing device according to any of (13) or (14), wherein the shooting control unit, when determining that the set reference shooting parameters cannot be reflected in the camera, acquires adjusted shooting parameters different from the manually input shooting parameters from the camera, and records the adjusted shooting parameters and information indicating the presence or absence of moiré in the database. (16) The information processing device according to any of (1) to (15), further comprising: the camera; a camera position tracking unit that estimates a current position of the camera; and the display that displays an image rendered based on the current shooting parameters and the current position of the camera. (17) An information processing method, wherein the information processing device refers to a database in which reference shooting parameters related to shooting by a camera that shoots a display and information indicating the presence or absence of moiré in a photographed image photographed by the camera are recorded in association with each other, and sets the reference shooting parameters associated with the information indicating the absence of moiré as the current shooting parameters of the camera. (18) A computer-readable recording medium having recorded thereon a program for executing a process of referring to a database in which reference shooting parameters for shooting by a camera that shoots a display are recorded in association with information indicating the presence or absence of moiré in the captured image shot by the camera, and setting the reference shooting parameters associated with the information indicating the absence of moiré as the current shooting parameters of the camera.

[0143] DESCRIPTION OF SYMBOLS 1 Photography system, 11 Camera, 12 LED display, 51 Camera position tracking unit, 52 Background image display unit, 53 Photography control unit, 61 Photography unit, 62 Camera setting value transmission unit, 63 Camera setting value control unit, 71 Camera position estimation unit, 72 Camera position transmission unit, 72, 81 Camera position acquisition unit, 82 Camera setting value acquisition unit, 91 Moire detection unit, 92 Recording unit, 93 Photography parameter calculation unit

Claims

1. An information processing device having a shooting control unit that refers to a database in which reference shooting parameters for shooting by a camera that shoots a display are associated with information indicating the presence or absence of moiré in an image captured by the camera, and sets the reference shooting parameters associated with information indicating the absence of moiré as the current shooting parameters of the camera.

2. The information processing device according to claim 1, wherein the reference shooting parameters include at least one of a reference setting value of the camera and a reference value representing a relative position of the camera with respect to the display.

3. The information processing device according to claim 2, wherein the reference value representing the relative position of the camera represents the distance between the camera and the display and the angle of the camera with respect to the display.

4. The information processing device according to claim 3, wherein the reference setting value of the camera represents at least one of a focal length and a focus, and corresponds to a reference value representing a relative position of the camera.

5. The information processing device according to claim 3, wherein the shooting control unit acquires current position information of the camera, searches for the reference shooting parameters based on the current position information of the camera and a reference value representing the relative position of the camera, and transmits the searched reference shooting parameters to the camera as the current shooting parameters.

6. The information processing device according to claim 5, wherein the photography control unit displays the set relative position of the camera on a monitor of the camera.

7. The information processing device according to claim 5, wherein the photography control unit transmits the set reference setting values ​​of the camera to the camera.

8. The information processing device according to claim 5, wherein the shooting control unit acquires manually input shooting parameters from the camera, and sets, as the current shooting parameter, the reference shooting parameter that has the smallest difference from the manually input shooting parameter among the reference shooting parameters associated with information in the database indicating the absence of moire.

9. The information processing device according to claim 8, wherein the shooting control unit sets a combination of a plurality of types of the reference shooting parameters as the combination of the current shooting parameters based on current position information of the camera.

10. The information processing device according to claim 9, wherein the database represents a correspondence between a plurality of combinations of the reference shooting parameters and information indicating the presence or absence of the moire.

11. The information processing device according to claim 10, wherein the shooting control unit selects one of a number of combinations of the reference shooting parameters corresponding to information indicating the absence of moire when setting the reference shooting parameters that have the smallest difference from the manually input shooting parameters.

12. The information processing device according to claim 5, wherein the shooting control unit acquires manually input shooting parameters from the camera, determines whether or not the moire is present in the captured image at the current position of the camera, and, based on a determination that the moire is not present in the captured image, associates the manually input shooting parameters with information indicating the absence of the moire and records them in the database.

13. The information processing device of claim 12, wherein the shooting control unit, when it is determined that the moire is present in the captured image, determines whether the set reference shooting parameters can be reflected in the camera based on a difference between the set reference shooting parameters and the manually input shooting parameters, and transmits the set reference shooting parameters to the camera based on a determination that the set reference shooting parameters can be reflected in the camera.

14. The information processing device according to claim 13, wherein the shooting control unit determines that the set reference shooting parameters can be reflected in the camera when a difference between the manually input shooting parameters and the set reference shooting parameters is smaller than a predetermined threshold value.

15. The information processing device of claim 13, wherein the shooting control unit, when determining that the set reference shooting parameters cannot be reflected in the camera, acquires adjusted shooting parameters from the camera that are different from the manually input shooting parameters, and records the adjusted shooting parameters and information indicating the presence or absence of moire in the database.

16. The information processing device according to claim 1, further comprising: the camera; a camera position tracking unit that estimates a current position of the camera; and the display that displays an image rendered based on the current shooting parameters and the current position of the camera.

17. An information processing method in which an information processing device refers to a database in which reference shooting parameters for shooting with a camera that shoots a display are recorded in association with information indicating the presence or absence of moiré in an image captured by the camera, and sets the reference shooting parameters associated with information indicating the absence of moiré as the current shooting parameters of the camera.

18. A computer-readable recording medium having recorded thereon a program for executing a process of referencing a database in which reference shooting parameters relating to shooting by a camera that shoots a display are associated with information indicating the presence or absence of moiré in an image captured by the camera, and setting the reference shooting parameters associated with information indicating the absence of moiré as the current shooting parameters of the camera.