Information processing apparatus, information processing method, and recording medium

Abstract

The present technology pertains to an information processing device, an information processing method, and a recording medium capable of easily avoiding the generation of moire patterns. An information processing device according to the present technology includes: an imaging control unit that refers to a database in which a reference imaging parameter relating to imaging by an imaging device that images a display and information indicating whether moire exists in a captured image captured by the imaging device are recorded in association with each other, a reference imaging parameter associated with the information indicating that there is no moire is set as a current imaging parameter of the imaging apparatus. The present technology can be applied, for example, to an imaging system for virtual fabrication in which a subject is imaged with an image displayed on an LED display as a background.

Description

Technical Field

[0001] This technology relates to information processing apparatus, information processing method, and recording medium, and more specifically, to information processing apparatus, information processing method, and recording medium capable of easily avoiding moiré patterns. Background Technology

[0002] In recent years, virtual production (in-camera VFX), an imaging method using large light-emitting diode (LED) displays, has been widely used in film and theater imaging.

[0003] In virtual production, imaging is performed using an image displayed on an LED display as the background and a subject arranged in front of the LED display as the foreground. Furthermore, in virtual production, the image used as the background is rendered based on the position of the camera device, the orientation of the camera device, and lens configuration parameters (see, for example, Patent Document 1). Citation List Patent documents

[0004] Patent Document 1: Japanese translation of PCT International Application Publication No. 2022-554415. Summary of the Invention The problem to be solved by the present invention

[0005] When an image displayed on an LED display is captured, moiré patterns may appear in the captured image due to slight discrepancies between the pixel pitch of the LED display and the pixel pitch of the camera device. Typically, the photographer manually adjusts the imaging parameters during image capture to avoid moiré patterns.

[0006] There are multiple imaging parameters that can be adjusted to avoid moiré patterns, and there are limitations to manually adjusting multiple image parameters. Furthermore, since the adjusted imaging parameters are not recorded, even if imaging is performed under the same conditions later, it is necessary to perform the imaging parameter adjustment operation again to avoid moiré patterns.

[0007] This technology was developed in view of such circumstances and is designed to avoid moiré patterns. Problem Solution

[0008] According to a first aspect of the present technology, the information processing apparatus includes an imaging control unit that refers to a database in which reference imaging parameters related to imaging performed by a camera device imaging a display and information indicating the presence or absence of moiré patterns in a captured image captured by the camera device are recorded together, and sets the reference imaging parameters associated with the information indicating the absence of moiré patterns as the current imaging parameters of the camera device.

[0009] The information processing method according to a first aspect of the present technology includes: an information processing device referring to a database in which reference imaging parameters related to imaging performed by a camera device imaging a display and information indicating the presence or absence of moiré patterns in a captured image captured by the camera device are recorded in association with each other, to set the imaging parameters associated with the information indicating the absence of moiré patterns as the current imaging parameters of the camera device.

[0010] According to a recording medium of a first aspect of the present technology, a program for performing a process is recorded, the process comprising: referencing a database in which reference imaging parameters associated with imaging performed by a camera device imaging a display and information indicating the presence or absence of moiré patterns in a captured image captured by the camera device are recorded in association with each other, to set the imaging parameters associated with the information indicating the absence of moiré patterns as the current imaging parameters of the camera device.

[0011] In a first aspect of this technology, imaging parameters are set to avoid moiré patterns by referring to a database that records reference imaging parameters related to imaging performed by a camera device imaging a display and information indicating whether moiré patterns exist in the captured image captured by the camera device. Attached Figure Description

[0012] Figure 1 This is a diagram used to describe the overview of an imaging system that applies this technology. Figure 2 This is a diagram showing an example of a captured image by a camera device. Figure 3 This is a diagram illustrating an example of the state during imaging. Figure 4 It is a diagram used to describe the reasons for the appearance of moiré patterns. Figure 5 It is a diagram used to describe the location of the camera device. Figure 6 It is a diagram used to describe the settings of a camera device. Figure 7 This is a diagram illustrating an example of focus adjustment used to avoid moiré patterns. Figure 8 It is a flowchart used to describe the main imaging process performed by a conventional imaging system. Figure 9 This is a block diagram illustrating an example configuration of an imaging system. Figure 10 This is a flowchart describing the imaging preparation process performed by the imaging system of this technology. Figure 11 This is a diagram showing an example of a list indicating combinations of imaging parameters. Figure 12This is a diagram showing examples of lattice patterns and moiré patterns. Figure 13 This is a flowchart describing the main imaging process performed by the imaging system of this technology. Figure 14 This is a flowchart used to describe the main imaging process without performing imaging preparation processing. Figure 15 This is a block diagram illustrating an example of a computer hardware configuration. Detailed Implementation

[0013] The following describes implementation methods for carrying out this technology. The descriptions will be given in the following order. 1. Overview of the Imaging System 2. Configuration and operation of the imaging system 3. Modification

[0014] <1. Overview of Imaging Systems> Figure 1 This is a diagram used to describe the outline of the imaging system 1 that applies this technology.

[0015] Figure 1 The imaging system 1 is a system for imaging via, for example, in-camera VFX. The imaging system 1 includes a camera device 11, a wall-mounted light-emitting diode (LED) display 12, an information processing device (not shown) for controlling the camera device 11, and a control device (not shown) for controlling the LED display 12.

[0016] LED display 12 is arranged in a studio or similar space. LED display 12 displays images of, for example, virtual spaces created through computer graphics (CG). The photographer P1 uses camera device 11 to capture images of the motorcycle M1 as the subject, with the image displayed on LED display 12 as a background. Hereinafter, the image displayed on LED display 12 is referred to as the background image.

[0017] Figure 2 This is a diagram showing an example of a captured image captured by camera device 11.

[0018] like Figure 2 As shown, the captured image captured by the camera device 11 is an image in which the motorcycle M1 appears to exist in a virtual space mapped by the background image. In this way, the photographer P1 can thus image a captured image in the studio in which the space appearing in the background image extends into the background of the motorcycle M1 by using the imaging system 1.

[0019] Figure 3 This is a diagram illustrating an example of the state during imaging.

[0020] like Figure 3 As shown, for example, during imaging, the entire background image is displayed on the entire LED display 12.

[0021] like Figure 3 As shown above, when the camera device 11 performs imaging from the right side of the motorcycle M1 arranged in the center, a portion of the background image is displayed in the imaging area A1 in such a way that it is superimposed on the entire background image. This imaging area A1 is the area on the LED display 12 included in the imaging range of the camera device 11. In the imaging area A1, for example, the portion of the image that appears in the virtual space of the background image and becomes part of the background when imaging is performed from the right side of the motorcycle M1 is cropped from the entire background image and displayed.

[0022] like Figure 3 As shown below, when the camera device 11 performs imaging from the left side of the motorcycle M1 arranged in the center, a portion of the background image is displayed in the imaging area A1 in such a way that it is superimposed on the entire background image. In the imaging area A1, for example, the portion of the image that appears in the virtual space of the background image and becomes the background when imaging is performed from the left side of the motorcycle M1 is cropped from the entire background image and displayed.

[0023] In this way, during imaging, the photographer P1 performs imaging while sequentially changing the position, orientation, focal length, focus, etc. of the imaging device 11. The focal length, focus, etc. of the imaging device 11 can also be changed by the information processing device.

[0024] The control device tracks the camera operation (position and posture) of the camera device 11 and controls the position of the camera device on the LED display 12 that displays the background image based on the camera operation of the camera device 11. Specifically, the control device detects the imaging area based on the position, posture and settings of the camera device 11 and controls the background image displayed in the imaging area.

[0025] Note that, although for the sake of easy understanding of the above description, Figure 3 The imaging area A1 is surrounded by a thick line, but the line surrounding the imaging area is not actually displayed.

[0026] When imaging is performed with the background image displayed on the LED display 12 as the background, moiré patterns may appear in the captured image.

[0027] Figure 4 It is a diagram used to describe the reasons for the appearance of moiré patterns.

[0028] like Figure 4As shown, the background image displayed on the LED display 12 is projected onto the virtual camera device plane CP, and imaging is performed using the background image as a background.

[0029] For example, in the case of displaying a uniform background image on the LED display 12, such as Figure 4 As shown by the portion indicated by arrow #1, each pixel Pi1 constituting the LED display 12 emits light of the same color. In the following text, the distance between the centers of adjacent pixels Pi is referred to as the pixel pitch of the LED display 12.

[0030] Furthermore, among the multiple pixels Pi11 of the sensor disposed in the imaging device 11, the distance between the centers of adjacent pixels Pi11 is called the pixel pitch of the imaging device 11. If the pixel pitch of the LED display 12 is different from the pixel pitch of the imaging device 11, a slight deviation may occur between the pixel pitch of the LED display 12 and the pixel pitch of the imaging device 11 depending on the imaging conditions.

[0031] For example, in the event of a slight deviation, light emitted from a pixel Pi1 may not only enter... Figure 4 The pixel Pi11 in the section indicated by arrow #2, which is surrounded by a dashed line, is not necessarily leaked into the adjacent pixel Pi11.

[0032] like Figure 4 As shown by the part indicated by arrow #3, when light emitted from a pixel Pi1 not only enters a pixel Pi11 but also leaks into adjacent pixels Pi11, the captured image will not become the image precisely projected onto the LED display 12, and moiré patterns (degradation) will appear in the captured image.

[0033] The relationship between the pixel pitch of the LED display 12 and the pixel pitch of the camera device 11 is influenced by several imaging parameters. These imaging parameters include values ​​representing the position of the camera device and camera device settings.

[0034] Figure 5 It is a diagram used to describe the location of the camera device. Figure 5 It is a floor plan of the studio where images are created virtually. Figure 5 Area A11 is the area where the camera device 11 can be moved, and area A12 is the area where the subject can be moved.

[0035] The value indicating the position of the camera device is a parameter that indicates the positional relationship of objects in the studio, such as the distance d between the camera device 11 and the LED display 12, and the angle θ of the camera device 11 relative to the LED display 12. The value indicating the position of the camera device is obtained, for example, through camera device tracking.

[0036] like Figure 5 As shown in A, distance d indicates the distance between the camera device 11 and the LED display 12 when the optical axis OA of the camera device 11 is perpendicular to the display surface of the LED display 12. Figure 5 As shown in B, angle θ indicates the angle of the optical axis OA of the camera device 11 relative to the axis perpendicular to the LED display 12, indicated by alternating long and short dashed lines.

[0037] Note that in Figure 5 In this description, for the sake of simplicity, the display surface of the LED display 12 is shown as a plane, but the shape of the display surface of the LED display 12 is not limited to a plane.

[0038] Figure 6 It is a diagram used to describe the settings of a camera device. Figure 6 This shows a state in which light emitted from pixel Pi1 of LED display 12 forms an image on pixel Pi11 on the sensor surface of camera device 11 via a lens.

[0039] The camera device setting values ​​are parameters that indicate the settings of the main body of the camera device 11, such as focus s and focal length F, and are parameters determined, for example, corresponding to values ​​indicating the position of the camera device. For example, the camera device setting values ​​are obtained from the main body of the camera device 11.

[0040] like Figure 6 As shown, the focal length F indicates the focal length of the lens, and the focal point s indicates the extension position of the lens, in other words, the focusing position. By adjusting the focal point s, the state of front focal (the state where the focal point is shifted to the front side of the LED display 12 (the front side when viewed from the camera device 11)) and rear focal (the state where the focal point is shifted to the rear side of the LED display 12 (the rear side when viewed from the camera device 11)) can be achieved.

[0041] exist Figure 6 In the example, distance d indicates the distance from the display surface of the LED display 12 to the center of the lens when the display surface of the LED display 12 is in focus. Furthermore, distance B indicates the distance from the center of the lens to the sensor surface when the display surface of the LED display 12 is in focus.

[0042] To avoid moiré patterns, it is generally recommended to keep the optical axis of the camera device 11 perpendicular to the display surface of the LED display 12 while... Figure 7 The focus point is adjusted as shown in the diagram. Figure 7 In the example, the focus s is adjusted so that it is not focused on the display surface of the LED display 12, but rather in a forward focus state.

[0043] Reference Figure 8 The flowchart describes the main imaging process performed by a conventional imaging system. This main imaging process enables imaging through virtual fabrication.

[0044] In step S1, the LED display shows the background image, and the camera device begins imaging.

[0045] In step S2, the imaging system determines whether moiré patterns have appeared. For example, the photographer views a monitor (monitoring display) showing the captured image captured by the camera device to confirm whether moiré patterns have appeared in the captured image, i.e., whether moiré patterns exist in the captured image, and operates the imaging system based on whether moiré patterns have appeared.

[0046] If moiré patterns are determined to have appeared in step S2, the process proceeds to step S3. In step S3, the camera device receives changes to the imaging parameters from the photographer. For example, to avoid moiré patterns, the photographer manually adjusts the focus s while keeping the optical axis of the camera device 11 perpendicular to the display surface of the LED display 12. After changing the imaging parameters, the process proceeds to step S4.

[0047] On the other hand, if it is determined in step S2 that no moiré pattern has appeared, the process in step S3 is skipped, and the process proceeds to step S4.

[0048] In step S4, the camera device captures an image of the subject against a background image and obtains the captured image.

[0049] In step S5, the imaging system determines whether to end the imaging process.

[0050] If it is determined in step S5 that imaging should not be terminated, the process returns to step S2 and subsequent processing is performed. If it is determined in step S5 that imaging should be terminated, the process ends.

[0051] As described above, in order to avoid moiré patterns, the photographer needs to manually adjust the focus while keeping the optical axis of the camera perpendicular to the display surface of the LED monitor.

[0052] Adjusting the focus point and limiting the angle of the camera relative to the LED display can cause undesirable resolution reduction and limitations on the imaging scene. Furthermore, since the adjusted focus and camera position (e.g., the distance between the camera and the LED display) are not recorded, adjustments such as refocusing are required to avoid moiré patterns even when imaging is performed under the same conditions later, which is time-consuming and laborious.

[0053] This technology has been conceived by focusing on the points mentioned above, and can automatically calculate imaging parameters suitable for avoiding moiré patterns by setting imaging parameters associated with information indicating the absence of moiré patterns using a reference database, thereby enabling efficient imaging. In this database, imaging parameters related to imaging by the camera device 11 and information indicating the presence or absence of moiré patterns in the captured image are recorded in association with each other. In this disclosure, the imaging parameters to be referenced by the camera device 11 recorded in the database can be distinguished as reference imaging parameters. On the other hand, the imaging parameters currently set for the camera device 11 can be referred to as current imaging parameters.

[0054] <2. Configuration and Operation of the Imaging System> - Imaging system configuration Figure 9 This is a block diagram illustrating an example configuration of imaging system 1.

[0055] As a preliminary imaging process before the main imaging (imaging of the subject), Figure 9 The imaging system 1 performs imaging using various combinations of imaging parameters to confirm the presence of moiré patterns and generates a database including the confirmed imaging parameters as reference imaging parameters. Subsequently, in the main imaging, the imaging system calculates reference imaging parameters that can avoid moiré patterns based on the database, controls the current imaging parameters to match the reference imaging parameters, and presents information so that the photographer can adjust the current imaging parameters according to the reference imaging parameters.

[0056] like Figure 9 As shown, the imaging system 1 includes a camera device 11, a camera device position tracking unit 51, a background image display unit 52, and an imaging control unit 53.

[0057] The camera device 11 includes an imaging unit 61, a camera device setting value transmission unit 62, and a camera device setting value control unit 63.

[0058] Imaging unit 61 uses the background image displayed on LED display 12 as a background to image the subject and acquire a captured image. Imaging unit 61 then sends the captured image to imaging control unit 53.

[0059] The camera device setting value sending unit 62 obtains the current camera device setting value from the imaging unit 61 and sends the information indicating the current camera device setting value to the background image display unit 52 and the imaging control unit 53.

[0060] The camera device setting control unit 63 controls the imaging unit 61 to obtain the camera device setting value (reference setting value recorded in the database) calculated by the imaging control unit 53.

[0061] The camera device position tracking unit 51 includes a camera device position estimation unit 71 and a camera device position transmission unit 72.

[0062] The camera device position estimation unit 71 estimates the current position of the camera device 11. In this disclosure, the current position of the camera device 11 may be simply referred to as the current value of the camera device position. The current value of the camera device position is estimated based on the results of tracking using, for example, an infrared (IR) camera device provided in a studio and markers including retroreflective material. The camera device position estimation unit 71 supplies information indicating the current value of the camera device position to the camera device position transmission unit 72.

[0063] The camera device position sending unit 72 sends the information provided by the camera device position estimation unit 71, indicating the current value of the camera device position, as the current position information of the camera device 11 to the background image display unit 52 and the imaging control unit 53.

[0064] Although not shown, for example, the camera device position sending unit 72 instructs the imaging control unit 53 to send information about the camera device position (a reference value recorded in a database) calculated based on the current position of the camera device 11 to a display device, such as a monitor installed in the camera device 11 or a monitor placed near the photographer, and causes the display device to display this information. As described later, the displayed reference position of the camera device 11 represents its relative position with respect to the display 12. In this disclosure, the camera device monitor / display device may be simply referred to as the camera device monitor or the camera device display device. The photographer can avoid moiré patterns by moving the camera device 11 or changing the orientation of the camera device 11 to obtain the reference position displayed on the display device.

[0065] Note that multiple candidate camera positions can be displayed on the screen. In this case, the photographer can select a camera position from the candidates that avoid moiré patterns to obtain the desired angle.

[0066] The background image display unit 52 includes an LED display 12, a camera device position acquisition unit 81, and a camera device setting value acquisition unit 82.

[0067] The camera device position acquisition unit 81 receives information from the camera device position tracking unit 51 indicating the current value of the camera device position, and supplies the information to the LED display 12.

[0068] The camera device setting value acquisition unit 82 receives information sent from the camera device 11 indicating the current camera device setting value, and supplies the information to the LED display 12.

[0069] The LED display 12 renders and displays a background image based on the current position of the camera device and the current camera device settings.

[0070] Note that the functions of the camera device position acquisition unit 81 and the camera device setting value acquisition unit 82, as well as the function of rendering the background image, are implemented by a control device such as a PC. In addition to these functions, the functions of the camera device position tracking unit 51 can also be implemented by a control device.

[0071] The imaging control unit 53 includes a moiré detection unit 91, a recording unit 92, and an imaging parameter calculation unit 93.

[0072] The moiré detection unit 91 detects moiré patterns in the captured image captured by the camera device 11 and supplies information indicating the presence or absence of moiré patterns as the moiré detection result to the recording unit 92.

[0073] The recording unit 92 records information indicating the current camera device setting value sent from the camera device 11, information indicating the current value of the camera device position sent from the camera device position tracking unit 51, and the moiré detection result from the moiré detection unit 91 in a database in a correlated manner. In this disclosure, the current camera device setting value and the current value of the camera device position received by the imaging control unit 53 can be referred to as the current imaging parameters.

[0074] The imaging parameter calculation unit 93 calculates reference imaging parameters that can avoid moiré patterns based on information indicating the current camera device setting value sent from the camera device 11, information indicating the current position of the camera device sent from the camera device position tracking unit 51, and a database recorded in the recording unit 92. For example, the imaging parameter calculation unit 93 searches for reference imaging parameters that can avoid moiré patterns based on the current position of the camera device and the reference value.

[0075] The imaging parameter calculation unit 93 sends the calculated (searched) reference imaging parameters to the camera device 11 as the current imaging parameters. Specifically, the imaging parameter calculation unit 93 sends the information indicating the camera device setting value (reference setting value) in the calculated reference imaging parameters to the camera device 11. In addition, the imaging parameter calculation unit 93 sends the information indicating the camera device position (reference value) in the calculated reference imaging parameters to the camera device 11 via the camera device position tracking unit 51.

[0076] Note that the functions of the imaging control unit 53 are implemented by an information processing device such as a PC. In addition to the functions of the imaging control unit 53, some functions of the camera position tracking unit 51 and the background image display unit 52 can also be implemented by the information processing device.

[0077] - Operation of Imaging System 1 The processing performed by the imaging system 1 of this technology includes: imaging preparation processing, which records reference imaging parameters and moiré detection results in a database as preliminary imaging, and main imaging processing, which actually images the subject as the main imaging.

[0078] Reference Figure 10 The flowchart describes the imaging preparation process performed by the imaging system 1 of this technology.

[0079] In step S21, the imaging control unit 53 creates a list of combinations of imaging parameters based on, for example, the size of the studio used as the imaging environment and the specifications of the camera device 11.

[0080] Figure 11 This is a diagram illustrating an example of a list indicating combinations of imaging parameters. For example, values ​​representing the camera device position and camera device settings have, for instance, values ​​such as... Figure 11 The correspondence shown is as follows.

[0081] exist Figure 11 In the examples, the angle θ between the camera device 11 and the LED display 12 includes minθ, minθ+intervalθ, minθ+2×intervalθ, and minθ+3×intervalθ. minθ is the minimum angle between the camera device 11 and the LED display 12 in the studio, and intervalθ is the interval of the angle used to detect the presence of moiré patterns.

[0082] exist Figure 11 In the examples, the distance d between the camera device 11 and the LED display 12 includes mind, mind+intervald, mind+2×intervald, and mind+3×intervald. mind is the shortest distance between the camera device 11 and the LED display 12 in the studio, and intervald is the interval of distance used to detect the presence of moiré patterns.

[0083] exist Figure 11 In the examples, examples of focal length F include minF, minF+intervalF, minF+2×intervalF, and minF+3×intervalF. minF is the shortest focal length in the imaging device 11, and intervalF is the interval of focal length used to detect the presence of moiré patterns.

[0084] exist Figure 11In the examples, the focal points s include mins, mins+intervals, mins+2×intervals, and mins+3×intervals. mins is the smallest focal point in the camera device 11, and intervals is the interval between focal points used to detect the presence of moiré patterns.

[0085] As described above, for example, with four values ​​for each type of imaging parameter, an exhaustive combination of imaging parameters yields 256 combinations. Note that the types of imaging parameters and the intervals used to detect the presence of moiré patterns vary appropriately depending on the imaging environment and the specifications of the imaging device.

[0086] Return to Figure 10 In step S22, the LED display 12 displays, for example... Figure 12 The lattice pattern shown in A.

[0087] In step S23, the imaging system 1 changes the current camera device position and camera device settings according to the list. For example, the imaging system 1 selects a combination from the listed combinations of imaging parameters and changes the current camera device position and camera device settings to obtain the selected camera device position and camera device settings.

[0088] Specifically, the camera device position sending unit 72 sends information indicating the camera device positions included in the combinations selected from the list to the camera device monitor, and causes the monitor to display the information. Furthermore, the camera device setting value control unit 63 controls the imaging unit 61 to obtain the camera device setting values ​​included in the combinations selected from the list.

[0089] In step S24, the imaging unit 61 images the lattice pattern displayed on the LED display 12 and acquires the captured image.

[0090] In step S25, the moiré detection unit 91 determines the presence of moiré patterns based on the captured image. For example... Figure 12 As shown in B, the moiré detection unit 91 detects moiré patterns appearing in the captured image in which the lattice pattern is imaged.

[0091] In step S26, the recording unit 92 records data in the database in which the camera device setting value, camera device position, and moiré detection result are correlated with each other. For example, data indicating the moiré detection result and combinations of various types of imaging parameters, such as [1 or 0, θ0, d0, F0, s0], are recorded in the database. The first item of this data indicates the moiré detection result, and if moiré is detected, the value of the item indicating the detection result is 1, and if no moiré is detected, the value of the item indicating the detection result is 0. In this disclosure, the value representing the position (relative position) of the camera device 11 relative to the display 12, recorded in the database as a reference imaging parameter, can be referred to as a reference value representing the relative position. Note that the value representing the position of the camera device 11 can be simply referred to as the position of the camera device 11. On the other hand, the camera device setting value recorded in the database as a reference imaging parameter can be referred to as a reference setting value.

[0092] In step S27, the imaging system 1 determines whether to terminate imaging. For example, if moiré patterns are confirmed to exist for all listed combinations, the system determines to terminate imaging.

[0093] If it is determined in step S27 that imaging should not be terminated, the process returns to step S23 and subsequent processing is performed. For example, for other listed combinations, it is confirmed whether moiré patterns exist, and the data such as [1 or 0, θ1, d1, F1, s1] are recorded in the database.

[0094] If the imaging process is determined to end in step S27, the process proceeds to step S28. In step S28, the recording unit 92 classifies the data recorded in the database. Specifically, the recording unit 92 classifies the recorded data into groups containing moiré patterns, such as [1, θ]. i d i F i s i ] and moiré patterns do not exist in data sets such as [0, θ m d m F m s m The 'i' indicates the index of the data group where the moiré pattern exists, and the 'm' indicates the index of the data group where the moiré pattern does not exist.

[0095] After classifying the data in step S27, the imaging preparation process ends. As described above, in the imaging preparation process, the recording unit 92 repeatedly records the presence of moiré patterns and imaging parameters in the captured image in a correlated manner while changing the imaging parameters, thereby generating a database.

[0096] Next, we will refer to Figure 13The flowchart describes the main imaging process performed by the imaging system 1 of this technology.

[0097] In step S41, the LED display 12 displays a background image, and the imaging unit 61 begins imaging.

[0098] In step S42, the imaging parameter calculation unit 93 obtains information about the current value indicating the position of the camera device (current position information).

[0099] In step S43, the imaging parameter calculation unit 93 obtains information indicating the current camera device setting value.

[0100] In step S44, the moiré detection unit 91 determines whether moiré patterns have appeared in the captured image, that is, whether moiré patterns exist in the captured image.

[0101] If moiré patterns are determined to have occurred in step S44, the process proceeds to step S45. In step S45, the imaging parameter calculation unit 93 calculates imaging parameters that can avoid moiré patterns by referring to the database recorded in the recording unit 92 based on the current value of the camera device position and the current camera device setting value.

[0102] Specifically, the imaging parameter calculation unit 93 calculates the data from the non-existent moiré pattern data set [0, θ]. m d m F m s m Search and indicate the current imaging parameters [1, θ] in the [] ] current d current F current s current The data closest to the target image. Here, the current imaging parameters are those determined by the photographer, i.e., manually entered imaging parameters.

[0103] For example, firstly, the imaging parameter calculation unit 93 calculates |θ current - θ m | becomes the smallest θ m (Assuming θ) m For θ p ), and from the moiré pattern, there is no data set [0, θ] m d m F m s m The search in [0, θ] includes [0, θ] p All data sets [0, θ] p d K F K s K (K ≤ (satisfying θ = θ)) p (The number of data).

[0104] Next, the imaging parameter calculation unit 93 calculates |d current - d K | becomes the smallest d K (Assuming that d) K For d q ), and from the data set [0, θ p d K F K s K The search in [0, θ] includes [0, θ] p d q All data sets [0, θ] p d q F K s K [M≤(satisfies θ = θ]] p And d = d q (The number of data).

[0105] Next, the imaging parameter calculation unit 93 calculates |F current - F M | becomes the smallest F M (Assuming that F) M For F s ), and from the data set [0, θ p d q F M s M The search in [0, θ] includes [0, θ] p d q F s All data sets [0, θ] p d q F s s N (N≤(satisfies θ = θ)) p And d = d q And F = F s (The number of data).

[0106] Finally, the imaging parameter calculation unit 93 calculates |s current - s N | is the smallest s N (Assuming that s) N For s r ), and from the data set [0, θ p d q F s s N Search data [0, θ] in p d q F s sr ]. θ p d q F s s r This represents a combination of various types of reference imaging parameters that can avoid moiré patterns. In this way, for example, the imaging parameter calculation unit 93 searches for combinations of various types of reference imaging parameters based on the current position information of the imaging device 11, and sets that combination as the current imaging parameter combination. Note that the database can be considered to represent the correspondence between multiple combinations of reference imaging parameters and information indicating the presence or absence of moiré patterns.

[0107] By repeatedly referencing the reference imaging parameters that have the smallest difference from the manually entered imaging parameters for each type of reference imaging parameter in a predetermined order, data (one of the combinations of reference imaging parameters) that is set as the current imaging parameter is selected from the group of data where moiré does not exist.

[0108] The data searched in the order of angle θ, distance d, focal length F, and focal point s [0, θ] p d q F s s r In the image parameters, the angle θ is the parameter whose value changes the least relative to its current value. Note that the photographer can set the order in which the image parameters to be searched are set as desired. For example, the photographer can start the search from the image parameters that the photographer values ​​(whose values ​​they do not want to change).

[0109] After calculating the reference imaging parameters that can avoid moiré patterns, the imaging control unit 53 sends the calculated combination of reference imaging parameters to the imaging device 11 and reflects this combination in the image. Specifically, the imaging device position sending unit 72 sends information indicating the reference value of the imaging device position included in the combination of imaging parameters that can avoid moiré patterns to the imaging device monitor and displays this information on the monitor. Furthermore, the imaging device setting value control unit 63 controls the imaging unit 61 to obtain the reference setting value of the imaging device 11 included in the combination of imaging parameters that can avoid moiré patterns.

[0110] On the other hand, if it is determined in step S44 that no moiré pattern has appeared, the process proceeds to step S46. In step S46, the recording unit 92 additionally records in the database data in which the current camera device setting value, the current value of the camera device position (manually input imaging parameters), and the moiré pattern detection result are correlated with each other. For example, data such as [0, θ] m+ 1,d m+ 1, F m+ 1,s m+1] Recorded in the database. Even during the main imaging process, combinations of imaging parameters that do not exhibit moiré patterns are recorded in the database, thereby improving the calculation accuracy of reference imaging parameters that can avoid moiré patterns.

[0111] After the current imaging parameters are changed in step S45, or after the data is recorded in step S46, the processing proceeds to step S47. In step S47, the imaging unit 61 images the subject using the background image as a background and acquires a captured image.

[0112] In step S48, the imaging system 1 determines whether to end the imaging process.

[0113] If it is determined in step S48 that imaging should not be terminated, the process returns to step S42, and subsequent processing is performed. If it is determined in step S48 that imaging should be terminated, the main imaging process ends.

[0114] Through the above processing, the imaging system 1 can automatically calculate imaging parameters suitable for avoiding moiré patterns. Since the imaging system 1 automatically controls the imaging unit 61 to obtain the camera device setting value suitable for avoiding moiré patterns, the photographer can easily avoid moiré patterns simply by moving the camera device 11 or changing the orientation of the camera device 11 to obtain the camera device position (reference position) displayed on the monitor or the like.

[0115] <3. Modified Example> For example, in situations where preliminary imaging is difficult to perform, such as when the studio cannot be used before the main imaging, a database can be built by recording imaging parameters and moiré detection results during the main imaging process.

[0116] Reference Figure 14 The flowchart describes the main imaging process without performing imaging preparation processing.

[0117] The processing from step S101 to step S105 and Figure 13 The processing from step S41 to step S45 is similar. That is, the background image is displayed on the LED display 12, and reference imaging parameters that can avoid moiré patterns are calculated.

[0118] In step S106, the imaging control unit 53 determines whether the calculated reference imaging parameters can be used, that is, whether these reference imaging parameters can be reflected in the imaging device 11. Without performing imaging preparation processing, especially after the start of main imaging, the database is considered insufficient. Therefore, it is necessary to determine whether the calculated reference imaging parameters can be reflected in the imaging device 11.

[0119] For example, when |θ current - θp | < Error, |d current - d q | < Error, |F current - F s | < error and |s current - s r Under all conditions of error, the calculated imaging parameters can be determined for use. The error indicator is a predetermined threshold. The photographer can set the threshold error based on the main image.

[0120] If it is determined in step S106 that the calculated reference imaging parameters cannot be used, the process proceeds to step S107. In step S107, the camera device 11 receives changes to the current imaging parameters from the photographer. For example, the camera device 11 receives the position information of the camera device 11 moved by the photographer to avoid moiré patterns, or the value of the manually adjusted focus s, as the changed imaging parameters (adjusted imaging parameters) and sends it to the recording unit 92. Note that if it is determined that the calculated reference imaging parameters cannot be used, the monitor of the camera device 11 can present information to the photographer indicating that the imaging parameters cannot be calculated. After changing the current imaging parameters and avoiding moiré patterns, the process proceeds to step S108.

[0121] In step S108, the recording unit 92 records the data in the database in which the adjusted imaging parameters and moiré detection results are correlated with each other.

[0122] After the data is recorded in the database in step S108, the process proceeds to step S110. Furthermore, if it is determined in step S106 that the calculated reference imaging parameters are usable, the processes in steps S107 and S108 are skipped, and the imaging system 1 sends the calculated reference imaging parameters as the current imaging parameters to the camera device 11, reflecting these parameters in the image. Thereafter, the process proceeds to step S110.

[0123] If no moiré pattern is detected in step S104, the process proceeds to step S109. In step S109, the recording unit 92 records data in the database that correlates the current camera device setting value, the current value of the camera device position, and the moiré detection result. Afterward, the process proceeds to step S110.

[0124] In step S110, the imaging unit 61 uses the background image as a background to image the subject and acquires the captured image.

[0125] In step S111, the imaging system 1 determines whether to end the imaging process.

[0126] If it is determined in step S111 that imaging should not be terminated, the process returns to step S102, and subsequent processing is performed. If it is determined in step S111 that imaging should be terminated, the main imaging process ends.

[0127] As described above, even in situations where preliminary imaging is difficult to perform, the imaging system 1 is able to record imaging parameters and moiré detection results during the main imaging process to build a database, and automatically calculate reference imaging parameters suitable for avoiding moiré patterns.

[0128] - About Computers The series of processes described above can be executed by hardware or by software. In the case where the series of processes are executed by software, the program included in the software is installed from the program recording medium onto dedicated hardware, such as a computer or general-purpose personal computer.

[0129] Figure 15 This is a block diagram illustrating an example configuration of the hardware of a computer performing the above series of processes according to a program. The imaging control unit 53 comprises, for example, components having... Figure 15 The configuration shown is similar to the configuration of a PC.

[0130] The central processing unit (CPU) 501, read-only memory (ROM) 502 and random access memory (RAM) 503 are interconnected via bus 504.

[0131] The input / output interface 505 is also connected to the bus 504. Input units 506, including a keyboard, mouse, etc., and output units 507, including a display, speakers, etc., are connected to the input / output interface 505. Furthermore, storage units 508, including hard disks, non-volatile memory, etc., communication units 509, including network interfaces, etc., and drivers 510 that drive removable media 511 are connected to the input / output interface 505.

[0132] In a computer configured as described above, for example, CPU 501 loads a program stored in storage unit 508 into RAM 503 via input / output interface 505 and bus 504, and executes the program to perform the series of processes described above.

[0133] For example, the program executed by CPU 501 is recorded in removable medium 511, or provided via wired or wireless transmission media such as a local area network, the Internet, or digital broadcasting, and then installed in storage unit 508.

[0134] Note that the program to be executed by the computer may be a program that is processed sequentially in the order described in this specification, or it may be a program that is processed in parallel or at necessary time points, such as when it is called.

[0135] In this specification, "system" means a collection of multiple components (devices, modules (parts), etc.), and it is not important whether all components are housed in the same housing. Therefore, multiple devices housed in separate housings and interconnected via a network, as well as a device comprising multiple modules housed in one housing, are both systems.

[0136] The effects described in this manual are merely illustrative and not limiting, and other effects may be provided.

[0137] The implementation of this technology is not limited to the implementation described above, but various modifications can be made without departing from the scope of this technology.

[0138] For example, this technology can be implemented in cloud computing, where multiple devices share and collaborate to process a function via a network.

[0139] Furthermore, each step described in the flowchart above can be performed by a single device or by multiple devices in a distributed manner.

[0140] Furthermore, in cases where a single step includes multiple processes, the multiple processes included in that single step may be performed by a single device or by multiple devices in a distributed manner.

[0141] - Examples of configuration combinations This technology can also be configured as follows. (1) An information processing device, comprising: The imaging control unit refers to a database that records reference imaging parameters related to imaging performed by a camera device imaging a display and information indicating the presence or absence of moiré patterns in the captured image captured by the camera device, and sets the reference imaging parameters associated with the information indicating the absence of moiré patterns as the current imaging parameters of the camera device. (2) According to the information processing apparatus described in (1), wherein, The reference imaging parameters include at least one of a reference setting value for the camera device and a reference value representing the relative position of the camera device with respect to the display. (3) According to the information processing device described in (2), wherein, The reference values ​​indicating the relative position of the camera device represent the distance between the camera device and the display, as well as the angle of the camera device relative to the display. (4) According to the information processing device described in (3), wherein, The reference setting value of the camera device represents at least one of the focal length and the focal point, and corresponds to a reference value representing the relative position of the camera device. (5) According to the information processing apparatus described in (3) or (4), wherein, Imaging control unit: Obtain the current location information of the camera device. Reference imaging parameters are searched based on the current position information of the camera device and reference values ​​representing the relative position of the camera device, and The searched reference imaging parameters are sent to the camera device as the current imaging parameters. (6) According to the information processing apparatus described in (5), wherein, The imaging control unit displays the relative position of the set camera on the monitor of the camera device. (7) According to the information processing apparatus described in (5) or (6), wherein, The imaging control unit sends the reference settings of the camera device to the camera device. (8) The information processing apparatus according to any one of (5) to (7), wherein, Imaging control unit: The imaging parameters are obtained manually from the camera device, and Set the reference imaging parameter that has the smallest difference from the manually entered imaging parameter from the reference imaging parameters associated with information in the database indicating the absence of moiré patterns as the current imaging parameter. (9) The information processing apparatus according to any one of (8), wherein, The imaging control unit sets a combination of various types of reference imaging parameters as the current imaging parameters based on the current position information of the camera device. (10) According to the information processing apparatus described in (9), wherein, The database represents the correspondence between multiple combinations of reference imaging parameters and information indicating the presence of moiré patterns. (11) According to the information processing apparatus described in (10), wherein, When setting reference imaging parameters that have the smallest difference from manually entered imaging parameters, the imaging control unit selects from multiple combinations of reference imaging parameters a combination associated with information indicating the absence of moiré patterns. (12) The information processing apparatus according to any one of (5) to (11), wherein, Imaging control unit: Obtain manually input imaging parameters from the camera device. Determine whether moiré patterns exist in the captured image at the current position of the camera device, and Based on the determination that no moiré patterns exist in the captured image, the manually entered imaging parameters are associated with each other and recorded in the database with information indicating the absence of moiré patterns. (13) According to the information processing apparatus described in (12), wherein, Imaging control unit: If moiré patterns are found in the captured image, the system determines whether the set reference imaging parameters can be reflected in the imaging device based on the difference between the set reference imaging parameters and the manually input imaging parameters. Based on the determination that the set reference imaging parameters can be reflected in the camera device, the set reference imaging parameters are sent to the camera device. (14) According to the information processing apparatus described in (13), wherein, If the difference between the manually input imaging parameters and the set reference imaging parameters is less than a predetermined threshold, the imaging control unit determines that the set reference imaging parameters can be reflected in the camera device. (15) According to the information processing apparatus described in (13) or (14), wherein, If it is determined that the set reference imaging parameters cannot be reflected in the imaging device, the imaging control unit obtains adjusted imaging parameters from the imaging device that differ from the manually input imaging parameters, and The adjusted imaging parameters and information indicating the presence of moiré patterns are recorded in the database. (16) The information processing apparatus according to any one of (1) to (15) further includes: Camera device; A camera device position tracking unit that estimates the current position of the camera device; and A display that shows an image rendered based on current imaging parameters and the current position of the camera device. (17) An information processing method, comprising: The information processing device, referring to a database that records reference imaging parameters related to imaging performed by a camera device imaging a display and information indicating the presence or absence of moiré patterns in a captured image captured by the camera device, sets the reference imaging parameters associated with the information indicating the absence of said moiré patterns as the current imaging parameters of the camera device. (18) A computer-readable recording medium for recording a program for performing a process, the process comprising: Referring to a database that records reference imaging parameters related to imaging performed by a camera device imaging a display and information indicating the presence or absence of moiré patterns in captured images captured by the camera device, the reference imaging parameters associated with the information indicating the absence of said moiré patterns are set as the current imaging parameters of the camera device. List of reference numerals

[0143] 1 Imaging System, 11 Camera Device, 12 LED Display, 51 Camera Device Position Tracking Unit, 52 Background Image Display Unit, 53 Imaging Control Unit, 61 Imaging Unit, 62 Camera Device Setting Value Transmission Unit, 63 Camera Device Setting Value Control Unit, 71 Camera Device Position Estimation Unit, 72 Camera Device Position Transmission Unit, 81 Camera Device Position Acquisition Unit, 82 Camera Device Setting Value Acquisition Unit, 91 Moiré Detection Unit, 92 Recording Unit, 93 Imaging Parameter Calculation Unit

Claims

1. An information processing apparatus, comprising: An imaging control unit, which refers to a database that records reference imaging parameters associated with imaging performed by a camera device imaging a display and information indicating whether moiré patterns exist in a captured image captured by the camera device, sets the reference imaging parameters associated with the information indicating that the moiré patterns do not exist as the current imaging parameters of the camera device.

2. The information processing apparatus according to claim 1, wherein, The reference imaging parameters include at least one of the reference setting value of the camera device and the reference value representing the relative position of the camera device with respect to the display.

3. The information processing apparatus according to claim 2, wherein, The reference value representing the relative position of the camera device represents the distance between the camera device and the display, as well as the angle of the camera device relative to the display.

4. The information processing apparatus according to claim 3, wherein, The reference setting value of the camera device represents at least one of focal length and focus, and corresponds to a reference value representing the relative position of the camera device.

5. The information processing apparatus according to claim 3, wherein, The imaging control unit: Obtain the current location information of the camera device. The reference imaging parameters are searched based on the current position information of the camera device and a reference value representing the relative position of the camera device, and The searched reference imaging parameters are sent to the camera device as the current imaging parameters.

6. The information processing apparatus according to claim 5, wherein, The imaging control unit displays the relative position of the camera device on the monitor of the camera device.

7. The information processing apparatus according to claim 5, wherein, The imaging control unit sends the reference setting values ​​of the camera device to the camera device.

8. The information processing apparatus according to claim 5, wherein, The imaging control unit: The imaging parameters are obtained manually from the camera device, and The reference imaging parameter that has the smallest difference from the manually entered imaging parameter among the reference imaging parameters in the database that are associated with information indicating the absence of the moiré pattern is set as the current imaging parameter.

9. The information processing apparatus according to claim 8, wherein, The imaging control unit sets a combination of various types of reference imaging parameters as the current imaging parameters based on the current position information of the camera device.

10. The information processing apparatus according to claim 9, wherein, The database represents the correspondence between multiple combinations of the reference imaging parameters and information indicating the presence or absence of the moiré pattern.

11. The information processing apparatus according to claim 10, wherein, When setting reference imaging parameters that have the smallest difference from the manually input imaging parameters, the imaging control unit selects from a plurality of combinations of the reference imaging parameters a combination associated with information indicating the absence of the moiré pattern.

12. The information processing apparatus according to claim 5, wherein, The imaging control unit: The imaging parameters are manually input from the camera device. Determine whether the moiré pattern exists in the captured image at the current position of the camera device, and Based on the determination that the moiré pattern does not exist in the captured image, the manually entered imaging parameters are recorded in the database in association with information indicating the absence of the moiré pattern.

13. The information processing apparatus according to claim 12, wherein, The imaging control unit: If moiré patterns are determined to exist in the captured image, it is determined whether the set reference imaging parameters can be reflected in the imaging device based on the difference between the set reference imaging parameters and the manually input imaging parameters. Based on the determination that the set reference imaging parameters can be reflected in the camera device, the set reference imaging parameters are sent to the camera device.

14. The information processing apparatus according to claim 13, wherein, If the difference between the manually input imaging parameters and the set reference imaging parameters is less than a predetermined threshold, the imaging control unit determines that the set reference imaging parameters can be reflected in the camera device.

15. The information processing apparatus according to claim 13, wherein, If it is determined that the set reference imaging parameters cannot be reflected in the imaging device, the imaging control unit obtains adjusted imaging parameters from the imaging device that are different from the manually input imaging parameters, and The adjusted imaging parameters and information indicating the presence of the moiré pattern are recorded in the database.

16. The information processing apparatus according to claim 1, further comprising: The camera device; A camera device position tracking unit that estimates the current position of the camera device; as well as The display shows an image rendered based on the current imaging parameters and the current position of the camera device.

17. An information processing method, comprising: The information processing device refers to a database that records reference imaging parameters related to imaging performed by a camera device imaging a display and information indicating whether moiré patterns exist in the captured image captured by the camera device, and sets the reference imaging parameters associated with the information indicating that the moiré patterns do not exist as the current imaging parameters of the camera device.

18. A computer-readable recording medium recording a program for performing processing, the processing comprising: Referring to a database that records reference imaging parameters related to imaging performed by a camera device imaging a display and information indicating the presence or absence of moiré patterns in captured images captured by the camera device, the reference imaging parameters associated with the information indicating the absence of moiré patterns are set as the current imaging parameters of the camera device.

Images