Image capturing system, control apparatus, image capturing apparatus, control method, and storage medium

Abstract

To adaptively reduce the influence of moire.SOLUTION: The image pickup system includes an image pickup device which picks up an image of a display device having a pixel structure, and a control device which controls an operation of the image pickup device, and the image pickup device includes an image pickup unit and a filter unit which is configured to be able to change a filter characteristic and is applied to the image pickup unit. The control apparatus includes an acquisition unit configured to acquire information about an image capturing condition of the image capturing apparatus, an estimation unit configured to output an estimation result obtained by estimating whether a moire occurs in a captured image output by the image capturing unit based on the information about the image capturing condition, and a control unit configured to change a filter characteristic of the filter unit according to the estimation result.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an imaging system, a control device, an imaging device, a control method, and a program, and more particularly to a photography technique that uses a display device as a background. [Background technology]

[0002] In recent years, there has been a technology that makes it possible to shoot an image in which the subject appears to exist in the scene depicted in the background image by placing a large display device such as an LED wall behind the subject and displaying a background image generated by computer graphics or the like on the display device. One example of the use of such technology is virtual production technology.

[0003] On the other hand, since display devices have a pixel structure in which display pixels (hereinafter referred to as display pixels) are arranged in a grid pattern, moire may occur when a display device on which a background image is displayed is captured. More specifically, since the display pixels are physically spaced apart in a display device, the background image displayed on the display device is presented in a manner that has regular gaps when viewed locally. Therefore, moire may occur when a pattern with such regular gaps is captured using an imaging element in which photoelectric conversion elements (hereinafter sometimes referred to as imaging pixels) are similarly arranged in a grid pattern.

[0004] When moiré occurs, unnatural patterns appear in the display area, and the video obtained by shooting (hereinafter referred to as the shot video) may not provide a viewing experience that makes it seem as if the subject is present in the scene depicted in the background video. In other words, the occurrence of moiré in the shot video may give the viewer the impression that the subject was shot in front of the display device. For this reason, there is also a shooting system that outputs a moiré alert when it is determined that moiré will occur, prompting the viewer to reset the shooting conditions (camera position and camera path) (Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2024-006362 Summary of the Invention [Problem to be solved by the invention]

[0006] However, as with the technology described in Patent Document 1, the process of repeatedly taking preliminary photographs and determining whether or not moiré occurs until photographing conditions that do not cause moiré are set is time-consuming. Furthermore, even under photographing conditions that may cause moiré, the photographer may still wish to take a photograph depending on the state of the subject, and it may be undesirable for the photographer to be prevented from taking a photograph by an alert.

[0007] The present invention has been made in consideration of the above-mentioned problems, and has as its object to provide an imaging system, a control device, an imaging device, a control method, and a program that adaptively reduce the influence of moire. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the imaging system of the present invention is an imaging system including an imaging device that images a display device having a pixel structure and a control device that controls the operation of the imaging device, wherein the imaging device has an imaging means and a filter means that is configured so that the filter characteristics can be changed and is applied to the imaging means, and the control device has an acquisition means that acquires information about the imaging conditions of the imaging device, an estimation means that outputs an estimation result that estimates whether or not moiré will occur in the captured image output by the imaging means based on the information about the imaging conditions, and a control means that changes the filter characteristics of the filter means in accordance with the estimation result. [Effects of the Invention]

[0009] With this configuration, the present invention makes it possible to adaptively reduce the influence of moire. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an imaging system according to an embodiment and a modification of the present invention; [Figure 2] FIG. 1 is a block diagram illustrating a functional configuration of a control device 100 according to an embodiment and a modification of the present invention. [Figure 3] FIG. 1 is a block diagram illustrating the functional configuration of an image capture device 200 according to an embodiment and a modification of the present invention. [Figure 4] FIG. 10 is a diagram illustrating an example of a filter unit 213 according to an embodiment and a modification of the present invention. [Figure 5] 1 is a diagram illustrating the occurrence of moire in the embodiment and the modified example of the present invention; [Figure 6] 1 is a flowchart illustrating a control process executed by the control device 100 according to an embodiment and a modification of the present invention; [Figure 7] FIG. 10 is a diagram illustrating an example of a filter section 213 according to a first modified example of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] [Embodiment 1] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] In the embodiment described below, the present invention is applied to an imaging system that is provided in a photography studio configured for virtual production and includes an imaging device, a display device, and a control device that controls these devices. However, the present invention can be applied to any system or device that enables shooting of a subject against a large display device that displays a background image.

[0013] <<Configuration of Imaging System>> FIG. 1 shows an example of the configuration of a photography studio in which an imaging system according to this embodiment is used.

[0014] As shown in the figure, in a photography studio, a subject 500 is positioned in front of a display device 300 configured as a large LED wall (closer to the imaging device 200 than the display device 300), and is photographed by the imaging device 200. In virtual production technology, an image (background image) related to a desired scene is displayed on the display device 300, and the imaging device 200 photographs the subject 500 together with the background image. This allows the imaging device 200 to directly record an image that makes it appear as if the subject 500 is actually present in the desired scene. In other words, virtual production simplifies post-shooting processing compared to conventional methods such as chromakey compositing, in which the subject 500 is placed in front of a screen of a predetermined color, a subject area is extracted from each frame of the image photographed, and the image is then composited with the background image.

[0015] In order to express changes in the appearance of the background according to the position of the image capturing device 200, the background image displayed on the display device 300 is generated as, for example, computer graphics (CG). More specifically, the background image is generated by arranging background objects in a three-dimensional space and depicting the three-dimensional space from a viewpoint synchronized with the position and orientation of the image capturing device 200.

[0016] For this reason, the imaging system is configured to be able to detect the position and orientation of the imaging device 200. The imaging system of this embodiment employs a detection method in which a viewpoint detection marker 303 provided at an arbitrary location in a photography studio is captured by the imaging device 200, and the position and orientation of the imaging device 200 are derived based on the image of the marker that appears in the captured image. Here, since it is not preferable for the viewpoint detection marker 303 to be captured together with the subject 500 and the display device 300, in the example of FIG. 1 , it is provided on the ceiling of the photography studio. For this reason, the imaging device 200 is provided with a second imaging optical system 221 for mainly capturing the ceiling of the photography studio, in addition to a first imaging optical system 211 for capturing the subject 500 and the display device 300.

[0017] 1, display device 300 is composed of LED wall 301 and LED wall 302. In one embodiment, LED wall 301 may be used for displaying a background image captured in the imaging angle of view of imaging device 200, and LED wall 302 may be used for displaying a background image to be reflected (e.g., reflected) on subject 500 or an object placed in the foreground, and the LED wall 302 may have different uses. In this embodiment, LED wall 302 may be a display device with a lower display resolution than LED wall 301.

[0018] In addition, the photography studio is provided with a lighting device 400 that illuminates the subject 500 according to a scene related to the background image. In this embodiment, it is assumed that the generation of the background image to be displayed on the display device 300, the detection of the position and orientation of the image capture device 200, and the lighting control of the lighting device 400 are performed by the control device 100. Furthermore, the control device 100 of this embodiment also controls the operation of the image capture device 200, as will be described in detail later.

[0019] <Configuration of the control device 100> Here, the functional configuration of the control device 100 will be described with reference to the block diagram of FIG.

[0020] The control unit 101 is a control device such as a CPU that controls the operation of each block of the control device 100. The control unit 101 controls the operation of each block by reading out an operation program for each block stored in, for example, a ROM 102, expanding it in a RAM 103, and executing it.

[0021] ROM 102 is a non-volatile storage device. ROM 102 stores information such as parameters required for the operation of each block in addition to the operation programs of each block of control device 100. RAM 103 is a volatile storage device. RAM 103 is used not only as an area for expanding the operation programs of each block, but also as a storage area for temporarily storing intermediate data output by the operation of each block.

[0022] The detection unit 104 detects the position and orientation of the imaging device 200. The detection unit 104 extracts an image corresponding to the viewpoint detection marker 303 from an image (hereinafter referred to as a detection image) captured by the imaging device 200 using the second imaging optical system 221, and detects the position and orientation of the imaging device 200 based on the state of the extracted image.

[0023] The estimation unit 105 estimates whether moiré will occur in an image (hereinafter simply referred to as a captured image) captured by the imaging device 200 using the first imaging optical system 211. Although details will be described later, in the imaging system of this embodiment, the imaging device 200 captures an image of the subject 500 and the display device 300 having a pixel structure using the first imaging optical system 211, and therefore moiré may occur in the captured image depending on the imaging conditions. Therefore, the estimation unit 105 estimates whether moiré will occur in the captured image based on information about the imaging conditions related to the imaging device 200, and outputs the estimation result.

[0024] The display control unit 106 controls the display of the background video on the display device 300. The display control unit 106 includes a drawing device such as a GPU. The display control unit 106 generates each frame of the background video by drawing a three-dimensional space relating to a predetermined scene based on a viewpoint corresponding to the position and orientation of the imaging device 200 detected by the detection unit 104.

[0025] The lighting control unit 107 controls the lighting state of the lighting device 400. As described above, the lighting state of the lighting device 400 is controlled according to the scene to be photographed, which corresponds to the background image. In one embodiment, the lighting control unit 107 can refer to the detection result by the detection unit 104 when controlling the lighting state.

[0026] The communication unit 108 is a communication interface included in the control device 100. Through the communication unit 108, the control device 100 can transmit and receive information to and from external devices such as the imaging device 200, the display device 300, and the lighting device 400 via a network (not shown).

[0027] <Configuration of imaging device 200> Next, the functional configuration of the imaging device 200 will be described with reference to the block diagram of FIG.

[0028] The camera control unit 201 is a control device that controls the operation of each block of the imaging device 200. The camera control unit 201 controls the operation of each block by, for example, reading out an operation program for each block stored in a camera ROM 202, expanding the program into a camera RAM 203, and executing the program.

[0029] The camera ROM 202 is a non-volatile storage device. The camera ROM 202 stores information such as parameters required for the operation of each block in addition to the operation programs of each block of the imaging device 200. The camera RAM 203 is a volatile storage device. The camera RAM 203 is used not only as an area for expanding the operation programs of each block, but also as a storage device for temporarily storing intermediate data output by the operation of each block.

[0030] The camera communication unit 204 is a communication interface provided in the imaging device 200. Through the camera communication unit 204, the imaging device 200 can transmit and receive information to and from an external device such as the control device 100 via a network (not shown).

[0031] As described above, the imaging device 200 is configured to be able to perform imaging using both the first imaging optical system 211 and the second imaging optical system 221, and therefore has a first imaging section 212 and a second imaging section 222, respectively.

[0032] The first imaging unit 212 is an imaging device such as a CCD or CMOS sensor. The first imaging unit 212 photoelectrically converts an optical image formed on an imaging plane via the first imaging optical system 211 to obtain an image signal related to the captured image. The first imaging optical system 211 is an interchangeable imaging lens composed of a lens group including, for example, a zoom lens. That is, the imaging device 200 is configured so that multiple types of imaging lenses can be attached as the first imaging optical system 211. The captured image obtained by the first imaging unit 212 is output to the image processing unit 214, where various image processing is applied. The image signal processed in the image processing unit 214 may include, for example, A / D conversion processing, development processing, white balance processing, etc. The captured image obtained by the first imaging unit 212 becomes a frame of a video to be recorded, capturing the subject 500 against the background video displayed on the imaging device 200. Therefore, the captured image obtained during shooting is subjected to various image processing by the image processing unit 214, and then sequentially output to the recording medium 230 as data of the video frame to be recorded.

[0033] The imaging device 200 of this embodiment includes a filter unit 213 configured to control whether or not to apply a filter on a region-by-region basis. The filter unit 213 of this embodiment is disposed between the first imaging optical system 211 and the first imaging unit 212, and is configured to variably apply a filter to a light beam incident via the first imaging optical system 211 on a region-by-region basis. For example, the filter unit 213 may employ an optical variable low-pass filter whose filter characteristics can be changed on a region-by-region basis by applying a voltage. As shown in FIG. 4( a), the optical variable low-pass filter normally passes an incident light beam related to one region so that the incident light beam is imaged on a single pixel associated with that region. On the other hand, when the filter effect is ON, the optical variable low-pass filter passes the incident light beam so that the incident light beam is dispersed into multiple pixels and imaged on those pixels, as shown in FIG. 4( b). With such variable filter characteristics, the filter unit 213 can blur any region of the captured image.

[0034] The second imaging unit 222 is an imaging device such as a CCD or CMOS sensor. The second imaging unit 222 photoelectrically converts an optical image formed on the imaging surface via the second imaging optical system 221 to obtain an image signal related to the captured image. The captured image obtained by the second imaging unit 222 is output to an A / D conversion unit 223, where it is subjected to A / D conversion processing and output as a digital image signal (detection image). The detection image obtained by the A / D conversion unit 223 is transmitted to the control device 100 via the camera communication unit 204, and is used in detection processing of the position and orientation of the imaging device 200 by the detection unit 104.

[0035] <<Filter Control Overview>> In the video recorded by virtual production, it is necessary to ensure that the display device 300 displaying the background video is not recognized as the display device itself. For this reason, the position of the imaging device 200 in the photography studio and the imaging magnification that can be adopted are limited, basically so that the image of the display pixel of the display device 300 formed on the first imaging unit 212 is smaller than one pixel of the imaging element (hereinafter referred to as an imaging pixel). In addition, restrictions are placed on the position of the subject 500 so that the display device 300 is not in focus.

[0036] On the other hand, from the perspective of enabling highly flexible photography, there is a certain demand for the ability to flexibly change the position of subject 500 and the composition in which imaging device 200 captures subject 500. However, allowing these changes may mean that the above-mentioned constraints are no longer guaranteed, and as a result, the following problems are expected when display device 300 is included in the depth of field.

[0037] As described above, the display device 300 of this embodiment has a pixel structure, and therefore, when capturing an image using the first imaging optical system 211, an image of the display device 300 may be formed on the imaging element as a pattern with regular gaps. In this case, patterns exhibiting frequencies above the Nyquist frequency of the imaging element cannot be suitably sampled (captured), which may result in aliasing and the appearance of unintended patterns (moiré) in the captured image. If moiré occurs in the captured image, the image quality may be reduced, resulting in an unnatural impression of the video to be recorded, or the viewer may simply be given the impression that the image was captured in front of a display device such as an LED wall.

[0038] For this reason, in the imaging system of this embodiment, the estimation unit 105 of the control device 100 estimates whether moiré will occur based on information about the imaging conditions, and if it estimates that moiré will occur, it controls the optical low-pass filter to be enabled, thereby avoiding the occurrence of moiré. More specifically, when the estimation unit 105 obtains an estimation result indicating that moiré will occur, the control unit 101 outputs a control signal via the communication unit 108 to change the filter characteristics of the filter unit 213 of the imaging device 200.

[0039] The Nyquist frequency of the imaging element will now be described with reference to FIG. 5(a). FIG. 5(a) shows imaging pixels arranged in a range of 1 mm in the horizontal direction of the imaging element of the first imaging unit 212. In the example shown, 10 imaging pixels are arranged in the horizontal direction per mm. In this case, the maximum number of lines that can be resolved and recorded in the horizontal direction is 5 lines per mm when a line is formed every other pixel. Therefore, the Nyquist frequency of the imaging element is 5 lines / mm (or lp / mm (lp: line pairs)).

[0040] The frequency of the image of the display device 300 is represented by an image of the pixel structure of the display device 300, which is formed on the first imaging unit 212 via the first imaging optical system 211. FIG. 5B illustrates an image 501 of horizontally arranged display pixels constituting the display device 300, which is formed on the imaging element of the first imaging unit 212. As shown in the figure, the size of the image of one display pixel is smaller than the size of a single imaging pixel, ensuring that the display pixels do not appear visible in the captured image. In the example shown in the figure, 14 display pixels are arranged horizontally per mm and formed, and the frequency of the image of the pixel structure of the display device 300 is 7 lines / mm. In this case, the frequency of the image of the pixel structure exceeds the Nyquist frequency of the imaging element, so aliasing occurs even when sampling (imaging) is performed, resulting in moire fringes.

[0041] The frequency indicated by the image of the pixel structure of the display device 300 may vary depending on the relative positional relationship between the imaging device 200 and the display device 300, the pitch (pixel spacing) of the display pixels on the display device 300, and the focal length set for the first imaging optical system 211. That is, the image of the display pixels becomes larger the closer the imaging device 200 is to the display device 300, and the longer the focal length set for the first imaging optical system 211. Therefore, when estimating whether moiré will occur, the estimation unit 105 refers to information indicating the relative position of the display device 300 with respect to the imaging device 200 and, as necessary, information about the focal length set for the first imaging optical system 211. In one aspect, the relative position of the display device 300 with respect to the imaging device 200 can be identified based on information about the position and orientation of the imaging device 200 detected by the detection unit 104, since the arrangement of the display device 300 in the photography studio is known.

[0042] <Identifying the area where moiré occurs> Incidentally, since enabling the filter unit 213 reduces the sharpness of the image, it is preferable to control the filter unit 213 so that it is enabled only in an area where moiré is estimated to occur, rather than over the entire area of ​​the image sensor. For this reason, the estimation unit 105 identifies an area in the captured image where moiré may occur based on the following conditions.

[0043] First, the estimation unit 105 determines whether the display device 300 is included within the depth of field of the first imaging optical system 211. As described above, moiré occurs when the frequency of the image of the pixel structure of the display device 300 exceeds the Nyquist frequency of the image sensor. On the other hand, if the display pixels are not included within the depth of field, the image of the display pixels is formed in a blurred state and does not exhibit a gap pattern. Therefore, the estimation unit 105 extracts, as a target region, a region included in the depth of field of the first imaging optical system 211 from the region in the captured image where the image of the display device 300 appears, and identifies from that region a region where moiré may occur. In other words, the estimation unit 105 of this embodiment treats the image of the display device 300 distributed in a region not included in the depth of field of the first imaging optical system 211 as a region not subject to determination of whether moiré occurs.

[0044] The depth of field of the first imaging optical system 211 can be determined based on information about the aperture value and focal length set in the first imaging optical system 211. In this embodiment, information about the aperture value and focal length set for the first imaging optical system 211 in the imaging device 200 is transmitted to the control device 100 as imaging condition information at any time, and the estimation unit 105 can refer to this information.

[0045] Next, the estimation unit 105 derives the frequency indicated by the image of the pixel structure of the display device 300 in the target region extracted as being included in the depth of field. Then, the estimation unit 105 estimates whether or not the image distributed in the target region will cause moiré based on the frequency indicated by the image of the pixel structure. Because the depth of field changes depending on the imaging conditions, the frequency indicated by the image of the pixel structure according to the position in the target region may be derived for each position in the target region. If the frequency indicated by the image of the pixel structure derived in this way exceeds the Nyquist frequency of the imaging element, the estimation unit 105 identifies the corresponding region as a region where moiré may occur.

[0046] Meanwhile, the ability to resolve the pixel structure of the display device 300 also varies depending on the performance (resolution limit) of the first imaging optical system 211. As described above, the first imaging optical system 211 is an interchangeable imaging lens, and each lens has a different resolution limit. The resolution limit of the first imaging optical system 211 indicates how fine the optical image formed on the imaging surface of the imaging element via that optical system can be resolved; specifically, how fine (black and white) lines can be reproduced per millimeter on the imaging element. That is, even if the frequency indicated by the derived image of the pixel structure of the display device 300 is high, if it exceeds the resolution limit (unit: lines / mm) of the first imaging optical system 211, the gaps between display pixels will be blurred when the image is formed on the imaging element via the first imaging optical system 211. In other words, if the frequency indicated by the image of the pixel structure of the display device 300 in a given region fundamentally exceeds the resolution limit of the first imaging optical system 211, it can be determined that the pixel structure will not be resolved in that region, and therefore moiré will not occur.

[0047] Therefore, the estimation unit 105 of this embodiment identifies, within the target region, a region where the frequency indicated by the image of the pixel structure of the display device 300 is higher than the Nyquist frequency of the image sensor and lower than the frequency of the resolution limit of the first imaging optical system 211 as a region where moiré may occur. The estimation unit 105 outputs information about the region where moiré may occur thus identified as an estimation result, and the control unit 101 transmits a control signal to the imaging device 200 to enable the filter unit 213 in the corresponding region based on the information of the estimation result.

[0048] Note that information on the resolution limit of the photographing lens can change even within the depth of field depending on the set focal length and aperture value (wider apertures result in blur due to aberration, and smaller apertures result in blur due to diffraction), but a standard value is provided by, for example, the manufacturer of the photographing lens. Therefore, the information on the imaging conditions transmitted from the imaging device 200 to the control device 100 may include identification information that identifies the first imaging optical system 211 attached to the imaging device 200. In this aspect, for example, information on the resolution limit for each photographing lens is registered in advance in the ROM 102, and the estimation unit 105 can acquire information on the resolution limit of the first imaging optical system 211 based on the identification information.

[0049] In this way, it is possible to identify areas where moiré may occur based on the positional relationship between the imaging device 200 and the display device 300 and information on the imaging conditions, but it is not necessary to enable the filter unit 213 for the entire area. In other words, when shooting in a photography studio, it is expected that the imaging angle of view will include the subject 500. If the filter unit 213 is enabled in the area where the subject 500 is distributed (hereinafter referred to as the subject area), an image of the subject will be captured in a blurred manner. Therefore, the estimation unit 105 excludes the subject area when identifying areas in the captured image where moiré may occur. In this embodiment, a captured image acquired via the first imaging optical system 211 is transmitted to the control device 100, and the estimation unit 105 detects the subject area based on the image. After detecting the subject area in this way, the estimation unit 105 controls the area excluding the subject area in the captured image to be the target area when identifying areas in which moiré may occur. By doing this, even if the estimation unit 105 outputs an estimation result indicating that moire will occur, the filter unit 213 is disabled (not enabled) for the subject area as a result of control by the control unit 101.

[0050] In summary, the estimation unit 105 extracts, as the target region, a region that is included in the depth of field and in which the display device 300 appears, excluding the subject region. Then, the estimation unit 105 identifies, within the target region, a region in which the frequency indicated by the image of the pixel structure of the display device 300 falls within the range from the Nyquist frequency of the image sensor to the frequency of the resolution limit of the first imaging optical system 211, as a region in which moiré may occur.

[0051] Control Processing Specific control processing executed in relation to operational control of the filter unit 213 in the control device 100 of this embodiment will be described below with reference to the flowchart in Fig. 6. The processing corresponding to this flowchart can be realized by the control unit 101 reading out a corresponding processing program stored in, for example, the ROM 102, and loading and executing the program in the RAM 103. This control processing will be described as being started when, for example, an operation input related to video shooting is made in the imaging device 200 and processing related to video shooting is executed based on the operation input.

[0052] In S601, the control unit 101 acquires information on the imaging conditions currently set in the imaging device 200. In this embodiment, for example, it is assumed that the camera control unit 201 periodically outputs information on the imaging conditions set in the imaging device 200 to the control device 100 via the camera communication unit 204. It is also assumed that the information on the imaging conditions thus output is stored and held in the RAM 103 when it is acquired via the communication unit 108. Therefore, it is assumed that the RAM 103 stores information on the imaging conditions currently set in the imaging device 200. In this step, the control unit 101 acquires information on the imaging conditions currently set in the imaging device 200 by reading it from the RAM 103.

[0053] In S602 , the detection unit 104 detects the position and orientation of the image capturing device 200 under the control of the control unit 101 .

[0054] In S603, under the control of the control unit 101, the estimation unit 105 determines whether the display device 300 is included within the depth of field of the first imaging optical system 211. Specifically, the estimation unit 105 identifies the depth of field of the first imaging optical system 211 based on information about the position and orientation of the imaging device 200 detected in S602 and information about the imaging conditions. The estimation unit 105 also determines whether the display device 300 is included within the depth of field based on information about the layout of the display device 300 and information about the position and orientation of the imaging device 200 that are predetermined for the photography studio. At this time, if the display device 300 is included within the depth of field, the estimation unit 105 stores information about an area on the captured image (pixel position on the imaging element) within the depth of field where the image of the display device 300 appears, as information about the target area, in the RAM 103. If the estimation unit 105 determines that the display device 300 is included within the depth of field of the first imaging optical system 211, the process proceeds to S604; if it determines that the display device 300 is not included, the process proceeds to S609.

[0055] In S604, the estimation unit 105 identifies the subject area in the captured image under the control of the control unit 101. In this embodiment, similar to the information on the imaging conditions, it is assumed that the captured image acquired by the imaging device 200 using the first imaging optical system 211 is periodically output to the control device 100 via the camera communication unit 204. Therefore, the estimation unit 105 executes subject detection processing on the captured image acquired via the communication unit 108, thereby acquiring information on the subject area in the captured image.

[0056] In S605, the estimation unit 105 determines the target area under the control of the control unit 101. More specifically, the estimation unit 105 determines the target area to be used in the estimation process for determining whether or not moiré occurs by excluding the subject area identified in S604 from the target area identified in S603.

[0057] In S606, under the control of the control unit 101, the estimation unit 105 derives, for each of the imaging pixels of the first imaging unit 212 distributed in the target area, the frequency indicated by the image of the pixel structure of the display device 300 formed at that imaging pixel. Specifically, for each of the imaging pixels distributed in the target area, the estimation unit 105 acquires information on the distance between the imaging device 200 and a part of the display device 300 corresponding to the pixel position, the pixel pitch of the display device 300, and the focal length set in the first imaging optical system 211. Then, based on this information, the estimation unit 105 can derive the frequency indicated by the image of the pixel structure formed at that pixel position.

[0058] In S607, under the control of the control unit 101, the estimation unit 105 executes estimation processing to estimate whether or not moiré will occur at each of the imaging pixels distributed in the target area. Through this estimation processing, the estimation unit 105 outputs an estimation result indicating whether or not moiré will occur at each of the imaging pixels distributed in the target area. In the estimation processing of this embodiment, estimation is performed based on whether or not the frequency indicated by the image of the pixel structure of the display device 300 derived for each imaging pixel falls within a range from the Nyquist frequency of the imaging element of the first imaging unit 212 to the resolution limit frequency of the first imaging optical system 211. If the frequency indicated by the image of the pixel structure derived for each imaging pixel falls within the above range, the estimation unit 105 outputs an estimation result indicating that moiré will occur at the imaging pixel. Furthermore, if the frequency indicated by the image of the pixel structure derived for each imaging pixel does not fall within the above range, the estimation unit 105 outputs an estimation result indicating that moiré will not occur at the imaging pixel.

[0059] In S608, the control unit 101 controls the operation of the filter unit 213 based on the estimation result of the estimation process executed in S607. More specifically, for an imaging pixel for which an estimation result indicating that moiré will occur has been output, the control unit 101 transmits to the imaging device 200 via the communication unit 108 a control signal for enabling the filter unit 213 at the corresponding position. Furthermore, for an imaging pixel for which an estimation result indicating that moiré will not occur has been output, or an imaging pixel that has been determined to be outside the target region, the control unit 101 transmits to the imaging device 200 via the communication unit 108 a control signal for disabling the filter unit 213 at the corresponding position (region). Using this control signal, the camera control unit 201 controls the operation of the filter unit 213, thereby reducing and blurring the high-frequency components of the image formed at the imaging pixel estimated to have moiré occurring.

[0060] On the other hand, if it is determined in S603 that the display device 300 is not included within the depth of field of the first imaging optical system 211, the control unit 101 transmits in S609 to the imaging device 200 via the communication unit 108 a control signal for initializing the operation of the filter unit 213. In this embodiment, the filter unit 213 is configured to be disabled in the initial state (no filter is applied), and the camera control unit 201 disables the filter over the entire area of ​​the filter unit 213 in response to the control signal for initialization.

[0061] In S610, the control unit 101 determines whether or not video shooting by the imaging device 200 has ended. The determination in this step can be made based on whether or not the communication unit 108 has received information relating to the end of shooting that was sent in response to an operation input relating to the end of video shooting being made in the imaging device 200. If the control unit 101 determines that video shooting by the imaging device 200 has ended, it completes this control process, and if it determines that video shooting has not ended, it returns the process to S601.

[0062] As described above, the control device of this embodiment can cause the imaging device 200 to perform imaging in which the influence of moiré is adaptively reduced. More specifically, the control device 100 can control the filter unit 213 to be enabled in an area where it is estimated that moiré may occur, and therefore the imaging device 200 can perform imaging in which degradation of image quality due to moiré is avoided.

[0063] In this embodiment, in order to facilitate understanding of the invention, it has been described that the control device 100 detects the position and orientation of the image capture device 200, generates and controls the display of a background image, and controls the operation of the filter unit 213; however, the implementation of the present invention is not limited to this. In implementing the present invention, the control device 100 only needs to estimate whether or not moiré will occur in a captured image output by the image capture device 200 based on information about the image capture conditions, and control the filter characteristics of the filter unit 213 based on the estimation result. Therefore, the detection of the position and orientation of the image capture device 200 and the generation and display control of a background image may be performed by an external device.

[0064] [Variation 1] In the above-described embodiment, the filter unit 213 is switched between enabled and disabled for each region in the captured image based on the estimation result by the estimation unit 105. That is, in the first embodiment, a filter with fixed filter characteristics is uniformly applied to a region where it is estimated that moiré will occur, but the present invention is not limited to this.

[0065] As described above, the frequency indicated by the image of the pixel structure of the display device 300 in the imaging element may vary from region to region depending on the distance between the display device 300 and the imaging device 200. In other words, when a filter with fixed filter characteristics is uniformly applied, depending on the arrangement of the imaging device 200 and the display device 300, the background image in some regions may be too blurred, resulting in an image that gives the viewer an unnatural impression. For this reason, in an aspect in which the filter characteristics of the filter unit 213 can be changed for each region, the filter characteristics of the filter unit 213 applied to each region may be controlled to be adaptively changed in accordance with the frequency indicated by the image of the pixel structure formed in that region.

[0066] The filter unit 213 with variable filter characteristics can be realized, for example, as shown in FIG. 7 , by a multi-stage hierarchical structure in which multiple optical variable low-pass filters, which can be switched between enabled and disabled on a region-by-region basis, are arranged in the optical axis direction. In this case, the control unit 101 can change the filter characteristics for a region where moiré is estimated to occur by controlling which optical variable low-pass filter layer to enable depending on the frequency indicated by the image of the pixel structure associated with that region. In other words, the filter thickness can be changed depending on the number of optical variable low-pass filters to be enabled, allowing the filter characteristics to be adaptively changed. By using a multi-stage filter structure for the filter unit 213 in this way, the frequency bands to be cut can be varied on a region-by-region basis, making it possible to control the operation of the imaging device 200 so that video with desirable image quality is recorded while suppressing the occurrence of moiré.

[0067] It goes without saying that the filter section 213 capable of changing the filter characteristics in units of regions is not limited to an optical variable low-pass filter, and various other filters can be used.

[0068] [Variation 2] In the above-described embodiment, the filter unit 213 is enabled on the condition that the frequency indicated by the image of the pixel structure of the display device 300 is within a range from the Nyquist frequency of the image sensor to the frequency of the resolution limit of the first imaging optical system 211. However, the implementation of the present invention is not limited to this. For example, for any region, the filter may be controlled to be enabled on the condition that the frequency indicated by the image of the pixel structure formed in that region exceeds the Nyquist frequency of the image sensor. Alternatively, in a situation where the arrangement conditions of the image sensor 200 or the like ensure that the frequency indicated by the image of the pixel structure of the display device 300 exceeds the Nyquist frequency of the image sensor, the filter unit 213 may be controlled based on a comparison with the frequency of the resolution limit of the first imaging optical system 211. In this case, for example, for any region, the filter may be controlled to be enabled on the condition that the frequency indicated by the image of the pixel structure formed in that region is below the frequency of the resolution limit of the first imaging optical system 211.

[0069] [Variation 3] In the above-described embodiment and modified example, an aspect has been described in which the filter unit 213 is capable of changing the filter characteristics (enabling / disabling) on ​​a region-by-region basis, but the implementation of the present invention is not limited to this. Enabling a filter only in a region where moiré may occur is preferable from the viewpoint of ensuring the sharpness of the entire recorded image, but a configuration that allows the filter characteristics to be changed on a region-by-region basis may increase the introduction cost of the filter unit 213 and the amount of calculation required for control. Therefore, the filter unit 213 does not necessarily have to be configured to be capable of changing the filter characteristics on a region-by-region basis, and may be configured to apply a filter to the entire image regardless of region when, for example, an estimation result indicating the occurrence of moiré is obtained.

[0070] In this case, the filter unit 213 can change its filter characteristics based on the frequency indicated by the image of the pixel structure derived for the region where moiré may occur, thereby appropriately avoiding the occurrence of moiré depending on the state of the imaging device 200. One example of such a filter unit 213 is a sensor (imaging element) driven low-pass filter. The sensor driven low-pass filter can blur the light beam incident on each region via the first imaging optical system 211 by translating the imaging element of the first imaging unit 212 in a plane perpendicular to the optical axis, thereby forming an image on different imaging pixels in a time-division manner. That is, in an example where a sensor driven low-pass filter is used as the filter unit 213, the filter unit 213 is not provided between the first imaging optical system 211 and the first imaging unit 212, but is implemented as a mechanism for driving the first imaging unit 212. Note that when a sensor driven low-pass filter is used, control of the filter characteristics depending on the region is not performed, and therefore it is not necessary to perform the process of estimating whether or not moiré will occur for each imaging pixel, as was performed in the control process of the first embodiment described above.

[0071] [Variation 4] In the above-described embodiment, it has been described that the placement information of the display device 300 is predetermined, and the distance between the image capture device 200 and the display device 300 is derived based on the detected position and orientation information of the image capture device 200. However, the present invention is not limited to this. For example, in an embodiment in which the image capture device 200 includes a distance sensor, or the image sensor of the first image capture unit 212 records multiple pupil-split images and is configured to perform distance measurement using an image capture plane phase difference ranging method, information indicating the depthwise distance distribution obtained by distance measurement can also be used. In this embodiment, distance information is periodically output from the image capture device 200, and the estimation unit 105 can derive the frequency indicated by the pixel structure of the display device 300 imaged on the first image capture unit 212 by referring to the distance information.

[0072] [Variation 5] In the above-described embodiment and modified example, a detection image obtained by capturing a viewpoint detection marker 303 provided in a photography studio using the second imaging optical system 221 is used to detect the position and orientation of the image capture device 200. However, it will be readily understood that other methods can be adopted for detecting the position and orientation of the image capture device 200. That is, in implementing the present invention, it is not essential to install the viewpoint detection marker 303 or for the image capture device 200 to have an imaging system including the second imaging optical system 221.

[0073] [Embodiment 2] In the above-described embodiment and modified example, a configuration has been described in which the control device 100 provided outside the imaging device 200 outputs a control signal related to the operation of the filter unit 213, and the camera control unit 201 controls the filter characteristics of the filter unit 213 based on the control signal. However, the implementation of the present invention is not limited to this, and, for example, the estimation unit 105 may be provided as a functional configuration of the imaging device 200. That is, in a configuration in which the imaging device 200 is configured to be able to acquire information such as the pixel pitch of the display device 300, an estimation unit provided in the imaging device 200 can estimate whether or not moiré will occur in the captured image, and control the filter unit 213 based on the estimation result.

[0074] [Variation 6] In the above-described first embodiment, the estimation unit 105 estimates that moiré will not occur in a corresponding region when the frequency indicated by the image of the pixel structure of the display device 300 exceeds the frequency of the resolution limit of the first imaging optical system 211. However, the implementation of the present invention is not limited to this. For example, for each of the first imaging optical systems 211 that can be attached to the imaging device 200, a frequency threshold at which moiré will not occur may be identified in advance through experiments or the like, and this information may be used for estimation. That is, in S607 of the control process, the estimation unit 105 may output an estimation result indicating that moiré will not occur in a corresponding region on the condition that the frequency indicated by the image of the pixel structure of the display device 300 exceeds the threshold determined for the first imaging optical system 211.

[0075] [Variation 7] In the above-described embodiment and modified examples, in order to facilitate understanding of the invention, the image capturing system of the first image capturing optical system 211 has been described as having only the filter unit 213 as a filter, but the implementation of the present invention is not limited to this. For example, in an aspect in which another filter is incorporated in the first image capturing unit 212, the frequency of the resolution limit determined by integrating the components included in the image capturing system other than the filter unit 213 may be referred to in the control processing, rather than the resolution limit of the first image capturing optical system 211 itself.

[0076] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0077] [Summary of the embodiment and modifications] The disclosure of this specification includes the following imaging system, control device, imaging device, control method, and program. (Item 1) An imaging system including an imaging device that captures an image of a display device having a pixel structure, and a control device that controls an operation of the imaging device, The imaging device is An imaging means; a filter means configured to change filter characteristics and applied to the imaging means; and The control device an acquisition means for acquiring information on imaging conditions of the imaging device; an estimation means for estimating whether or not moiré will occur in the captured image output by the imaging means based on the information on the imaging conditions, and outputting an estimation result; a control means for changing the filter characteristics of the filter means in accordance with the estimation result; have An imaging system characterized by: (Item 2) the imaging means includes an imaging optical system and an imaging element; The estimation means estimates whether or not moire occurs in the captured image based on a frequency indicated by an image of the pixel structure of the display device formed on the image sensor. 2. The imaging system according to item 1, (Item 3) The estimation means outputs the estimation result indicating that moire will occur in the captured image, on the condition that the frequency indicated by the image of the pixel structure exceeds the Nyquist frequency of the image sensor. 3. The imaging system according to item 2, (Item 4) The estimation means outputs the estimation result indicating that moire will occur in the captured image, on condition that the frequency indicated by the image of the pixel structure is lower than the frequency of the resolution limit of the imaging optical system. 3. The imaging system according to item 2, (Item 5) The estimation means outputs the estimation result indicating that moire will occur in the captured image, on condition that the frequency indicated by the image of the pixel structure is higher than the Nyquist frequency of the image sensor and lower than the frequency of the resolution limit of the imaging optical system. 3. The imaging system according to item 2, (Item 6) the imaging means is configured to be able to mount a plurality of types of imaging optical systems; The frequency of the resolution limit of the imaging optical system is predetermined for each of the plurality of types of imaging optical systems. 6. The imaging system according to item 4 or 5, (Item 7) the information on the imaging conditions includes information indicating a relative position of the display device with respect to the imaging device and information on a focal length of the imaging optical system, The estimation means derives a frequency indicated by an image of the pixel structure based on information on the distance between display pixels constituting the display device and the imaging conditions. 7. The imaging system according to any one of items 2 to 6, (Item 8) The control means controls the filter characteristics of the filter means so as to filter frequencies represented by the image of the pixel structure. 8. The imaging system according to any one of items 2 to 7, wherein: (Item 9) the filter means is configured to be able to change filter characteristics in units of regions of the image sensor, the estimation means includes a specification means for specifying an area in the captured image where moiré is estimated to occur, The control means enables application of the filtering means to the region identified by the identifying means. 9. The imaging system according to any one of items 2 to 8, wherein: (Item 10) the information on the imaging conditions includes information indicating a relative position of the display device with respect to the imaging device, and information on an aperture value and a focal length of the imaging optical system; The estimation means outputs the estimation result indicating that moire will occur in the captured image, on condition that the display device is included in the depth of field of the imaging optical system. 10. The imaging system according to item 9, (Item 11) The specifying means specifies an area in which moire is expected to occur from among areas in the captured image in which the image of the display device included in the depth of field is distributed. 11. The imaging system according to item 10. (Item 12) the control device further includes a detection means for detecting a subject area in which subjects are distributed in the captured image; The control means disables application of the filter means to the subject region. 12. The imaging system according to any one of items 9 to 11, (Item 13) The estimation means specifies an area in the captured image where moire is estimated to occur, excluding the subject area. Item 13. The imaging system according to item 12. (Item 14) A control device for controlling an operation of an imaging device that captures an image of a display device having a pixel structure, The imaging device is An imaging means; a filter means configured to change filter characteristics and applied to the imaging means; and The control device an acquisition means for acquiring information on imaging conditions of the imaging device; an estimation means for estimating whether or not moiré will occur in the captured image output by the imaging means based on the information on the imaging conditions, and outputting an estimation result; a control means for changing the filter characteristics of the filter means in accordance with the estimation result; have A control device characterized by: (Item 15) An imaging device for imaging a display device having a pixel structure, An imaging means; a filter means configured to change filter characteristics and applied to the imaging means; an acquisition means for acquiring information on imaging conditions of the imaging device; an estimation means for estimating whether or not moiré will occur in the captured image output by the imaging means based on the information on the imaging conditions, and outputting an estimation result; a control means for changing the filter characteristics of the filter means in accordance with the estimation result; have An imaging device characterized by: (Item 16) A control method for a control device that controls an operation of an imaging device that captures an image of a display device having a pixel structure, comprising: The imaging device is An imaging means; a filter means configured to change filter characteristics and applied to the imaging means; and The control method includes: an acquisition step of acquiring information on imaging conditions of the imaging device; an estimation step of estimating whether or not moiré occurs in the captured image output by the imaging means based on the information on the imaging conditions and outputting an estimation result; a control step of changing the filter characteristics of the filter means in accordance with the estimation result; have A control method comprising: (Item 17) A program for causing a computer to function as each means of the control device of the imaging system according to any one of items 1 to 13.

[0078] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0079] 100: control device, 200: imaging device, 300: display device, 101: control unit, 105: estimation unit, 108: communication unit, 201: camera control unit, 204: camera communication unit, 211: first imaging optical system, 212: first imaging unit, 213: filter unit

Claims

1. An imaging system including an imaging device that captures an image of a display device having a pixel structure, and a control device that controls an operation of the imaging device, The imaging device is An imaging means; a filter means configured to change filter characteristics and applied to the imaging means; and The control device an acquisition means for acquiring information on imaging conditions of the imaging device; an estimation means for estimating whether or not moiré will occur in the captured image output by the imaging means based on the information on the imaging conditions, and outputting an estimation result; a control means for changing the filter characteristics of the filter means in accordance with the estimation result; have An imaging system characterized by:

2. the imaging means includes an imaging optical system and an imaging element; The estimation means estimates whether or not moire occurs in the captured image based on a frequency indicated by an image of the pixel structure of the display device formed on the image sensor.

2. The imaging system according to claim 1, wherein:

3. The estimation means outputs the estimation result indicating that moire will occur in the captured image on condition that the frequency indicated by the image of the pixel structure exceeds the Nyquist frequency of the image sensor.

3. The imaging system according to claim 2.

4. The estimation means outputs the estimation result indicating that moire will occur in the captured image, on condition that the frequency indicated by the image of the pixel structure is lower than the frequency of the resolution limit of the imaging optical system.

3. The imaging system according to claim 2.

5. The estimation means outputs the estimation result indicating that moire will occur in the captured image, on condition that the frequency indicated by the image of the pixel structure exceeds the Nyquist frequency of the image sensor and is lower than the frequency of the resolution limit of the imaging optical system.

3. The imaging system according to claim 2.

6. the imaging means is configured to be able to mount a plurality of types of imaging optical systems; The frequency of the resolution limit of the imaging optical system is predetermined for each of the plurality of types of imaging optical systems.

5. The imaging system according to claim 4.

7. the information on the imaging conditions includes information indicating a relative position of the display device with respect to the imaging device and information on a focal length of the imaging optical system, The estimation means derives a frequency indicated by an image of the pixel structure based on information on the distance between display pixels constituting the display device and the imaging conditions.

3. The imaging system according to claim 2.

8. The control means controls the filter characteristics of the filter means so as to filter frequencies represented by the image of the pixel structure.

3. The imaging system according to claim 2.

9. the filter means is configured to be able to change filter characteristics in units of regions of the image sensor, the estimation means includes a specification means for specifying an area in the captured image where moiré is estimated to occur, The control means enables application of the filtering means to the region identified by the identifying means.

3. The imaging system according to claim 2.

10. the information on the imaging conditions includes information indicating a relative position of the display device with respect to the imaging device, and information on an aperture value and a focal length of the imaging optical system; The estimation means outputs the estimation result indicating that moire will occur in the captured image, on condition that the display device is included in the depth of field of the imaging optical system.

10. The imaging system according to claim 9.

11. The specifying means specifies an area in which moire is expected to occur from among areas in the captured image in which the image of the display device included in the depth of field is distributed. The imaging system according to claim 10 .

12. the control device further includes a detection means for detecting a subject area in which subjects are distributed in the captured image; The control means disables application of the filter means to the subject region.

10. The imaging system according to claim 9.

13. The estimation means specifies an area in the captured image where moire is estimated to occur, excluding the subject area.

13. The imaging system according to claim 12.

14. A control device for controlling an operation of an imaging device that captures an image of a display device having a pixel structure, The imaging device is An imaging means; a filter means configured to change filter characteristics and applied to the imaging means; and The control device an acquisition means for acquiring information on imaging conditions of the imaging device; an estimation means for estimating whether or not moiré will occur in the captured image output by the imaging means based on the information on the imaging conditions, and outputting an estimation result; a control means for changing the filter characteristics of the filter means in accordance with the estimation result; have A control device characterized by:

15. An imaging device for imaging a display device having a pixel structure, An imaging means; a filter means configured to change filter characteristics and applied to the imaging means; an acquisition means for acquiring information on imaging conditions of the imaging device; an estimation means for estimating whether or not moiré will occur in the captured image output by the imaging means based on the information on the imaging conditions, and outputting an estimation result; a control means for changing the filter characteristics of the filter means in accordance with the estimation result; have An imaging device characterized by:

16. A control method for a control device that controls an operation of an imaging device that captures an image of a display device having a pixel structure, comprising: The imaging device is An imaging means; a filter means configured to change filter characteristics and applied to the imaging means; and The control method includes: an acquisition step of acquiring information on imaging conditions of the imaging device; an estimation step of estimating whether or not moiré occurs in the captured image output by the imaging means based on the information on the imaging conditions and outputting an estimation result; a control step of changing the filter characteristics of the filter means in accordance with the estimation result; have A control method comprising:

17. A program for causing a computer to function as each means of the control device of the imaging system according to any one of claims 1 to 13.

Images