Information processing device

Abstract

To provide a technology that makes it possible to obtain a more natural image (an image with less sense of incongruity) as a composite image obtained by combining a real subject with a graphic.SOLUTION: An information processing device according to the present invention includes acquisition means for acquiring the aperture value of the lens of an imaging device, and output means for outputting, together with the image captured by the imaging device, the aperture value at the time of capturing the image and blur shape related information regarding the shape of the blur that occurs in the captured image.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an information processing device, and more particularly to a technique for combining a real subject (a real object) with a graphic (a CG (Computer Graphics) image). [Background technology]

[0002] In recent years, there has been a growing demand for video content that uses visual effects technology to combine real subjects with graphics (CG images). Filming video content that uses visual effects technology is generally referred to as VFX filming. There are two main methods for VFX filming. The first is a post-production method in which filming is performed against a special background, such as a green screen, and then graphics are composited into the background of the filmed image in post-production. The second is an in-camera method in which filming is performed against a background of graphics displayed on a large display device. VFX filming uses CG to create realistic images without having to travel to the actual location, and to capture images with angles and compositions that would be difficult to achieve in reality, thereby reducing production costs. This has led to a growing demand for VFX filming. The desire for VFX filming is to produce natural composite images (composite images that do not look unnatural).

[0003] Patent Document 1 discloses a technique for changing the composite position of a graphic depending on the camera orientation. Patent Document 2 discloses a technique for generating a point spread function based on the distance from the entrance pupil of the lens and the size of the exit pupil, and generating a graphic based on the point spread function. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-227332 [Patent Document 2] Special Publication No. 2021-532649 Summary of the Invention [Problem to be solved by the invention]

[0005] While cameras with 2 / 3-inch or 1-inch image sensors were mainstream in the past, in recent years cameras with larger image sensors, such as Super 35mm and full-frame 35mm, have been increasing. As image sensors become larger, the depth of field that can be set becomes wider, increasing the degree of freedom in expressing blur. Important factors in blur include its intensity (degree of spread), which can be expressed by the point spread function, and its shape. Conventional VFX filming does not generate graphics that take into account the shape of the blur that occurs in the captured image, making it difficult to obtain natural-looking composite images.

[0006] An object of the present invention is to provide a technique that makes it possible to obtain a more natural image (an image that looks less unnatural) as a composite image that combines a real subject with a graphic. [Means for solving the problem]

[0007] A first aspect of the present invention is an information processing device characterized by having an acquisition means for acquiring an aperture value of a lens of an imaging device, and an output means for outputting, together with an image captured by the imaging device, the aperture value at the time of capturing the image and blur shape related information regarding the shape of blur that occurs in the captured image.

[0008] A second aspect of the present invention is an acquisition means for acquiring the aperture value of a lens of an imaging device in real time. and output means for outputting in real time the aperture value and blur shape related information relating to the shape of blur occurring in an image captured by the imaging device.

[0009] A third aspect of the present invention is an information processing method comprising the steps of acquiring an aperture value of a lens of an imaging device, and outputting, together with an image captured by the imaging device, the aperture value at the time of capturing the image and blur shape related information regarding the shape of blur occurring in the captured image.

[0010] A fourth aspect of the present invention is an information processing method comprising the steps of: acquiring an aperture value of a lens of an imaging device in real time; and outputting, in real time, the aperture value and blur shape related information relating to the shape of blur that occurs in an image captured by the imaging device.

[0011] A fifth aspect of the present invention is a program for causing a computer to function as each of the means of the information processing device.A sixth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device. [Effects of the Invention]

[0012] According to the present invention, it is possible to obtain a more natural image (an image that looks less unnatural) as a composite image obtained by combining a real subject and a graphic. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram of a camera according to a first embodiment. [Figure 2] FIG. 1 is a schematic diagram of the filming location. [Figure 3] FIG. 2 is a schematic diagram of a composite image. [Figure 4] FIG. 1 is a schematic diagram of the filming location. [Figure 5] FIG. 2 is a schematic diagram of a composite image. [Figure 6] FIG. [Figure 7] FIG. 10 is a schematic diagram of a blurred shape. [Figure 8] FIG. 2 is a schematic diagram of correspondence relationship information. [Figure 9]FIG. 10 is a schematic diagram of a blurred shape. [Figure 10] FIG. 2 is a schematic diagram of correspondence relationship information. [Figure 11] FIG. 2 is a schematic diagram of communication between a camera body and a lens unit. [Figure 12] FIG. 2 is a schematic diagram of communication between a camera body and a lens unit. [Figure 13] FIG. 2 is a schematic diagram of communication between a camera body and a lens unit. [Figure 14] FIG. 1 is a schematic diagram of communication between the camera body and the CG generation device. [Figure 15] FIG. 1 is a schematic diagram of communication between a lens unit, a camera body, and a CG generation device. [Figure 16] FIG. 10 is a block diagram of a camera according to a second embodiment. [Figure 17] FIG. 10 is a block diagram of a lens unit according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] First Embodiment A first embodiment of the present invention will now be described. Fig. 1 is a block diagram showing the configuration of a camera body 100 as an example of an information processing device according to the first embodiment. The camera body 100 is an imaging device in which the lens unit (lens, lens device) is interchangeable. In Fig. 1, a lens unit 200 is attached to the camera body 100.

[0015] The camera body 100 includes a memory 101, a CPU (Central Processing Unit), The image sensor 102 includes an image sensor, an image sensor 103, a communication terminal 104, an output terminal 105, and a recording medium 106.

[0016] The memory 101 is a storage unit that stores various data (various information) including images. The data stored in the memory 101 can be read from the memory 101 when necessary. The memory 101 has a volatile area that can hold data only while power is applied, and a non-volatile area that can hold data even when power is cut off.

[0017] CPU 102 is a control unit that controls each part of camera body 100 and each part of an accessory (lens unit 200 in FIG. 1) attached to camera body 100. For example, programs and various parameters for operating CPU 102 are stored in memory 101, and CPU 102 performs various controls by reading the programs from a non-volatile area of ​​memory 101, expanding them into a volatile area, and executing them.

[0018] The image sensor 103 is a charge-storage solid-state image sensor such as a CMOS or CCD, which receives a light beam (light beam from a subject) guided into the camera body 100 via the lens unit 200 and converts it into an electrical image signal. The image (signal) obtained by the image sensor 103 is used for live view display, recording to a recording medium 106 (described later), external output using an output terminal 105, and the like, under the control of the CPU 102. The CPU 102 can also control the exposure time of the image sensor 103, the timing of shooting (the timing of shooting), and the like.

[0019] The communication terminal 104 is a terminal for communicating with the lens unit attached to the camera body 100 (lens unit 200 in FIG. 1).

[0020] The output terminal 105 is, for example, an Ethernet terminal, an SDI terminal, an HDMI (registered trademark) terminal, etc., and is used to output various data (various information) including images to the outside, and to acquire various data (various information) from the outside.

[0021] The recording medium 106 is a recording medium that is detachable from the camera body 100, such as an SD card or CFExpress. The recording medium 106 is capable of recording various types of data (various types of information) including images.

[0022] The lens unit 200 includes a memory 201, an LPU (Lens Processing Unit) 202, a communication terminal 203, an aperture 204, and a lens group 205. The lens unit 200 is a so-called interchangeable lens that is detachable from the imaging device.

[0023] The memory 201 is a storage unit that stores various data (various information). The data stored in the memory 201 can be read from the memory 201 when necessary. The memory 201 has a volatile area that can hold data only while power is applied, and a non-volatile area that can hold data even when power is cut off.

[0024] The LPU 202 is a control unit that controls each part of the lens unit 200. For example, programs and various parameters for operating the LPU 202 are stored in the memory 201, and the LPU 202 performs various controls by reading the programs from the nonvolatile area of ​​the memory 201, expanding them into the volatile area, and executing them.

[0025] The communication terminal 203 is a terminal for communicating with the imaging device (camera body 100 in FIG. 1) to which the lens unit 200 is attached. In FIG. 1, the LPU 202 of the lens unit 200 and the CPU 102 of the camera body 100 are connected to each other via the communication terminal 203 of the lens unit 200 and the communication terminal 104 of the camera body 100. The LPU 202 is Each part of the lens unit 200 can be driven (controlled) in response to a control instruction from U102.

[0026] The diaphragm 204 is a light amount control member that controls (adjusts) the amount of light of the light beam guided into the camera body 100. For example, by changing the aperture value, the aperture diameter of the diaphragm 204 is changed to an aperture diameter corresponding to the changed aperture value, and the amount of light is changed. In addition to the amount of light, the depth of field and blur can also be changed.

[0027] The lens group 205 includes a focus lens, a zoom lens, a shift lens, etc. A light beam from the subject passes through the lens group 205 and the aperture 204 and is guided into the camera body 100. The position of each lens included in the lens group 205 can be controlled (changed). For example, the focus on the subject can be adjusted by moving the focusing lens in the optical axis direction.

[0028] Camera body 100 (and lens unit 200) is used, for example, in VFX photography, which combines a real subject with graphics (CG images).

[0029] Post-production VFX shooting will be explained using Figures 2 and 3. Figure 2 is a schematic diagram showing an example of a shooting location, and Figure 3 is a schematic diagram showing an example of a composite image in which a real subject and a graphic are combined.

[0030] In post-production VFX photography, photography is first performed against a special background. In the example of Fig. 2, photography of a real subject 301 is performed against a green screen 300. Note that the background is not limited to a green screen (a background of a single green color), and may be, for example, a background of another single color or a background of another pattern.

[0031] Next, the CG compositing device imports the photographed image into CG compositing editing software, detects a background area from the photographed image, and composites the background graphic into the detected area. Graphics other than the background can also be composited. In the example of FIG. 3, the background graphic 400 is composited with non-background graphics 402 to 404. The non-background graphics 402 to 404 are objects placed in front of the background graphic 400.

[0032] Information such as the focus position at the time of shooting is recorded as metadata for the captured image, and the blur of the graphics 400, 402 to 405 is individually controlled (adjusted) based on this information. For example, the blur of the background graphic 400 is made different from the blur of the non-background graphics 402 to 404. By generating and combining the graphics 400, 402 to 405 so that the blur of the real subject 401 (301) matches the blur of the graphics 400, 402 to 405, a natural composite image (a natural composite image) can be generated.

[0033] The in-camera VFX shooting will be explained using Figures 4 and 5. Figure 4 is a schematic diagram showing an example of a shooting location, and Figure 5 is a schematic diagram showing an example of a composite image in which a real subject and a graphic are combined.

[0034] In in-camera VFX shooting, shooting is performed with a graphic displayed on a large display device as the background. In the example of FIG. 4, shooting of real subjects 501 to 504 is performed with a graphic displayed on display device 500 as the background. At this time, information such as the focus position is output from the camera in real time (successively during shooting) and input to a CG generation device. The CG generation device generates a graphic with a blur based on the input information as the graphic to be displayed on display device 500. The camera shoots with an angle of view that includes the graphic displayed on display device 500 and real subjects 501 to 504, resulting in the composite image of FIG. It should be noted that information such as the focus position may be transmitted from the camera body to the CG generation device, or may be transmitted from the lens unit to the CG generation device without going through the camera body. Information such as the focus position may also be transmitted from the lens unit to the CG generation device via a device different from the camera body (for example, a device that converts information obtained from an external source into information that can be input to the CG generation device).

[0035] Let's talk about bokeh. Important factors in bokeh are its intensity (extent) and shape. In conventional VFX photography, the intensity of the blur in graphics is controlled (adjusted) based on the aperture value, focus position, etc., but the graphics generated do not take into account the shape of the blur that occurs in the photographed image, and natural-looking composite images cannot be obtained.

[0036] In a captured image, out-of-focus subjects are blurred. The intensity of the blur is determined by the degree of out-of-focus, such as the amount of deviation from the depth of field based on the aperture value and the size of the image sensor, and the amount of deviation from the subject distance corresponding to the in-focus position (focus position). On the other hand, the shape of the blur is not determined by the aperture value, image sensor, in-focus position, etc., but by the shape of the aperture opening.

[0037] FIG. 6 is a schematic diagram showing an example of a circular aperture and a polygonal aperture (iris aperture) with a polygonal aperture. FIG. 6 shows an example of a polygonal aperture with an octagonal aperture. FIG. 7 is a schematic diagram showing an example of the shape of blur that occurs in a captured image. A circular aperture generates circular blur, while a polygonal aperture generates polygonal blur. In the case of a polygonal aperture, if the number of aperture blades is eight, octagonal blur occurs. When the edges of the aperture blades that form the aperture opening have curvature, polygonal blur that is close to a circle rather than a regular polygon may occur. Even in the case of a polygonal aperture, if the aperture opening becomes circular when fully open, circular blur occurs. As such, the shape of blur depends on the specifications of the lens unit (aperture) and the aperture value. For simplicity, the blur shape when an ideal point light source is photographed has been described here. However, blur of a shape similar to the above-described shape also occurs in areas of a subject other than a point light source.

[0038] If information capable of identifying (determining) the blur shape cannot be communicated to a CG generating device or a CG compositing device, the blur shape will not match between the generated graphic and the actual subject, resulting in the generation of an unnatural composite image (an unnatural composite image). Therefore, in the first embodiment, the camera body 100 outputs blur shape-related information, which is information regarding the blur shape, from the output terminal 105 to the outside or records it on the recording medium 106 so that the CG generating device or the CG compositing device can acquire it. In the case of a post-production method, an aperture value is acquired and, together with the captured image, blur shape-related information regarding the aperture value at the time of capture of the captured image and the shape of the blur that appears in the captured image is output. This processing may or may not be performed in real time. In the case of an in-camera method, an aperture value is acquired in real time, and the aperture value and blur shape-related information are output in real time. The output aperture value is used to determine the blur intensity of the graphic, and the output blur shape-related information is used to determine the blur shape of the graphic. If the blur shape-related information does not indicate the blur shape, the aperture value may be used to determine the blur shape.

[0039] A method for generating blur shape related information will be described. In the first embodiment, it is assumed that the blur shape related information indicates a blur shape. As described above, the blur shape depends on the specifications of the lens unit (aperture) and the aperture value. Therefore, if the lens unit to be used is determined in advance, the blur shape related information can be generated (acquired) based only on the aperture value. In the first embodiment, the memory 101 stores correspondence information indicating the correspondence between the aperture value and the blur shape related information. Then, the CPU 102 acquires and outputs the blur shape related information corresponding to the aperture value to be output based on the correspondence information.

[0040] FIG. 8 is a schematic diagram showing an example of correspondence information. Tables 1 to 3 are shown as the correspondence information. In Table 1, blur shape-related information indicates a blur shape, and multiple combinations of aperture value and blur shape are described. Table 1 can be used to determine the blur shape from the aperture value. Note that the aperture value described in the table may be only a representative value, and blur shape-related information corresponding to an aperture value not described in the table may be obtained by interpolation processing using the information described in the table. In Table 2, the blur-related information indicates a blur shape and circularity (the similarity of the blur shape to a perfect circle), and multiple combinations of aperture value, blur shape, and circularity are described. Table 2 can be used to determine a more accurate blur shape when the circularity changes depending on the aperture value. Table 3 shows a threshold value, which is the aperture value at which the resulting blur switches between circular blur and polygonal blur, and the shape of the blur (polygon) that occurs when the aperture value is smaller than the threshold value (on the aperture side). By using Table 3, it can be determined that circular blur occurs when the aperture value is larger than the threshold (on the open side), and that polygonal blur occurs when the aperture value is smaller than the threshold (on the closed side).The threshold may be the lower limit of the aperture value at which circular blur occurs, or the upper limit of the aperture value at which polygonal blur occurs.

[0041] An example of a special blur shape is the blur shape that occurs when an anamorphic lens is used. When an anamorphic lens is used, a captured image is obtained in which the subject is compressed horizontally. By stretching the captured image horizontally in post-production, an image is generated that has a wider angle of view than a normal captured image. In the captured image, the subject is compressed horizontally, and the blur is also compressed horizontally. Figure 9 is a schematic diagram showing an example of the shape of blur that occurs in an image captured using an anamorphic lens. Circular blur becomes elliptical blur, and regular octagonal blur becomes vertically elongated octagonal blur.

[0042] FIG. 10 is a schematic diagram showing an example of correspondence information that takes into account special blur shapes. Tables 4 to 6 are shown as the correspondence information. In Table 4, the blur shape-related information indicates the blur shape, and multiple combinations of aperture value and blur shape are described. An ellipse is shown as the blur shape at full aperture, and a vertically elongated hexagon is shown as another blur shape. In Table 5, the blur-related information indicates the blur shape and flattening ratio (the degree of squashing of the blur shape relative to a perfect circle or regular polygon), and multiple combinations of aperture value, blur shape, and flattening ratio are described. The degree of squashing is not limited to flattening ratio, and may be, for example, eccentricity. Table 6 shows the threshold value, which is the aperture value at which the resulting blur switches between elliptical blur and vertically elongated polygonal blur, the shape of blur (vertical elongated polygon) that occurs when the aperture value is smaller than the threshold value (on the stopped-down side), and the flattening ratio common to elliptical blur and vertically elongated polygonal blur. By using Table 3, it can be determined that circular blur occurs when the aperture value is larger than the threshold (on the open side), and that polygonal blur occurs when the aperture value is smaller than the threshold (on the closed side).The threshold may be the lower limit of the aperture value at which circular blur occurs, or the upper limit of the aperture value at which polygonal blur occurs.

[0043] Although an example has been given in which the correspondence information is a table, the correspondence information may be any information that can generate (acquire) blur shape related information from the aperture value, and may be, for example, a function. Also, an example has been given in which the blur shape related information indicates the blur shape, but the blur shape related information may be any information that can determine the shape of the blur from the aperture value, and may indicate, for example, the number of aperture blades, the shape of the aperture blades, the type of aperture (circular aperture / polygonal aperture), etc. In order to determine the blur shape from these pieces of information, it is also sufficient to have the correspondence information as a table.

[0044] The timing for acquiring blur shape related information will be described with reference to Figures 11 and 12. Figures 11 and 12 are schematic diagrams showing communication between the camera body 100 (communication terminal 104) and the lens unit 200 (communication terminal 203).

[0045] 11, a plurality of pieces of correspondence relationship information corresponding to a plurality of lens units are pre-stored in a non-volatile area of ​​memory 101. When lens unit 200 is attached to camera body 100, CPU 102 requests identification information of lens unit 200 from LPU 202. When power is applied to lens unit 200, LPU 202 reads the identification information of lens unit 200 from the non-volatile area of ​​memory 201 in response to the request from CPU 102 and transmits it to CPU 102. The identification information of lens unit 200 may be any information that can identify lens unit 200, such as a name, model number, or serial number.

[0046] Although the identification information is transmitted from lens unit 200 to camera body 100 in the above description, the identification information may instead be generated within camera body 100. For example, camera body 100 may be provided with a plurality of identification buttons that can be pressed by the lens units, and a plurality of pieces of identification information corresponding to the plurality of lens units may be stored in advance in a non-volatile area of ​​memory 101. CPU 102 may then obtain one of the plurality of pieces of identification information from memory 101 depending on the press states of the plurality of identification buttons.

[0047] The timing for acquiring the identification information is not limited to the above timing. For example, the identification information may be deleted from the memory 101 upon transition to the power-saving state, and may be acquired again upon recovery from the power-saving state.

[0048] When the identification information is acquired, the CPU 102 acquires the correspondence relationship information corresponding to the lens unit 200 from the memory 101 based on the identification information.

[0049] Thereafter, the CPU 102 requests the LPU 202 for the aperture value, and the LPU 202 transmits the aperture value to the CPU 102 in response to the request. The CPU 102 acquires blur shape related information corresponding to the acquired aperture value based on the correspondence information. The transmission and reception of the aperture value and the acquisition of the blur shape related information are repeatedly performed. These processes may be performed for each frame, or may be performed only when there is a change in the aperture value.

[0050] 12, correspondence relationship information for the lens unit 200 is pre-stored in a non-volatile area of ​​the memory 201. When the lens unit 200 is attached to the camera body 100, the CPU 102 requests the correspondence relationship information for the lens unit 200 from the LPU 202. When power is applied to the lens unit 200, the LPU 202 reads the correspondence relationship information for the lens unit 200 from the non-volatile area of ​​the memory 201 in response to the request from the CPU 102, and transmits the information to the CPU 102. The CPU 102 stores the obtained correspondence relationship information in a volatile area of ​​the memory 101.

[0051] The timing for acquiring the correspondence information is not limited to the above timing. For example, the identification information may be deleted from the memory 101 upon transition to the power-saving state, and the identification information may be acquired again upon recovery from the power-saving state.

[0052] When the correspondence information is acquired, the CPU 102 requests the LPU 202 for the aperture value, and the LPU 202 transmits the aperture value to the CPU 102 in response to the request. The CPU 102 acquires blur shape related information corresponding to the acquired aperture value based on the acquired correspondence information (the correspondence information stored in the memory 101).

[0053] The CPU 102 outputs the aperture value and blur shape related information acquired by the above method. For example, the CPU 102 associates the aperture value and blur shape related information with a captured image (frame) and outputs the information to the outside from the output terminal 105 or records it on the recording medium 106. At this time, the CPU 102 stores the aperture value and blur shape related information in a format such as EXIF ​​or SMPTE RDD-18. The information may be recorded or output in a manufacturer-specific area of ​​existing metadata or protocol, such as a manufacturer-specific area. The CPU 102 may record or output the aperture value and blur shape related information using a manufacturer-specific standard (for example, a manufacturer-specific communication standard). The CPU 102 may record or output an aperture value defined with a resolution or format suitable for adding blur, separate from the aperture value stored in the existing metadata. For example, the aperture value may be an F-number with a resolution of 0.01, which is 100 times the aperture value, or an F-number in a log format with a 16-bit resolution.

[0054] As described above, the output aperture value is used to determine the blur intensity of the graphic, and the output blur shape-related information is used to determine the blur shape of the graphic. However, the target to which the output aperture value and blur shape-related information are applied differs depending on the VFX shooting method. In the in-camera method, the aperture value and blur shape-related information are applied to a graphic displayed on a display device, which is one of the shooting targets, and in the post-production method, the aperture value and blur shape-related information are applied to a graphic to be composited with a shot image.

[0055] The in-camera method will be described in more detail. CPU 102 outputs the aperture value and blur shape related information to a CG generation device connected to output terminal 105 in real time (sequential processing of input captured images). The CG generation device generates a graphic based on the aperture value and blur shape related information acquired from camera body 100 (CPU 102) and outputs the generated graphic to a display device. After generating a blur-free graphic, blur based on the aperture value and blur shape related information may be added to the graphic, or a graphic to which blur based on the aperture value and blur shape related information may be generated without generating a blur-free graphic. The method of adding blur is not particularly limited, but for example, blur may be added to each region by filter processing using a filter based on the aperture value and blur shape related information. The display device displays the graphic generated by the CG generation device. Thereafter, the camera body 100 captures an image at an angle of view that includes the graphic displayed on the display device and the actual subject, thereby obtaining a natural composite image (a natural composite image) in which the blur of the actual subject and the blur of the graphic are consistent as the captured image. Note that the device that generates the graphic and the device that outputs the graphic to the display device (the device that controls the display on the display device) may be different. A CG generation system consisting of multiple devices may be used as the CG generation device.

[0056] The post-production method will be described in more detail. The CPU 102 associates the captured image, aperture value, and blur shape related information with each other and records them on the recording medium 106. The captured image file, aperture value file, and blur shape related information file may be the same or different. The captured image, aperture value, and blur shape related information may be recorded on different media. The recording medium 106 is then removed from the camera body 100 and input to a CG synthesis device, which then imports the captured image, aperture value, and blur shape related information into the CG synthesis device. The CG synthesis device generates a graphic based on the aperture value and blur shape related information. As with the in-camera method, a blur-free graphic may be generated, and then blur based on the aperture value and blur shape related information may be applied to the graphic. A graphic with blur based on the aperture value and blur shape related information may be generated without generating a blur-free graphic. The method of applying blur is not particularly limited. For example, blur may be applied to each region by filtering using a filter based on the aperture value and blur shape related information. The CG synthesizer then synthesizes the generated graphic with the captured image, resulting in a natural composite image (a natural composite image) in which the blur of the actual subject matches the blur of the graphic.

[0057] Finally, the timing of each process in in-camera VFX shooting is shown in Figures 13-15. 15 will be used to explain.

[0058] 13 is a schematic diagram showing an example of the timing of communication between the camera body 100 (communication terminal 104) and the lens unit 200 (communication terminal 203), and the timing of image capture by the camera body 100 (image capture element 103). The image capture element 103 captures images at regular time intervals according to a set frame rate.

[0059] In FIG. 13 , the diaphragm 204 is driven during frame 2 due to a change in the brightness of the subject or a user operation. Specifically, after the start of frame 2, the CPU 102 instructs the LPU 202 to start driving the diaphragm 204. Then, in response to the instruction from the CPU 102, the LPU 202 drives the motor that controls the diaphragm 204 and changes the opening diameter of the diaphragm 204. When the driving of the diaphragm 204 is completed, the LPU 202 notifies the CPU 102 that the driving of the diaphragm 204 is completed. The notification that the driving of the diaphragm 204 is completed may include a notification of the changed aperture value, etc. To avoid delays in the notification due to communication time, the notification may be initiated before the driving of the diaphragm 204 is completed. For example, the timing at which the driving of the diaphragm 204 is completed may be predicted, and the notification may be initiated at a timing that is earlier than the predicted timing by the communication time. In FIG. 13 , the CPU 102 receives a notification that the driving of the diaphragm 204 is completed during frame 3. Therefore, captured images corresponding to the changed aperture value are obtained as captured images from frame 3 onwards.

[0060] In Figure 13, time t1 is required to drive the aperture 204. Note that time t1 in Figure 13 is an example, and the time required to drive the aperture 204 may be longer or shorter than time t1. An upper limit to the drive speed of the aperture 204 may be set in consideration of noise during shooting, and the time required to drive the aperture 204 (to change the aperture diameter) may depend on the drive amount (change amount) of the aperture 204. Furthermore, time t1 is the time from when the CPU 102 issues an instruction to start driving the aperture 204 to when it receives a notification that driving of the aperture 204 is complete, but the time required to drive the aperture 204 may be the time until driving of the aperture 204 is completed or the time from when driving of the aperture 204 is started.

[0061] FIG. 14 is a schematic diagram showing an example of the timing of communication between the camera body 100 (output terminal 105) and the CG generation device. The transmission and reception of captured images and aperture values ​​are omitted. In FIG. 14, a change in blur shape related information occurs during frame 1. The CPU 102 notifies the CG generation device of the changed blur shape related information, and the CG generation device generates (updates) a graphic based on the changed blur shape related information. The CG generation device then outputs the generated graphic to the display device, and the display device displays the graphic. In FIG. 14, during frame 5, display of a graphic based on the changed blur shape related information begins. Therefore, captured images from frame 5 onwards include a graphic corresponding to the changed blur shape related information.

[0062] In FIG. 14, time 2 is required from when the CPU 102 outputs the blur shape related information until the display device starts displaying the graphic to which the blur shape related information has been applied.

[0063] Generally, time t1 and time t2 are different. Although it depends on the machine power of the CG generation device, generally, a relatively long time is required to generate a graphic, and time t2 is longer than time t1. Therefore, if CPU 102 outputs an aperture drive start instruction and blur shape related information at the same time, the timing at which the changed aperture value is applied to the captured image (actual subject) does not match the timing at which the changed blur shape related information is applied to the graphic. Then, during the period when only one of the changed aperture value and the changed blur shape related information is applied, an unnatural composite image (an unnatural composite image) is obtained as the captured image, in which the blur shape does not match between the graphic and the actual subject.

[0064] Therefore, in the first embodiment, the CPU 102 controls at least one of the timing of issuing an instruction to start driving the diaphragm 204 and the timing of outputting the blur shape related information based on times t1 and t2. FIG. 15 shows an example of various timings controlled based on times t1 and t2. In FIG. 15, time t2 is longer than time t1, so the CPU 102 issues an instruction to start driving the diaphragm 204 after outputting the blur shape related information. More specifically, the CPU 102 issues an instruction to start driving the diaphragm 204 when a time period (t2-t1) has elapsed since outputting the blur shape related information. When issuing an instruction to start driving the diaphragm 204, the CPU 102 knows the changed aperture value in advance and outputs the blur shape related information corresponding to the changed aperture value. Then, the CPU 102 issues an instruction to start driving the diaphragm 204 by a drive amount based on the changed aperture value. By doing so, the timing when the changed aperture value is applied to the photographed image (actual subject) and the timing when the changed blur shape related information is applied to the graphic can be made closer (matched), which in turn can shorten (eliminate) the period during which an unnatural photographed image (unnatural composite image) is acquired.

[0065] The CPU 102 may control the timing at which the camera body 100 (image sensor 103) captures each frame based on times t1 and t2 so that changes to the aperture value and blur shape related information do not occur midway through a frame. The CPU 102 controls at least one of the timing at which blur shape related information is output, the timing at which an instruction is given to start driving the aperture 204, and the timing at which imaging is performed, based on times t1 and t2. The information on times t1 and t2 may or may not be obtained via communication. The user may measure times t1 and t2 and input them into the camera body 100.

[0066] <Second embodiment> A second embodiment of the present invention will be described below. In the first embodiment, the present invention is applied to an imaging device with an interchangeable lens unit (an interchangeable lens camera). In the second embodiment, the present invention is applied to an imaging device with a non-interchangeable lens unit (an integrated lens camera). Note that a description of parts common to the first embodiment will be omitted.

[0067] 16 is a block diagram showing the configuration of a camera 1400 according to the second embodiment. The camera 1400 has a memory 1401, a CPU 1402, an image sensor 1403, an output terminal 1404, a recording medium 1405, an aperture 1406, and a lens group 1407. The memory 1401, the CPU 1402, the image sensor 1403, the output terminal 1404, and the recording medium 1405 have the same functions as the memory 101, the CPU 102, the image sensor 103, the output terminal 105, and the recording medium 106 in FIG. 1. The aperture 1406 and the lens group 1407 have the same functions as the aperture 204 and the lens group 205. The memory 1401 further has at least some of the functions of the memory 201, and the CPU 1402 further has at least some of the functions of the LPU 202.

[0068] Because the camera 1400 has a configuration in which the lens unit and camera body are integrated, the aperture 1406 and the lens group 1407 are directly connected to the CPU 1402, and the CPU 1402 transmits control signals directly to the aperture 1406 and the lens group 1407. This allows the CPU 1402 to know the aperture value of the aperture 1406 in real time. Furthermore, correspondence information indicating the correspondence between aperture values ​​and blur shape related information is stored in a nonvolatile area of ​​the memory 1401, and the CPU 1402 can acquire blur shape related information corresponding to the aperture value to be output from the memory 1401. The CPU 1402 may acquire the blur shape related information from the nonvolatile area of ​​the memory 1401, or may load the correspondence information from the nonvolatile area of ​​the memory 1401 to a volatile area and acquire the blur shape related information from the volatile area.

[0069] <Third embodiment> A third embodiment of the present invention will be described below. In the third embodiment, the present invention is applied to a lens unit that is detachable from an imaging device. Note that a description of parts common to the first embodiment will be omitted.

[0070] 17 is a block diagram showing the configuration of a lens unit 1500 according to the third embodiment. The lens unit 1500 has a memory 1501, an LPU 1502, an operation member 1503, a communication terminal 1504, an aperture 1505, a lens group 1506, and an output terminal 1507. The memory 1501, the LPU 1502, the communication terminal 1504, the aperture 1505, the lens group 1506, and the output terminal 1507 have functions similar to those of the memory 201, the LPU 202, the communication terminal 203, the aperture 204, the lens group 205, and the output terminal 105 in FIG. 1. The memory 1501 further has functions similar to at least some of the functions of the memory 101, and the LPU 1502 further has functions similar to at least some of the functions of the CPU 102.

[0071] The operation member 1503 is an operation member that can accept an operation to change the aperture diameter (aperture value) of the aperture 1505, and is, for example, an aperture ring or a touch panel. The LPU 1502 drives the aperture 1505 in response to an operation on the operation member 1503, and acquires the changed aperture value. The camera body to which the lens unit 1500 is attached may instruct the LPU 1502 to drive the aperture 1505 via the communication terminal 1504. In this case, the LPU 1502 also drives the aperture 1505 in response to an instruction from the camera body, and acquires the changed aperture value. The LPU 1502 may acquire a captured image via the communication terminal 1504 from the camera body to which the lens unit 1500 is attached.

[0072] Furthermore, correspondence information indicating the correspondence between aperture values ​​and blur shape-related information is stored in a nonvolatile area of ​​the memory 1501, and the LPU 1502 can acquire blur shape-related information corresponding to the aperture value to be output from the memory 1501. The LPU 1502 may acquire the blur shape-related information from the nonvolatile area of ​​the memory 1501, or may load the correspondence information from the nonvolatile area of ​​the memory 1501 to a volatile area and acquire the blur shape-related information from the volatile area. The LPU 1502 outputs the acquired captured image, aperture value, blur shape-related information, etc. to the outside from the output terminal 1507 or the communication terminal 1504.

[0073] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0074] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0075] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0076] Furthermore, in the above-described embodiment, the present invention is applied to a camera or a lens unit, but the present invention can be applied to any electronic device (information processing device) that can output information useful for VFX shooting.

[0077] <Other embodiments> The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0078] The disclosure of this embodiment includes the following configuration, method, program, and medium. (Configuration 1) an acquisition means for acquiring an aperture value of a lens of an imaging device; an output means for outputting, together with the image captured by the imaging device, the aperture value at the time of capturing the image and blur shape related information regarding the shape of blur generated in the image; An information processing device comprising: (Configuration 2) an acquisition means for acquiring the aperture value of the lens of the imaging device in real time; an output means for outputting, in real time, the aperture value and blur shape related information relating to the shape of blur occurring in the captured image of the imaging device; An information processing device comprising: (Configuration 3) The method further includes a storage means for storing correspondence information indicating a correspondence relationship between an aperture value and blur shape related information, The output means acquires blur shape related information corresponding to the aperture value to be output based on the correspondence information and outputs the information. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The storage means stores the correspondence information in advance. 4. The information processing device according to configuration 3. (Configuration 5) The lens of the imaging device is interchangeable; the storage means stores in advance a plurality of pieces of correspondence relationship information corresponding to a plurality of lenses, the information processing device further includes a second acquisition means for acquiring identification information of a lens of the imaging device; The output means acquires and outputs blur shape related information corresponding to the aperture value to be output based on correspondence information corresponding to the lens of the imaging device. 5. The information processing device according to configuration 4. (Configuration 6) The system further includes a third acquisition means for acquiring the correspondence relationship information from an external source. 4. The information processing device according to configuration 3. (Configuration 7) The blur shape related information indicates the shape of the blur. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) The blur shape related information further indicates the similarity of the blur shape to a perfect circle. 8. The information processing device according to configuration 7. (Configuration 9) The blur shape related information further indicates the degree of deformation of the blur shape relative to a perfect circle or a regular polygon. Shown in 8. The information processing device according to configuration 7. (Configuration 10) The output means outputs the aperture value and the blur shape related information to an external device. 10. The information processing device according to any one of configurations 1 to 9. (Configuration 11) The output means records the aperture value and the blur shape related information on a recording medium provided in the information processing device. 10. The information processing device according to any one of configurations 1 to 9. (Configuration 12) The blur shape related information is applied to a graphic to be synthesized with the photographed image. 2. The information processing device according to any one of configurations 1 to 11, wherein: (Configuration 13) The blur shape related information is applied to a graphic displayed on a display device photographed by the imaging device. 3. The information processing device according to any one of configurations 2. (Configuration 14) an instruction means for instructing the start of driving the diaphragm; Control means and and The control means a first time period that is a time period required to drive the aperture of the lens; a second time period that is a time period from when the output means outputs the blur shape related information to when the display device starts displaying a graphic after the blur shape related information is applied; and at least one of the timing at which the output unit outputs the blur shape related information, the timing at which the instruction unit instructs the aperture to start driving, and the timing at which the imaging device takes a photograph, is controlled based on the above. 14. The information processing device according to configuration 13, (Configuration 15) The control means controls at least one of the timing at which the output means outputs the blur shape related information and the timing at which the instruction means instructs the start of driving the diaphragm, so that the instruction means instructs the start of driving the diaphragm after the output means outputs the blur shape related information. 15. The information processing device according to configuration 14. (Configuration 16) The imaging device has an interchangeable lens. 15. The information processing device according to any one of configurations 1 to 14. (Configuration 17) The imaging device has a non-replaceable lens. 15. The information processing device according to any one of configurations 1 to 14. (Configuration 18) The lens is detachable from the imaging device. 15. The information processing device according to any one of configurations 1 to 14. (Configuration 19) The output means outputs the metadata in which the aperture value and the blur shape related information are stored in a manufacturer-specific area. 19. The information processing device according to any one of configurations 1 to 18. (Configuration 20) The output means outputs the aperture value and the blur shape related information using a manufacturer's own standard. 20. The information processing device according to any one of configurations 1 to 19. (Configuration 21) The aperture value is a 16-bit resolution logarithmic value. 21. The information processing device according to any one of configurations 1 to 20. (Method 1) obtaining an aperture value of a lens of an imaging device; outputting, together with the captured image of the imaging device, an aperture value at the time of capturing the captured image and blur shape related information regarding the shape of blur occurring in the captured image; An information processing method comprising: (Method 2) acquiring an aperture value of a lens of an imaging device in real time; outputting, in real time, the aperture value and blur shape related information relating to the shape of blur occurring in the captured image of the imaging device; An information processing method comprising: (program) 22. A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 21. (medium) 22. A computer-readable storage medium storing a program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 21. [Explanation of symbols]

[0079] 100: Camera body 102: CPU 1400: Camera 1402: CPU 1500: Lens unit 1502: LPU

Claims

1. an acquisition means for acquiring an aperture value of a lens of an imaging device; an output means for outputting, together with the image captured by the imaging device, the aperture value at the time of capturing the image and blur shape related information regarding the shape of blur generated in the image; An information processing device comprising:

2. an acquisition means for acquiring the aperture value of the lens of the imaging device in real time; an output means for outputting, in real time, the aperture value and blur shape related information relating to the shape of blur occurring in the captured image of the imaging device; An information processing device comprising:

3. The method further includes a storage means for storing correspondence information indicating a correspondence relationship between an aperture value and blur shape related information, The output means acquires blur shape related information corresponding to the aperture value to be output based on the correspondence information and outputs the information.

2. The information processing apparatus according to claim 1, wherein:

4. The storage means stores the correspondence information in advance.

4. The information processing apparatus according to claim 3,

5. the storage means stores in advance a plurality of pieces of correspondence relationship information corresponding to a plurality of lenses, the information processing device further includes a second acquisition unit that acquires identification information of a lens of the imaging device; The output means acquires and outputs blur shape related information corresponding to the aperture value to be output based on correspondence information corresponding to the lens of the imaging device.

5. The information processing apparatus according to claim 4,

6. The apparatus further includes a third acquisition means for acquiring the correspondence relationship information from an external source.

4. The information processing apparatus according to claim 3,

7. The blur shape related information indicates the shape of the blur.

2. The information processing apparatus according to claim 1, wherein:

8. The blur shape related information further indicates the similarity of the blur shape to a perfect circle.

8. The information processing apparatus according to claim 7,

9. The blur shape related information further indicates the degree of deformation of the blur shape relative to a perfect circle or a regular polygon.

8. The information processing apparatus according to claim 7,

10. The output means outputs the aperture value and the blur shape related information to an external device.

2. The information processing apparatus according to claim 1, wherein:

11. The output means records the aperture value and the blur shape related information on a recording medium provided in the information processing device.

2. The information processing apparatus according to claim 1, wherein:

12. The blur shape related information is applied to a graphic to be synthesized with the photographed image.

2. The information processing apparatus according to claim 1, wherein:

13. The blur shape related information is applied to a graphic displayed on a display device photographed by the imaging device.

3. The information processing apparatus according to claim 2, wherein:

14. an instruction means for instructing the start of driving the diaphragm; Control means and and The control means a first time period that is a time period required to drive the aperture of the lens; a second time period that is a time period from when the output means outputs the blur shape related information to when the display device starts displaying a graphic after the blur shape related information is applied; and at least one of the timing at which the output unit outputs the blur shape related information, the timing at which the instruction unit instructs the aperture to start driving, and the timing at which the imaging device takes a photograph, is controlled based on the above.

14. The information processing apparatus according to claim 13,

15. The control means controls at least one of the timing at which the output means outputs the blur shape related information and the timing at which the instruction means instructs the start of driving the diaphragm, so that the instruction means instructs the start of driving the diaphragm after the output means outputs the blur shape related information.

15. The information processing apparatus according to claim 14,

16. The imaging device has an interchangeable lens.

2. The information processing apparatus according to claim 1, wherein:

17. The imaging device has a non-replaceable lens.

2. The information processing apparatus according to claim 1, wherein:

18. The lens is detachable from the imaging device.

2. The information processing apparatus according to claim 1, wherein:

19. The output means outputs the metadata in which the aperture value and the blur shape related information are stored in a manufacturer-specific area.

2. The information processing apparatus according to claim 1, wherein:

20. The output means outputs the aperture value and the blur shape related information using a manufacturer's own standard.

2. The information processing apparatus according to claim 1, wherein:

21. The aperture value is a value in 16-bit resolution and log format.

2. The information processing apparatus according to claim 1, wherein:

22. obtaining an aperture value of a lens of an imaging device; outputting, together with the captured image of the imaging device, an aperture value at the time of capturing the captured image and blur shape related information regarding the shape of blur occurring in the captured image; An information processing method comprising:

23. acquiring an aperture value of a lens of an imaging device in real time; The aperture value and blur shape related information relating to the shape of blur occurring in the captured image of the imaging device and outputting the information in real time. An information processing method comprising:

24. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 21.

25. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 21.

Images