Information processing device

Abstract

To provide a technology that makes it possible to obtain a more natural image (an image with less sense of incongruity) as a composite image obtained by combining a real subject with a graphic.SOLUTION: An information processing device according to the present invention includes acquisition means for acquiring the aperture value of the lens of an imaging device, and output means for outputting, together with the image captured by the imaging device, the aperture value at the time of capturing the image and blur shape related information regarding the shape of the blur that occurs in the captured image.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, and particularly to a technique for combining an actual subject (real subject) and graphics (Computer Graphics (CG) images).

Background Art

[0002] In recent years, there is a growing demand for video content that uses Visual Effects technology to combine actual subjects and graphics (CG images). Shooting of video content using Visual Effects technology is generally referred to as VFX shooting. There are roughly two methods for VFX shooting. The first method is a post-production method in which shooting is performed against a special background such as a green screen, and then graphics are combined with the background of the shot image in post-production. The second method is an in-camera method in which shooting is performed with graphics displayed on a large display device as the background. In VFX shooting, by using CG, it is possible to obtain immersive images without going to actual locations, or to obtain images with angles of view and compositions that would be difficult to achieve in reality, so production costs can be kept low. As a result, the demand for VFX shooting is increasing. In VFX shooting, it is desired to obtain a natural combined image (a combined image with no sense of incongruity).

[0003] Patent Document 1 discloses a technique for changing the combining position of graphics according to the orientation of a camera. Patent Document 2 discloses a technique for generating a point spread function based on the distance from the entrance pupil of a lens and the size of the exit pupil, and generating graphics based on the point spread function.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

[0005] Incidentally, while cameras with 2 / 3-inch or 1-inch image sensors were previously the mainstream, in recent years there has been an increase in cameras with larger image sensors, such as SUPER35mm and full-frame 35mm. With the increase in image sensor size, the adjustable depth of field widens, and the freedom of bokeh expression increases. Bokeh involves the intensity (spread) and shape, which can be expressed by the point image intensity distribution function, as important factors. In conventional VFX shooting, graphics that take into account the bokeh shape that occurs in the captured image are not generated, and natural composite images cannot be obtained.

[0006] The present invention aims to provide a technology that enables the creation of more natural images (images with less unnaturalness) as composite images by combining real subjects and graphics. [Means for solving the problem]

[0008] This invention 1 The embodiment includes an acquisition means for acquiring the aperture value of the lens of the imaging device in real time, an output means for outputting the aperture value and blur shape-related information relating to the shape of the blur that occurs in the captured image of the imaging device in real time, and a timing control means. The aforementioned blur shape-related information is applied to the graphics displayed on the display device captured by the imaging device. The timing control means is The first time is the time required to drive the aperture of the aforementioned lens, After the output means outputs the blur shape-related information, The aforementioned The second hour is the time until the display device begins displaying the graphic after applying the blur shape-related information. and The information processing device is characterized by controlling, based on the timing at which the output means outputs the blur shape-related information and the timing at which the imaging device takes a picture.

[0010] This invention 2The embodiment includes an acquisition step of acquiring the aperture value of the lens of the imaging device in real time, an output step of outputting the aperture value and blur shape-related information relating to the shape of the blur that occurs in the captured image of the imaging device in real time, and a timing control step. The aforementioned blur shape-related information is applied to the graphics displayed on the display device captured by the imaging device. In the aforementioned timing control step, The first time is the time required to drive the aperture of the aforementioned lens, After outputting the blur shape-related information in the output step, The aforementioned table The second hour is the time until the display device starts displaying the graphic after applying the blur shape-related information. and The information processing method is characterized by controlling, based on the above, the timing of outputting the blur shape-related information in the output step and the timing of the imaging device taking a picture.

[0011] This invention 3 This embodiment is a program that causes a computer to function as each of the means of the above-mentioned information processing device. 4 The embodiment is a computer-readable storage medium that stores a program for causing the computer to function as each of the means of the information processing device described above. [Effects of the Invention]

[0012] According to the present invention, it becomes possible to obtain a more natural image (an image with less unnaturalness) as a composite image by combining a real subject and a graphic. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram of the camera according to the first embodiment. [Figure 2] This is a schematic diagram of the filming location. [Figure 3] This is a schematic diagram of a composite image. [Figure 4] This is a schematic diagram of the filming location. [Figure 5] This is a schematic diagram of a composite image. [Figure 6] This is a schematic diagram of an aperture. [Figure 7] It is a schematic diagram of a blur shape. [Figure 8] It is a schematic diagram of correspondence relationship information. [Figure 9] It is a schematic diagram of a blur shape. [Figure 10] It is a schematic diagram of correspondence relationship information. [Figure 11] It is a schematic diagram of communication between a camera body and a lens unit. [Figure 12] It is a schematic diagram of communication between a camera body and a lens unit. [Figure 13] It is a schematic diagram of communication between a camera body and a lens unit. [Figure 14] It is a schematic diagram of communication between a camera body and a CG generation device. [Figure 15] It is a schematic diagram of communication between a lens unit, a camera body, and a CG generation device. [Figure 16] It is a block diagram of a camera according to the second embodiment. [Figure 17] It is a block diagram of a lens unit according to the third embodiment. [Description of Embodiments]

[0014] <First Embodiment> Hereinafter, the first embodiment of the present invention will be described. FIG. 1 is a block diagram showing the configuration of a camera body 100 as an example of the information processing apparatus according to the first embodiment. The camera body 100 is an imaging apparatus with an interchangeable lens unit (lens, lens device). In FIG. 1, a lens unit 200 is attached to the camera body 100.

[0015] The camera body 100 includes a memory 101, a CPU (Central Processing Unit) 102, an image sensor 103, a communication terminal 104, an output terminal 105, and a recording medium 106.

[0016] Memory 101 is a storage unit that stores various types of data (various types of information), including images. The data stored in memory 101 can be read from memory 101 when needed. Memory 101 has a volatile area that can hold data only while power is supplied, and a non-volatile area that can hold data even when power is stopped.

[0017] The CPU 102 is a control unit that controls various parts of the camera body 100 and various parts of the accessories attached to the camera body 100 (lens unit 200 in Figure 1). For example, programs and various parameters for operating the CPU 102 are stored in the memory 101, and the CPU 102 performs various controls by reading the program from the non-volatile area of ​​the memory 101, expanding it into the volatile area, and executing it.

[0018] The image sensor 103 is a charge-accumulating solid-state image sensor, such as a CMOS or CCD, which receives a light beam (light beam from the subject) guided into the camera body 100 via the lens unit 200 and converts it into an electrical image signal. The image (signal) obtained by the image sensor 103 is used for live view display, recording to the recording medium 106 (described later), and external output using the output terminal 105, under the control of the CPU 102. The CPU 102 can also control the exposure time of the image sensor 103 and the shooting timing (the timing of taking a picture).

[0019] The communication terminal 104 is a terminal for communicating with the lens unit (lens unit 200 in Figure 1) attached to the camera body 100.

[0020] Output terminal 105 is, for example, an Ethernet terminal, an SDI terminal, or an HDMI® terminal, and is used to output various data (various information), including images, to an external source, or to acquire various data (various information) from an external source.

[0021] The recording medium 106 is a recording medium that can be attached to and removed from the camera body 100, and is, for example, an SD card or CFExpress. Various types of data (various types of information), including images, can be recorded on the recording medium 106.

[0022] The lens unit 200 includes a memory 201, an LPU (Lens Processing Unit) 202, a communication terminal 203, an aperture 204, and a lens group 205. The lens unit 200 is a so-called interchangeable lens that can be attached to and detached from the imaging device.

[0023] Memory 201 is a storage unit that stores various types of data (various types of information). The data stored in memory 201 can be read from memory 201 when needed. Memory 201 has a volatile area that can retain data only when power is supplied, and a non-volatile area that can retain data even when power is stopped.

[0024] The LPU202 is a control unit that controls various parts of the lens unit 200. For example, the program and various parameters for operating the LPU202 are stored in the memory 201, and the LPU202 performs various controls by reading the program from the non-volatile area of ​​the memory 201, expanding it into the volatile area, and executing it.

[0025] Communication terminal 203 is a terminal for communicating with the imaging device (camera body 100 in Figure 1) to which the lens unit 200 is attached. In Figure 1, the LPU 202 of the lens unit 200 and the CPU 102 of the camera body 100 are connected to each other via the communication terminal 203 of the lens unit 200 and the communication terminal 104 of the camera body 100. The LPU 202 is a CP In response to control instructions from U102, each part of the lens unit 200 can be driven (controlled).

[0026] The aperture 204 is a light intensity control component that controls (adjusts) the amount of light beam guided into the camera body 100. For example, by changing the aperture value, the aperture diameter of the aperture 204 is changed to an aperture diameter corresponding to the changed aperture value, and the amount of light is changed. In addition to the amount of light, the depth of field and bokeh can also be changed.

[0027] The lens group 205 includes focusing lenses, zoom lenses, and shift lenses. The light beam from the subject is guided into the camera body 100 through the lens group 205 and the aperture 204. The position of each lens included in the lens group 205 is controllable (changeable). For example, the focus lens... vinegar The lens can be moved along the optical axis to adjust the focus on the subject.

[0028] The camera body 100 (and lens unit 200) is used, for example, for VFX shooting, which involves compositing a real subject with a graphic (CG image).

[0029] Figures 2 and 3 will be used to explain post-production VFX shooting. Figure 2 is a schematic diagram showing an example of a shooting location, and Figure 3 is a schematic diagram showing an example of a composite image created by combining a real subject with graphics.

[0030] In post-production VFX shooting, the first step is to shoot against a special background. In the example in Figure 2, the actual subject 301 is shot against a green screen 300. Note that the background is not limited to a green screen (a background of a single color green); for example, it could be a background of another single color or a background with another pattern.

[0031] Next, the CG compositing device imports the captured image into CG compositing editing software, detects the background area from the captured image, and composites the background graphic onto the detected area. It is also possible to further composite graphics other than the background. In the example in Figure 3, the background graphic 400 and non-background graphics 402-404 are composited. Non-background graphics 402-404 are objects that were placed before the background graphic 400.

[0032] The metadata of the captured image includes information such as the focus position at the time of shooting, and based on this information, graphics 400, 402~40 4 The blur is controlled (adjusted) individually. For example, the way the background graphic 400 is blurred is different from the way the other graphics 402-404 are blurred. The blur of the actual subject 401 (301) and graphics 400, 402-404 are blurred. 4 Graphics 400, 402~40 to match the way the blurring occurs. 4 By generating and synthesizing these elements, it is possible to create natural-looking composite images (composite images that do not appear unnatural).

[0033] Figures 4 and 5 will be used to explain in-camera VFX shooting. Figure 4 is a schematic diagram showing an example of a shooting location, and Figure 5 is a schematic diagram showing an example of a composite image created by combining a real subject with graphics.

[0034] In in-camera VFX shooting, the image is shot with a graphic displayed on a large display device as the background. In the example in Figure 4, the actual subjects 501-504 are shot with the graphic displayed on the display device 500 as the background. At this time, information such as the focus position is output from the camera in real time (sequentially during shooting) and input to the CG generation device. The CG generation device generates a graphic with blur based on the input information to be displayed on the display device 500. The composite image in Figure 5 is produced when the camera shoots with a field of view that includes the graphic displayed on the display device 500 and the actual subjects 501-504. This can be obtained. Note that information such as the focus position may be transmitted from the camera body to the CG generation device, or it may be transmitted from the lens unit to the CG generation device without going through the camera body. Furthermore, information such as the focus position may be transmitted from the lens unit to the CG generation device via a device other than the camera body (for example, a device that converts information acquired from an external source into information that can be input to the CG generation device).

[0035] Let's explain bokeh. Bokeh involves two important factors: intensity (degree of spread) and shape. In conventional VFX shooting, the intensity of the graphic bokeh is controlled (adjusted) based on factors such as aperture value and focus position, but the generated graphics do not take into account the shape of the bokeh that occurs in the captured image, resulting in an unnatural composite image.

[0036] The captured image will have blur around out-of-focus subjects. The intensity of the blur is determined by the degree of out-of-focus, such as the amount of deviation from the depth of field based on the aperture value and the size of the image sensor, and the amount of deviation from the subject distance corresponding to the focus position. On the other hand, the shape of the blur is determined by the aperture value and the image sensor. Size It is not determined by the focus position, but rather by the shape of the aperture opening.

[0037] Figure 6 is a schematic diagram showing examples of a circular aperture and a polygonal aperture (iris aperture). In Figure 6, an octagonal aperture is shown as an example of a polygonal aperture. Figure 7 is a schematic diagram showing an example of the shape of bokeh that occurs in a captured image. In the case of a circular aperture, circular bokeh occurs, and in the case of a polygonal aperture, polygonal bokeh occurs. In the case of a polygonal aperture, if the number of aperture blades is 8, octagonal bokeh occurs. If the edges that form the aperture opening of the aperture blades have curvature, bokeh that is not a regular polygon but a polygon close to a circle may occur. Furthermore, even in the case of a polygonal aperture, if the aperture opening becomes circular when wide open, circular bokeh occurs. Thus, the shape of the bokeh depends on the specifications of the lens unit (aperture) and the aperture value. Here, for simplicity, we have mentioned the shape of bokeh when photographing an ideal point light source, but even in the area of ​​subjects other than point light sources, bokeh of a shape similar to the above shape occurs.

[0038] If information that can identify (determine) the bokeh shape cannot be notified to the CG generation device or CG compositing device, the bokeh shape will not match between the generated graphic and the actual subject, resulting in an unnatural composite image (a composite image that looks unnatural). Therefore, in the first embodiment, the camera body 100 outputs bokeh shape-related information, which is information about the bokeh shape, to the outside via the output terminal 105 or records it on the recording medium 106 so that the CG generation device or CG compositing device can acquire it. In the post-production method, the aperture value is acquired and, along with the captured image, the aperture value at the time of shooting and bokeh shape-related information regarding the shape of the bokeh that occurs in the captured image are output. This processing may or may not be performed in real time. In the in-camera method, the aperture value is acquired in real time and the aperture value and bokeh shape-related information are output in real time. The output aperture value is used to determine the bokeh intensity of the graphic, and the output bokeh shape-related information is used to determine the bokeh shape of the graphic. If the bokeh shape-related information is not information that indicates the bokeh shape, the aperture value may be used to understand the bokeh shape.

[0039] The method for generating bokeh shape-related information will be described. In the first embodiment, the bokeh shape-related information is assumed to represent the bokeh shape. As mentioned above, the bokeh shape depends on the specifications of the lens unit (aperture) and the aperture value. Therefore, if the lens unit to be used is predetermined, bokeh shape-related information can be generated (acquired) based only on the aperture value. In the first embodiment, the memory 101 stores correspondence information that shows the correspondence between the aperture value and the bokeh shape-related information. The CPU 102 then acquires the bokeh shape-related information corresponding to the output aperture value based on the correspondence information and outputs it.

[0040] Figure 8 is a schematic diagram showing an example of correspondence information. Figure 8 shows Tables 1 to 3 as correspondence information. Table 1 shows bokeh shape-related information indicating the bokeh shape, and multiple combinations of aperture value and bokeh shape are described. Using Table 1, the bokeh shape can be determined from the aperture value. Note that only representative aperture values ​​are required in the table; bokeh shape-related information corresponding to aperture values ​​not listed in the table may be obtained through interpolation processing using the information listed in the table. Table 2 shows bokeh... shape The related information shows the bokeh shape and circularity (the degree of similarity of the bokeh shape to a perfect circle), and multiple combinations of aperture value, bokeh shape, and circularity are described. Using Table 2, it is possible to determine the bokeh shape more accurately when the circularity changes depending on the aperture value. Table 3 shows the threshold aperture value at which the resulting bokeh switches between circular and polygonal bokeh, and the shape of the bokeh (polygonal) that occurs when the aperture value is smaller than the threshold (closer aperture). Using Table 3, it can be determined that the bokeh is circular when the aperture value is larger than the threshold (wider aperture), and polygonal bokeh when the aperture value is smaller than the threshold (closer aperture). Note that the threshold may be the lower limit of the aperture value at which circular bokeh occurs, or the upper limit of the aperture value at which polygonal bokeh occurs.

[0041] One example of a special bokeh shape is the bokeh shape produced when using an anamorphic lens. When using an anamorphic lens, the captured image is compressed horizontally. In post-production, the captured image is stretched horizontally to produce an image with a wider field of view compared to a normal captured image. In the captured image, the subject is compressed horizontally, and the bokeh is also compressed horizontally. Figure 9 is a schematic diagram showing an example of the bokeh shape that occurs in an image obtained using an anamorphic lens. Circular bokeh becomes elliptical bokeh, and regular octagonal bokeh becomes vertically elongated octagonal bokeh.

[0042] Figure 10 is a schematic diagram showing an example of correspondence information considering special bokeh shapes. Tables 4-6 are shown in Figure 10 as correspondence information. Table 4 shows bokeh shape-related information indicating the bokeh shape, describing multiple combinations of aperture value and bokeh shape. An ellipse is shown as the bokeh shape at wide open aperture, and a vertically elongated hexagon is shown as another bokeh shape. Table 5 shows the bokeh... shape The related information shows the bokeh shape and flattening ratio (the degree of distortion of the bokeh shape relative to a perfect circle or regular polygon), and multiple combinations of aperture value, bokeh shape, and flattening ratio are described. The degree of distortion is not limited to flattening ratio; for example, it could be eccentricity. In Table 6, the resulting bokeh switches between elliptical bokeh and vertically elongated polygonal bokeh. This table shows the threshold aperture value, the shape of the bokeh that occurs when the aperture value is smaller than the threshold (closer to the aperture side) (vertical polygon), and the flattening ratio common to both elliptical and vertical polygonal bokeh. 6 Using this method, when the aperture value is greater than the threshold (wide open) oval When it can be determined that the bokeh is circular and the aperture value is smaller than the threshold (closer to the aperture side) Vertical It can be determined that this is polygonal blur. The threshold is, oval It could also be the lower limit of the aperture value at which circular bokeh occurs. Vertical This could also be the upper limit of the aperture value at which polygonal bokeh occurs.

[0043] While the example shown uses a table for correspondence information, any information that allows for the generation (acquisition) of bokeh shape-related information from the aperture value is acceptable; for example, it could be a function. Similarly, while the example shown uses bokeh shape-related information to indicate the bokeh shape, any information that allows for the determination of the bokeh shape from the aperture value is acceptable; for example, it could indicate the number of aperture blades, the shape of the aperture blades, or the type of aperture (circular aperture / polygonal aperture). To determine the bokeh shape from this information, the correspondence information can similarly be stored as a table.

[0044] The timing for acquiring bokeh shape-related information will be explained using Figures 11 and 12. Figures 11 and 12 are schematic diagrams showing communication between the camera body 100 (communication terminal 104) and the lens unit 200 (communication terminal 203).

[0045] In the example shown in Figure 11, multiple correspondence information items corresponding to multiple lens units are pre-stored in the non-volatile area of ​​memory 101. When a lens unit 200 is attached to the camera body 100, the CPU 102 requests identification information for the lens unit 200 from the LPU 202. When power is supplied to the lens unit 200, the LPU 202 reads the identification information for the lens unit 200 from the non-volatile area of ​​memory 201 in response to the request from the CPU 102 and sends it to the CPU 102. The identification information for the lens unit 200 can be any information that can identify the lens unit 200, such as its name, model number, or serial number.

[0046] Although it has been stated that identification information is transmitted from the lens unit 200 to the camera body 100, the identification information may also be generated within the camera body 100. For example, the camera body 100 may be provided with multiple identification buttons that can be pressed by the lens unit, and multiple pieces of identification information corresponding to each of the multiple lens units may be pre-stored in the non-volatile area of ​​the memory 101. The CPU 102 may then retrieve one of the multiple pieces of identification information from the memory 101 depending on the pressing status of the multiple identification buttons.

[0047] Furthermore, the timing for acquiring identification information is not limited to the timing described above. For example, the identification information may be deleted from memory 101 in response to a transition to a power-saving state, and then reacquired when the system returns from the power-saving state.

[0048] Upon obtaining identification information, the CPU 102 retrieves correspondence information corresponding to the lens unit 200 from the memory 101 based on that identification information.

[0049] Subsequently, CPU 102 requests the aperture value from LPU 202, and LPU 202 sends the aperture value to CPU 102 in response to the request. Based on the correspondence information, CPU 102 obtains bokeh shape-related information corresponding to the obtained aperture value. The transmission and reception of aperture values ​​and the acquisition of bokeh shape-related information are repeated. These processes may be performed for each frame, or only when there is a change in the aperture value.

[0050] In the example shown in Figure 12, the correspondence information for the lens unit 200 is pre-stored in the non-volatile area of ​​memory 201. When the lens unit 200 is attached to the camera body 100, the CPU 102 requests the correspondence information for the lens unit 200 from the LPU 202. When power is supplied to the lens unit 200, the LPU 202 reads the correspondence information for the lens unit 200 from the non-volatile area of ​​memory 201 in response to the request from the CPU 102 and sends it to the CPU 102. The CPU 102 stores the acquired correspondence information in the volatile area of ​​memory 101.

[0051] Note that the timing for acquiring correspondence information is not limited to the timings mentioned above. For example, in response to a transition to a power-saving state. Correspondence The information is deleted from memory 101, and when the power is restored from the power-saving state, Correspondence You may retrieve the information again.

[0052] Upon obtaining the correspondence information, the CPU 102 requests the aperture value from the LPU 202, and the LPU 202 sends the aperture value to the CPU 102 in response to the request. Based on the obtained correspondence information (the correspondence information stored in memory 101), the CPU 102 obtains the bokeh shape-related information corresponding to the obtained aperture value.

[0053] The CPU 102 outputs aperture value and bokeh shape-related information obtained by the method described above. For example, the CPU 102 associates the aperture value and bokeh shape-related information with the captured image (frame) and outputs it externally from the output terminal 105 or records it on the recording medium 106. At this time, the CPU 102 outputs the aperture value and bokeh shape-related information as EXIF ​​or SMPTE RDD-18. Alternatively, the data may be recorded or output in the manufacturer's proprietary area of ​​existing metadata or protocols. CPU102 may record or output aperture value and bokeh shape-related information using the manufacturer's proprietary standard (e.g., the manufacturer's proprietary communication standard). CPU102 may record or output aperture values ​​defined in a resolution and format suitable for applying bokeh, separate from the aperture values ​​stored in the existing metadata. For example, this could be an F-number with a resolution of 0.01 obtained by multiplying the aperture value by 100, or an F-number in log format with 16-bit resolution.

[0054] As mentioned above, the output aperture value is used to determine the blur intensity of the graphic, and the output blur shape-related information is used to determine the blur shape of the graphic. However, the target to which the output aperture value and blur shape-related information are applied differs depending on the VFX shooting method. In the in-camera method, the aperture value and blur shape-related information are applied to the graphic displayed on the display device, which is one of the shooting targets, while in the post-production method, the aperture value and blur shape-related information are applied to the graphic to be composited into the captured image.

[0055] The in-camera method will be explained in more detail. The CPU 102 outputs aperture value and bokeh shape-related information in real time (processed sequentially on the input captured image) to the CG generation device connected to the output terminal 105. The CG generation device generates a graphic based on the aperture value and bokeh shape-related information obtained from the camera body 100 (CPU 102) and outputs the generated graphic to the display device. After generating a graphic without bokeh, bokeh based on the aperture value and bokeh shape-related information may be added to the graphic, or a graphic with bokeh based on the aperture value and bokeh shape-related information may be generated without generating a graphic without bokeh. The method of adding bokeh is not particularly limited, but for example, bokeh may be added by filtering using a filter based on the aperture value and bokeh shape-related information for each region. The display device displays the graphic generated by the CG generation device. Subsequently, the camera body 100 takes a photograph with a field of view that includes both the graphic displayed on the display device and the actual subject, thereby obtaining a natural composite image (a composite image without any unnaturalness) as the captured image, in which the blur of the actual subject and the blur of the graphic are consistent. Note that the device that generates the graphic and the device that outputs the graphic to the display device (the device that controls the display on the display device) may be different. A CG generation system consisting of multiple devices may be used as the CG generation device.

[0056] The post-production method will be explained in more detail. The CPU 102 associates the captured image, aperture value, and bokeh shape-related information with each other and records them on the recording medium 106. The files for the captured image, aperture value, and bokeh shape-related information may be the same or different. The captured image, aperture value, and bokeh shape-related information may be recorded on different media. Subsequently, the recording medium 106 is removed from the camera body 100 and input to the CG compositing device, and the captured image, aperture value, and bokeh shape-related information are taken into the CG compositing device. The CG compositing device generates a graphic based on the aperture value and bokeh shape-related information. Similar to the in-camera method, a graphic without bokeh may be generated, and then bokeh based on the aperture value and bokeh shape-related information may be added to the graphic. Alternatively, a graphic with bokeh based on the aperture value and bokeh shape-related information may be generated without generating a graphic without bokeh. The method of adding bokeh is not particularly limited, but for example, bokeh may be added by filtering using a filter based on the aperture value and bokeh shape-related information for each region. The CG compositing device then combines the generated graphics with the captured image. This results in a natural composite image (a composite image without any unnaturalness) where the blur of the real subject and the blur of the graphics are consistent.

[0057] Finally, regarding the timing of each process in in-camera VFX shooting, see Figure 13~ Let's explain using 15.

[0058] Figure 13 is a schematic diagram showing an example of the timing of communication between the camera body 100 (communication terminal 104) and the lens unit 200 (communication terminal 203), and the timing of when the camera body 100 (image sensor 103) takes a picture. The image sensor 103 takes pictures at regular time intervals according to the set frame rate.

[0059] In Figure 13, the aperture 204 is driven during the duration of frame 2 due to a change in the brightness of the subject or user operation. Specifically, after the start of the duration of frame 2, the CPU 102 instructs the LPU 202 to start driving the aperture 204. The LPU 202 then drives the motor that controls the aperture 204 in response to the instruction from the CPU 102, changing the aperture diameter of the aperture 204. Once the driving of the aperture 204 is complete, the LPU 202 notifies the CPU 102 that the driving of the aperture 204 is complete. As a notification that the driving of the aperture 204 is complete, a notification of the changed aperture value may also be given. To avoid delays in notification due to communication time, the notification may be started before the driving of the aperture 204 is complete. For example, the timing of the completion of the driving of the aperture 204 may be predicted, and the notification may be started at a time that is the communication time backward from the predicted timing. In Figure 13, the CPU 102 receives notification that the driving of the aperture 204 is complete during the duration of frame 3. Therefore, from frame 3 onwards, images corresponding to the changed aperture value are obtained.

[0060] In Figure 13, it takes time t1 to drive the aperture 204. Note that time t1 in Figure 13 is just an example, and the time required to drive the aperture 204 may be longer or shorter than time t1. An upper limit may be set for the drive speed of the aperture 204 to take into consideration the noise during shooting, and the time required to drive the aperture 204 (change in aperture diameter) may depend on the amount of drive (change) of the aperture 204. In addition, time t1 is the time from when the CPU 102 instructs the start of driving the aperture 204 until notification of completion of driving the aperture 204 is received, but the time required to drive the aperture 204 may be the time until the completion of driving the aperture 204, or the time from the start of driving the aperture 204.

[0061] Figure 14 is a schematic diagram showing an example of the timing of communication between the camera body 100 (output terminal 105) and the CG generation device. The transmission and reception of captured images and aperture values ​​are omitted. In Figure 14, a change in bokeh shape-related information occurs during the period of frame 1. The CPU 102 notifies the CG generation device of the changed bokeh shape-related information, and the CG generation device generates (updates) a graphic based on the changed bokeh shape-related information. The CG generation device then outputs the generated graphic to the display device, and the display device displays the graphic. In Figure 14, the display of the graphic based on the changed bokeh shape-related information begins during the period of frame 5. Therefore, captured images from frame 5 onwards include a graphic corresponding to the changed bokeh shape-related information.

[0062] Figure 14 shows the time from when the CPU 102 outputs blur shape-related information until the display device starts displaying the graphic after applying the blur shape-related information. t It requires 2.

[0063] Generally, time t1 and time t2 are different. Depending on the machine power of the CG generation device, graphic generation generally takes a relatively long time, and time t2 is longer than time t1. Therefore, if the CPU 102 outputs the aperture drive start instruction and the bokeh shape related information at the same time, the timing at which the changed aperture value is applied to the captured image (actual subject) and the timing at which the changed bokeh shape related information is applied to the graphic will not coincide. And during the period when only one of the changed aperture value or the changed bokeh shape related information is applied, an unnatural composite image (an unnatural composite image) is obtained as the captured image, in which the bokeh shape does not match between the graphic and the actual subject.

[0064] Therefore, in the first embodiment, the CPU 102 controls at least one of the timing for instructing the start of driving the aperture 204 and the timing for outputting bokeh shape-related information, based on times t1 and t2. Figure 15 shows an example of various timings controlled based on times t1 and t2. In Figure 15, since time t2 is longer than time t1, the CPU 102 instructs the start of driving the aperture 204 after outputting the bokeh shape-related information. More specifically, the CPU 102 instructs the start of driving the aperture 204 at a timing after (t2-t1) has elapsed since outputting the bokeh shape-related information. When the CPU 102 instructs the start of driving the aperture 204, it already knows the changed aperture value and outputs bokeh shape-related information corresponding to the changed aperture value. Then, as an instruction to start driving the aperture 204, the CPU 102 instructs to drive the aperture 204 with a drive amount based on the changed aperture value. By doing this, the timing at which the changed aperture value is applied to the captured image (actual subject) and the timing at which the changed bokeh shape-related information is applied to the graphic can be brought closer together (matched). In turn, the period during which unnatural captured images (unnatural composite images) are acquired can be shortened (eliminated).

[0065] The CPU 102 may control the timing of when the camera body 100 (image sensor 103) takes a picture of each frame, based on times t1 and t2, so that the aperture value and bokeh shape-related information do not change in the middle of a frame. Based on times t1 and t2, the CPU 102 controls at least one of the following: the timing of outputting bokeh shape-related information, the timing of instructing the start of driving the aperture 204, and the timing of taking a picture. The information for times t1 and t2 may or may not be obtained by communication. The user may measure times t1 and t2 and input them to the camera body 100.

[0066] <Second Embodiment> A second embodiment of the present invention will be described below. In the first embodiment, the present invention was applied to an imaging device with a replaceable lens unit (interchangeable lens camera). In the second embodiment, the present invention is applied to an imaging device with a non-replaceable lens unit (integrated lens camera). Descriptions of parts common to the first embodiment will be omitted.

[0067] Figure 16 is a block diagram showing the configuration of a camera 1400 according to a second embodiment. The camera 1400 includes a memory 1401, a CPU 1402, an image sensor 1403, an output terminal 1404, a recording medium 1405, an aperture 1406, and a lens group 1407. The memory 1401, CPU 1402, image sensor 1403, output terminal 1404, and recording medium 1405 have functions similar to the memory 101, CPU 102, image sensor 103, output terminal 105, and recording medium 106 in Figure 1. The aperture 1406 and lens group 1407 have functions similar to the aperture 204 and lens group 205. The memory 1401 further has functions similar to at least some of the functions of the memory 201, and the CPU 1402 further has functions similar to at least some of the functions of the LPU 202.

[0068] Since the camera 1400 has a configuration in which the lens unit and the camera body are integrated, the aperture 1406 and lens group 1407 are directly connected to the CPU 1402, and the CPU 1402 directly transmits control signals to the aperture 1406 and lens group 1407. For this reason, the CPU 1402 can know the aperture value of the aperture 1406 in real time. In addition, correspondence information showing the relationship between the aperture value and bokeh shape related information is stored in the non-volatile area of ​​memory 1401, and the CPU 1402 can obtain the bokeh shape related information corresponding to the output aperture value from memory 1401. The CPU 1402 may obtain the bokeh shape related information from the non-volatile area of ​​memory 1401, or it may expand the correspondence information from the non-volatile area of ​​memory 1401 to the volatile area and obtain the bokeh shape related information from the volatile area.

[0069] <Third Embodiment> A third embodiment of the present invention will be described below. In the third embodiment, the present invention is applied to a lens unit that is detachable from an imaging device. Descriptions of parts common to the first embodiment will be omitted.

[0070] Figure 17 is a block diagram showing the configuration of a lens unit 1500 according to a third embodiment. The lens unit 1500 includes a memory 1501, an LPU 1502, an operating member 1503, a communication terminal 1504, an aperture 1505, a lens group 1506, and an output terminal 1507. The memory 1501, LPU 1502, communication terminal 1504, aperture 1505, lens group 1506, and output terminal 1507 have functions similar to those of the memory 201, LPU 202, communication terminal 203, aperture 204, lens group 205, and output terminal 105 in Figure 1. The memory 1501 further has functions similar to at least some of the functions of the memory 101, and the LPU 1502 further has functions similar to at least some of the functions of the CPU 102.

[0071] The operating member 1503 is an operating member that can accept operations to change the aperture diameter (aperture value) of the aperture 1505, and is, for example, an aperture ring or a touch panel. The LPU 1502 drives the aperture 1505 in response to the operation on the operating member 1503 and obtains the changed aperture value. The camera body to which the lens unit 1500 is attached may instruct the LPU 1502 to drive the aperture 1505 via the communication terminal 1504. In this case as well, the LPU 1502 drives the aperture 1505 in response to the instruction from the camera body and obtains the changed aperture value. The LPU 1502 may also obtain the captured image from the camera body to which the lens unit 1500 is attached via the communication terminal 1504.

[0072] Furthermore, correspondence information showing the relationship between aperture value and bokeh shape-related information is stored in the non-volatile area of ​​memory 1501, and the LPU 1502 can obtain bokeh shape-related information corresponding to the output aperture value from memory 1501. The LPU 1502 may obtain bokeh shape-related information from the non-volatile area of ​​memory 1501, or it may expand the correspondence information from the non-volatile area of ​​memory 1501 to the volatile area and obtain bokeh shape-related information from the volatile area. The LPU 1502 outputs the acquired captured image, aperture value, bokeh shape-related information, etc. to the outside from the output terminal 1507 or the communication terminal 1504.

[0073] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0074] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0075] Furthermore, although embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms that do not depart from the spirit of the invention are also included in the present invention. Moreover, each of the embodiments described above is merely one embodiment of the present invention, and it is possible to combine each embodiment as appropriate.

[0076] Furthermore, although the present invention was applied to a camera or lens unit in the embodiments described above, the present invention can be applied to any electronic device (information processing device) capable of outputting information useful for VFX shooting.

[0077] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0078] This embodiment includes the following configurations, methods, programs, and media. (Composition 1) An acquisition means for acquiring the aperture value of the lens of an imaging device, An output means that outputs, along with the image captured by the imaging device, the aperture value at the time of capturing the image, and blur shape-related information relating to the shape of the blur that occurs in the image. An information processing device characterized by having the following features. (Configuration 2) An acquisition means for acquiring the aperture value of the lens of the imaging device in real time, An output means that outputs the aperture value and blur shape-related information relating to the shape of the blur that occurs in the captured image of the imaging device in real time. An information processing device characterized by having the following features. (Composition 3) The system further includes a storage means for storing correspondence information that shows the correspondence between aperture value and bokeh shape related information. The output means acquires bokeh shape-related information corresponding to the aperture value to be output, based on the correspondence information, and outputs it. An information processing device according to configuration 1 or 2, characterized by the above. (Composition 4) The storage means stores the correspondence information in advance. The information processing apparatus according to configuration 3, characterized by the features described herein. (Composition 5) The lens of the imaging device is replaceable. The storage means pre-stores multiple correspondence information corresponding to multiple lenses, The information processing device further includes a second acquisition means for acquiring identification information of the lens of the imaging device, The output means acquires bokeh shape-related information corresponding to the aperture value to be output, based on correspondence information corresponding to the lens of the imaging device, and outputs it. The information processing apparatus according to configuration 4, characterized by the features described above. (Composition 6) The system further includes a third acquisition means for acquiring the aforementioned correspondence information from an external source. The information processing apparatus according to configuration 3, characterized by the features described herein. (Composition 7) The aforementioned bokeh shape-related information indicates the shape of the bokeh. An information processing device according to any one of configurations 1 to 6, characterized by the above. (Composition 8) The aforementioned blur shape-related information further indicates the degree of similarity of the blur shape to a perfect circle. The information processing apparatus according to configuration 7, characterized by the features described above. (Composition 9) The aforementioned blur shape-related information indicates the degree of distortion of the blur shape relative to a perfect circle or regular polygon. Ra The information processing apparatus according to configuration 7, characterized by the features described above. (Composition 10) The output means outputs the aperture value and the bokeh shape-related information to an external source. An information processing device according to any one of configurations 1 to 9, characterized by the above. (Composition 11) The output means records the aperture value and the bokeh shape-related information on a recording medium provided in the information processing device. An information processing device according to any one of configurations 1 to 9, characterized by the above. (Composition 12) The aforementioned blur shape-related information is applied to the graphics to be composited onto the captured image. An information processing device according to any of the features of configuration 1. (Composition 13) The aforementioned blur shape-related information is applied to the graphics displayed on the display device captured by the imaging device. An information processing device according to any of the features of configuration 2. (Composition 14) An instruction means for instructing the start of the operation of the aperture, Control means and It further possesses, The control means is The first time is the time required to drive the aperture of the aforementioned lens, The second time is the time from when the output means outputs the blur shape-related information until the display device starts displaying the graphic after the blur shape-related information has been applied. Based on this, the timing at which the output means outputs the bokeh shape-related information, the timing at which the instruction means instructs the start of the aperture drive, and the timing at which the imaging device takes a picture are controlled. The information processing device described in configuration 13, characterized in that way. (Composition 15) The control means controls at least one of the timing at which the output means outputs the bokeh shape-related information and the timing at which the instruction means instructs the start of the aperture drive, such that the instruction means instructs the start of the aperture drive after the output means outputs the bokeh shape-related information. The information processing apparatus according to configuration 14, characterized by the features described herein. (Composition 16) The imaging device has interchangeable lenses. An information processing device according to any one of configurations 1 to 14, characterized by the above. (Composition 17) The imaging device has a non-replaceable lens. An information processing device according to any one of configurations 1 to 14, characterized by the above. (Composition 18) The lens is detachable from the imaging device. An information processing device according to any one of configurations 1 to 14, characterized by the above. (Composition 19) The output means outputs metadata containing the aperture value and bokeh shape-related information stored in a manufacturer-specific area. An information processing device according to any one of configurations 1 to 18, characterized by the above. (Composition 20) The output means outputs the aperture value and the bokeh shape-related information using the manufacturer's proprietary standard. An information processing device according to any one of configurations 1 to 19, characterized by the above. (Composition 21) The aperture value is a 16-bit resolution, logarithmic value. An information processing device according to any one of configurations 1 to 20, characterized by the above. (Method 1) A step of obtaining the aperture value of the lens of the imaging device, The steps include outputting, along with the image captured by the imaging device, the aperture value at the time of capturing the image, and blur shape-related information regarding the shape of the blur that occurs in the image. An information processing method characterized by having the following features. (Method 2) The steps include acquiring the aperture value of the lens of the imaging device in real time, A step of outputting the aperture value and blur shape-related information relating to the shape of blur generated in the image captured by the imaging device in real time. An information processing method characterized by having the following features. (program) A program for causing a computer to function as one of the means of the information processing device described in any of configurations 1 to 21. (medium) A computer-readable storage medium containing a program that causes the computer to function as one of the means of the information processing device described in any of configurations 1 to 21. [Explanation of Symbols]

[0079] 100: Camera body 102: CPU 1400: Camera 1402: CPU 1500: Lens unit 1502: LPU

Claims

1. An acquisition means for acquiring the aperture value of the lens of the imaging device in real time, An output means that outputs the aperture value and blur shape-related information relating to the shape of the blur that occurs in the captured image of the imaging device in real time, Timing control means and It has, The aforementioned blur shape-related information is applied to the graphics displayed on the display device captured by the imaging device. The timing control means controls at least one of the timings at which the output means outputs the bokeh shape-related information and the timing at which the imaging device takes a picture, based on a first time which is the time required to drive the aperture of the lens and a second time which is the time from when the output means outputs the bokeh shape-related information until the display device starts displaying the graphic after the bokeh shape-related information has been applied. An information processing device characterized by the following:

2. The system further includes a storage means for storing correspondence information that shows the correspondence between aperture value and bokeh shape related information. The output means acquires bokeh shape-related information corresponding to the aperture value to be output, based on the correspondence information, and outputs it. The information processing apparatus according to feature 1.

3. The storage means stores the correspondence information in advance. The information processing apparatus according to feature 2.

4. The storage means pre-stores multiple correspondence information corresponding to multiple lenses, The information processing device further includes a second acquisition means for acquiring identification information of the lens of the imaging device, The output means acquires bokeh shape-related information corresponding to the aperture value to be output, based on correspondence information corresponding to the lens of the imaging device, and outputs it. The information processing apparatus according to claim 3.

5. The system further includes a third acquisition means for acquiring the aforementioned correspondence information from an external source. The information processing apparatus according to feature 2.

6. The aforementioned bokeh shape-related information indicates the shape of the bokeh. The information processing apparatus according to feature 1.

7. The aforementioned blur shape-related information further indicates the degree of similarity of the blur shape to a perfect circle. The information processing apparatus according to feature 6.

8. The aforementioned blur shape-related information further indicates the degree of distortion of the blur shape relative to a perfect circle or regular polygon. The information processing apparatus according to feature 6.

9. The output means outputs the aperture value and the bokeh shape-related information to an external source. The information processing apparatus according to feature 1.

10. The output means records the aperture value and the bokeh shape-related information on a recording medium provided in the information processing device. The information processing apparatus according to feature 1.

11. The imaging device has interchangeable lenses. The information processing apparatus according to feature 1.

12. The imaging device has a non-replaceable lens. The information processing apparatus according to feature 1.

13. The lens is detachable from the imaging device. The information processing apparatus according to feature 1.

14. The output means outputs metadata containing the aperture value and bokeh shape-related information stored in a manufacturer-specific area. The information processing apparatus according to feature 1.

15. The output means outputs the aperture value and the bokeh shape-related information using the manufacturer's proprietary standard. The information processing apparatus according to feature 1.

16. The aperture value is a 16-bit resolution, logarithmic value. The information processing apparatus according to feature 1.

17. An acquisition step to acquire the aperture value of the lens of the imaging device in real time, An output step that outputs the aperture value and blur shape-related information relating to the shape of the blur that occurs in the captured image of the imaging device in real time, Timing control step and It has, The aforementioned blur shape-related information is applied to the graphics displayed on the display device captured by the imaging device. In the timing control step, a first time is the time required to drive the aperture of the lens, and after outputting the bokeh shape-related information in the output step, the display device displays the bokeh shape. Based on a second time, which is the time until the display of the graphic after applying the shape-related information begins, the timing of outputting the blur shape-related information in the output step and the timing of the imaging device taking a picture are controlled. An information processing method characterized by the following:

18. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 16.

19. A computer-readable storage medium storing a program for causing a computer to function as one of the means of the information processing apparatus described in any one of claims 1 to 16.

Images