Information processing device, information processing method and computer-readable medium

Abstract

An information processing apparatus according to the present invention includes an acquisition unit configured to acquire an aperture value of a lens of an imaging device, and an output unit configured to output the aperture value acquired during capturing of a captured image of the imaging device and blur shape related information related to a shape of blur occurring in the captured image.

Description

BACKGROUND OF THE INVENTIONField of the invention

[0001] The present invention relates to an information processing apparatus, and more particularly relates to a technique for synthesizing a real object (object in real life) with a graphic (a computer graphics (CG) image). Description of related technology

[0002] Recently, there has been a growing demand for video content using a visual effects technique to synthesize a real object with a graphic (a CG image). Shooting video content using a visual effects technique is generally referred to as VFX shooting. There are broadly two types of VFX shooting. A first method is a post-production method in which a shot is made using a special background such as a green screen, and then a graphic is synthesized onto the background of the shot image during post-production. A second method is an in-camera method in which a shot is made with a graphic displayed on a large display device as the background.In VFX shooting, using CG, a realistic image can be obtained without going to a physical location, or an image can be obtained with a camera angle or composition that is difficult to achieve in reality, thus reducing production costs to a minimum. Therefore, the demand for VFX shooting is increasing. In VFX shooting, there is a desire to obtain a natural synthetic image (a synthetic image without discomfort).

[0003] JP 2004-227332 A discloses a technique for changing a synthetic position of a graphic according to a camera orientation. JP 2021-532649 A discloses a technique for generating a point spread function based on a distance from an entrance aperture of a lens and a size of an exit aperture, and generating a graphic based on the point spread function.

[0004] Meanwhile, in related art, a camera with a 2 / 3-inch or 1-inch imaging element has generally been used. However, in recent years, the number of cameras with a large imaging element, such as SUPER 35mm or full-frame 35mm, has increased. As the imaging element size increases, the adjustable depth of field becomes wider, and the degree of freedom of blur expression increases. Important factors of blur include intensity (degree of spread) and shape, which can be expressed by a point-image intensity distribution function. In VFX shooting of related art, graphics that consider the blur shape appearing in a captured image cannot be generated, and a natural synthetic image cannot be obtained. SUMMARY OF THE INVENTION

[0005] The present invention provides a technique that makes it possible to obtain a more natural image (image with less discomfort) than a synthetic image obtained by synthesizing a real object with a graphic.

[0006] According to its first aspect, the present invention provides an information processing apparatus comprising an acquisition unit configured to acquire an aperture value of a lens of an imaging device, and an output unit configured to output the aperture value acquired during capturing a captured image of the imaging device and blur shape-related information related to a shape of blur occurring in the captured image.

[0007] According to its second aspect, the present invention provides an information processing method comprising a step of acquiring an aperture value of a lens of an imaging device and a step of outputting the aperture value acquired during capturing a captured image of the imaging device and blur shape related information related to a shape of blur occurring in the captured image.

[0008] According to its third aspect, the present invention provides a computer-readable medium storing a program for causing a computer to operate as each unit of the above-described information processing apparatus.

[0009] Further features of the present invention will become apparent from the following description of embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 shows a block diagram of a camera according to a first embodiment, Fig. 2 shows a schematic representation of a recording location, Fig. 3 shows a schematic representation of a synthetic image, Fig. 4 shows a schematic representation of a recording location, Fig. Figure 5 shows a schematic representation of a synthetic image, Fig. 6 shows a schematic representation of an aperture, Fig. 7 shows a schematic representation of a blur shape, Fig. 8 shows a schematic representation of correspondence relationship information, Fig. 9 shows a schematic representation of a blur shape, Fig. 10 shows a schematic representation of correspondence relationship information, Fig. 11 is a schematic diagram showing communication between a camera main body and a lens unit, Fig. 12 is a schematic diagram showing communication between the camera main body and the lens unit, Fig. 13 is a schematic diagram showing communication between the camera main body and the lens unit, Fig. 14 is a schematic diagram showing communication between the camera main body and a CG generating device, Fig. 15 is a schematic diagram showing communication between the lens unit, the camera main body and the CG generating device, Fig. 16 shows a block diagram of a camera according to a second embodiment, and Fig. 17 shows a block diagram of a lens unit according to a third embodiment. DESCRIPTION OF THE EMBODIMENTSFirst embodiment

[0010] A first embodiment of the present invention will be described below. Fig. Figure 1 shows a block diagram of the structure of a camera main body 100 as an example of an information processing device according to a first embodiment. The camera main body 100 is an imaging device in which a lens unit (lens and lens device) is interchangeable. According to Fig. 1, a lens unit 200 is attached to the camera main body 100.

[0011] The camera main body 100 includes a memory 101, a central processing unit (CPU) 102, an imaging element 103, a communication port 104, an output port 105, and a recording medium 106.

[0012] Memory 101 is a storage unit that stores various types of data (various types of information), including images. The data stored in memory 101 can be read from memory 101 as needed. Memory 101 includes a volatile region that can store data only during power supply and a non-volatile region that can store data even while the power supply is stopped.

[0013] The CPU 102 is a control unit that controls each unit of the camera main body 100 and each unit of an accessory attached to the camera main body 100 (the lens unit 200 in Fig. 1). For example, programs, various parameters, and the like for operating the CPU 102 are stored in the memory 101, and the CPU 102 performs various controls by reading the programs from the non-volatile region of the memory 101, loading the programs into the volatile region, and executing the programs.

[0014] The imaging element 103 is, for example, a charge storage type solid-state imaging element such as a CMOS or a CCD, and captures a light flux (light flux of an object) introduced into the camera main body 100 via the lens unit 200 and converts the light flux into an electrical image signal. An image (signal) obtained by the imaging element 103 is used for live view display, recording on the recording medium 106 (described below), external output using the output terminal 105, and the like under the control of the CPU 102. The CPU 102 can also control an exposure time of the imaging element 103, a capture timing (a timing for performing capture), and the like.

[0015] The communication port 104 is a port for communicating with a lens unit (lens unit 200 in Fig. 1).

[0016] The output terminal 105 is, for example, an Ethernet terminal, an SDI terminal, an HDMI (registered trademark) terminal, or the like, and is used to output various types of data (various types of information) including an image to the outside or acquire various types of data (various types of information) from the outside.

[0017] The recording medium 106 is a recording medium detachable from the camera main body 100, and is, for example, an SD card, CFExpress, or the like. Various types of data (various types of information) including images can be recorded on the recording medium 106.

[0018] The lens unit 200 includes a memory 201, a lens processing unit (LPU) 202, a communication port 203, a diaphragm 204, and a lens group 205. The lens unit 200 is a so-called interchangeable lens that is removable from an imaging device.

[0019] Memory 201 is a storage unit that stores various types of data (various types of information). The data stored in memory 201 can be read from memory 201 as needed. Memory 201 includes a volatile region that can store data only during power supply and a non-volatile region that can store data even while power supply is stopped.

[0020] The LPU 201 is a control unit that controls each unit of the lens unit 200. For example, programs, various parameters, and the like for operating the LPU 202 are stored in the memory 201, and the LPU 202 performs various controls by reading the programs from the non-volatile region of the memory 201, loading the programs into the volatile region, and executing the programs.

[0021] The communication port 203 is a port for communicating with an imaging device (the camera main body 100 in Fig. 1), to which the lens unit 200 is attached. In Fig. 1, the LPU 202 of the lens unit 200 and the CPU 102 of the camera main body 100 are connected to each other via the communication port 203 of the lens unit 200 and the communication port 104 of the camera main body 100. The LPU 202 can drive (control) each unit of the lens unit 200 according to a control instruction from the CPU 102.

[0022] The aperture is a light quantity control component that controls (adjusts) the amount of light from the light flux introduced into the camera main body 100. For example, by changing an aperture value, the aperture diameter of the aperture 204 is changed to an aperture diameter corresponding to the changed aperture value, and the amount of light is changed. Not only the amount of light, but also the depth of field and blur can be changed.

[0023] The lens group 205 includes a focusing lens, a zoom lens, a shift lens, and the like. The light flux of the object is guided into the camera main body 100 via the lens group 205 and the aperture 204. The position of each lens included in the lens group 205 can be controlled (changed). For example, the focusing lens can be moved in an optical axis direction to adjust the focal point on the object.

[0024] The camera main body 100 (and the lens unit 200) is used, for example, for VFX shooting to synthesize a real object with a graphic (a CG image).

[0025] A VFX shot of the post-production process is taken with reference to the Fig. 2 and Fig. 3 described. Fig. 2 shows a schematic representation of an example of a recording location, and Fig. 3 shows a schematic diagram of an example of a synthetic image obtained by synthesizing a real object with a graphic.

[0026] In the VFX recording of the post-production process, a shot with a special background is first taken. In the example of Fig. 2, a real object 301 is captured with a green screen 300 as the background. Note that the background is not limited to the green screen (solid green background) and may be, for example, another solid color background or another patterned background.

[0027] Next, a CG synthesis device transfers a captured image to CG synthesis editing software, captures a background region from the captured image, and synthesizes a background graphic at the captured region. Graphics different from the background can also be synthesized. In the example of Fig. 3, a background graphic 400 and graphics 402 to 404 different from the background are synthesized. The graphics 402 to 404 different from the background are objects arranged in front of the background graphic 400.

[0028] As metadata of the captured image, information such as an in-focus position is recorded during capture, and blurring of each graphic 404 and 402 to 404 is individually controlled (adjusted) based on the information. For example, a blurring style of the background graphic 400 differs from a blurring style of the graphics 402 to 404 other than the background. By generating and synthesizing the graphics 400 and 402 to 404 so that a blurring style of the real object 401 (301) and the blurring styles of the graphics 400 and 402 to 404 match each other, a natural synthetic image (synthetic image without discomfort) can be generated.

[0029] A VFX shot of the in-camera process is made with reference to the Fig. 4 and Fig. 5 described. Fig. 4 shows a schematic representation of an example of a recording location, and Fig. 5 shows a schematic diagram of an example of a synthetic image obtained by synthesizing a real object with a graphic.

[0030] In-camera VFX shooting involves shooting with a graphic displayed on a large display device as the background. In the example of Fig. 4, real objects 501 to 504 are captured with a graphic displayed on a display device 500 as a background. Here, information such as an in-focus position is output from a camera in real time (sequentially during shooting) and input to a CG generation device. The CG generation device generates a graphic with a blur based on the input information as a graphic to be displayed on the display device 500. By the camera capturing an image at a camera angle including the graphic displayed on the display device 500 and the real objects 501 to 505, a synthetic image of Fig. 5. Note that information such as an in-focus position may be transmitted from the camera main body to the CG generating device, or from the lens unit to the CG generating device without passing through the camera main body. Information such as an in-focus position may be transmitted from the lens unit to the CG generating device via a device other than the camera main body (for example, a device that converts externally acquired information into information that can be input to the CG generating device).

[0031] A blur is described. Blur includes intensity (degree of spread) and shape as important factors. In VFX shooting of the related art, the intensity of a blur graphic is controlled (adjusted) based on an aperture value, an in-focus position, and the like. However, a graphic that takes into account the shape of a blur appearing in a captured image cannot be generated, and a natural synthetic image cannot be obtained.

[0032] In the captured image, blur occurs for an object that is not in focus. Blur intensity is determined according to a degree of defocus, such as a deviation amount from a depth of field based on an aperture value or an imaging element size, or a deviation amount from an object distance corresponding to an in-focus position (focal point position). Meanwhile, a blur shape is not determined according to the aperture value, the imaging element size, the in-focus position, and the like, but is determined according to a shape of an aperture opening.

[0033] Fig. Figure 6 shows a schematic representation of an example of a circular diaphragm with a circular opening and a polygonal diaphragm (iris diaphragm) with a polygonal opening. Fig. 6 shows an aperture with an octagonal opening as a polygonal aperture. Fig. 7 is a schematic diagram showing an example of a blur shape that occurs in a captured image. For a circular aperture, a circular blur occurs, and for a polygonal aperture, a polygonal blur occurs. When the number of aperture blades of a polygonal aperture is eight, an octagonal blur occurs. If an edge of the aperture blade that forms the aperture opening has a curvature, a blur with a polygonal shape close to a circle may occur instead of a regular polygon. If the aperture of the aperture becomes circular due to opening, a circular blur occurs even with a polygonal aperture. As described above, the blur shape depends on the specification of the lens unit (aperture) and the aperture value.For the convenience of description, the aperture shape is described here when an ideal point light source is shot, but blur having a shape according to the shape described above also occurs in a region of an object other than the point light source.

[0034] If information that can specify (determine) the blur shape is not communicated to the CG generation device or the CG synthesis device, the blur shape of the generated graphic and the blur shape of the real object will not match, and an unnatural synthetic image (synthetic image with discomfort) will be generated. Therefore, in the first embodiment, the camera main body 100 outputs blur shape-related information, which is information regarding the blur shape, from the output terminal 105 or records the information on the recording medium 106, so that the CG generation device or the CG synthesis device can acquire the blur shape-related information.In the post-production method, the aperture value is acquired, and a captured image is output along with the aperture value during capture of the captured image and blur shape-related information regarding the shape of the blur occurring in the captured image. The processing may or may not be performed in real time. In the in-camera method, the aperture value is acquired in real time, and the aperture value and blur shape-related information are output in real time. The output aperture value is used to determine the blur intensity of the graphic, and the output blur shape-related information is used to determine the blur shape of the graphic. If the blur shape-related information is not information indicating the blur shape, the aperture value can be used to identify the blur shape.

[0035] A method for generating the blur shape-related information will be described. In the first embodiment, it is assumed that the blur shape-related information indicates a blur shape. As described above, the blur shape depends on the specification of the lens unit (aperture) and the aperture value. Therefore, if the lens unit to be used is determined in advance, the blur shape-related information can be generated (acquired) based only on the aperture value. In the first embodiment, the memory 101 stores correspondence information indicating a correspondence relationship between the aperture value and the blur shape-related information. Then, the CPU 102 acquires the blur shape-related information corresponding to the aperture value to be output based on the correspondence information and outputs the blur shape-related information.

[0036] Fig. Figure 8 shows a schematic representation of an example of the correspondence relationship information. In Fig. 8, Tables 1 to 3 are illustrated as the correspondence relationship information. In Table 1, the blur shape-related information indicates the blur shape, and a variety of combinations of the aperture value and the blur shape are described. Using Table 1, the blur shape can be known from the aperture value. Note that the aperture value described in the table may only be a representative value, and the blur shape-related information corresponding to the aperture value not described in the table can be acquired by interpolation processing or the like using the information described in the table.In Table 2, the blur shape-related information indicates the blur shape and roundness (similarity of the blur shape compared to a perfect circle), and a variety of combinations of the aperture value, blur shape, and roundness are described. Using Table 2, a more accurate blur shape can be known when the roundness changes depending on the aperture value. Table 3 illustrates a threshold value, which is an aperture value at which a generated blur alternates between a circular blur and a polygonal blur, and a shape (polygonal shape) of blur that occurs when the aperture value is smaller than the threshold value (on the aperture side).Using Table 3, it can be determined that circular blur occurs when the aperture value is larger than the threshold (on the aperture side), and it can be determined that polygonal blur occurs when the aperture value is smaller than the threshold (on the aperture side). Note that the threshold may be a lower limit of an aperture value at which circular blur occurs or an upper limit of an aperture value at which polygonal blur occurs.

[0037] Examples of a special blur form include the blur form created when an anamorphic lens, or distorting lens, is attached. When the distorting lens is used, a captured image is obtained in which an object is compressed horizontally. By decompressing the captured image horizontally during post-production, an image is created that shows a landscape camera angle compared to a normal captured image. In the captured image, the object is compressed horizontally, and a blur is also compressed horizontally. Fig. Figure 9 is a schematic diagram illustrating an example of a blur shape that appears in a captured image obtained using a distorting lens. A circular blur becomes an elliptical blur, and a regular octagonal blur becomes a vertically long octagonal blur.

[0038] Fig. Figure 10 shows a schematic representation of an example of correspondence relationship information in which a special blur form is taken into account. Fig. 10, Tables 4 to 6 illustrate the correspondence relationship information. In Table 4, the blur shape-related information indicates the blur shape, and a variety of combinations of the aperture value and the blur shape are described. An elliptical shape is illustrated as the blur shape during opening, and in other cases, a vertically elongated hexagon is illustrated as the blur shape. In Table 5, the blur shape-related information indicates the blur shape and a flatness ratio (a degree of collapse of the blur shape compared to a perfect circle or a regular polygon), and a variety of combinations of the aperture value, the blur shape, and the flatness ratio are described. The degree of collapse is not limited to the flatness ratio and can be, for example, eccentricity.Table 6 shows a threshold value, which is an aperture value at which a generated blur changes between an elliptical blur and a vertically long polygonal blur, a blur shape (vertically long polygon) that occurs when the aperture value is smaller than the threshold value (on the aperture side), and the flatness ratio common to an elliptical blur and a vertically long polygonal blur. Using Table 6, it can be determined that an elliptical blur occurs when the aperture value is larger than the threshold value (on the aperture side), and it can be determined that a vertically long polygonal blur occurs when the aperture value is smaller than the threshold value (on the aperture side).It is noted that the threshold may be a lower limit of an aperture value at which elliptical blur occurs or an upper limit of an aperture value at which vertically long polygonal blur occurs.

[0039] Although an example is described in which the correspondence information is a table, the correspondence information may be any type of information that can be used to generate (obtain) the blur shape-related information from the aperture value, and may be, for example, a function. Although an example is described in which the blur shape-related information indicates the blur shape, the blur shape-related information may be any type of information as long as the blur shape can be determined from the aperture value, and may, for example, indicate a number of aperture blades, a shape of aperture blades, a type of aperture (circular aperture / polygonal aperture), and the like. To determine the blur shape from this type of information, the correspondence information must be stored as a table in a similar manner.

[0040] A time for obtaining the information related to the blur form is determined with reference to the Fig. 11 and Fig. 12. The Fig. 11 and Fig. 12 show schematic diagrams of communication between the camera main body 100 (communication port 104) and the lens unit 200 (communication port 203).

[0041] In the example of Fig. 11, a plurality of pieces of correspondence relationship information, each corresponding to the plurality of lens units, is stored in advance in the non-volatile region of the memory 101. When the lens unit 200 is mounted on the camera main body 100, the CPU 102 requests identification information of the lens unit 200 from the LPU 202. When the power supply to the lens unit 200 is started, the LPU 202 reads identification information of the lens unit 200 from the non-volatile region of the memory 201 in response to the request from the CPU 102 and transmits the read identification information to the CPU 102. The identification information of the lens unit 200 may be any type of information that identifies the lens unit 200, for example, a name, a model number, a manufacturing number, and the like.

[0042] Although the identification information is transmitted from the lens unit 200 to the camera main body 100, the identification information may be generated in the camera main body 100. For example, a plurality of identification buttons that can be pressed by the lens unit may be provided in the camera main body 100, and a plurality of pieces of identification information corresponding to the plurality of lens units may be stored in advance in the non-volatile region of the memory 101. Then, the CPU 102 may retrieve any of the plurality of pieces of identification information from the memory 101 according to a pressed state of the plurality of identification buttons.

[0043] The timing of acquiring the identification information is not limited to the timing described above. For example, the identification information may be deleted from the memory 101 according to a transition to a power-saving state, and the identification information may be reacquired at a time of returning from the power-saving state.

[0044] When the identification information is acquired, the CPU 102 acquires the correspondence relationship information corresponding to the lens unit 200 from the memory 101 based on the identification information.

[0045] Next, the CPU 102 requests the aperture value from the LPU 202, and the LPU 202 transmits the aperture value to the CPU 102 in response to the request. The CPU acquires the blur shape-related information corresponding to the acquired aperture value based on the correspondence information. The transmission and reception of the aperture value and the acquisition of the blur shape-related information are performed repeatedly. These processes can be performed for each frame or can be performed only when a change in the aperture value occurs.

[0046] In the example of Fig. 12, the correspondence information of the lens unit 200 is stored in advance in the non-volatile region of the memory 201. When the lens unit 200 is mounted on the camera main body 100, the CPU 102 requests the correspondence information of the lens unit 200 from the LPU 202. When power is supplied to the lens unit 200, the LPU 202 reads the correspondence information of the lens unit 200 from the non-volatile region of the memory 201 in response to the request from the CPU 102 and transfers the read correspondence information to the CPU 102. The CPU 102 stores the acquired correspondence information in the volatile region of the memory 101.

[0047] The timing of acquiring the correspondence information is not limited to the timing described above. For example, the correspondence information may be deleted from the memory 101 according to a transition to a power-saving state, and the correspondence information may be reacquired at a time of returning from the power-saving state.

[0048] When the correspondence relationship information is acquired, the CPU 102 requests the aperture value from the LPU 202, and the LPU 202 transmits the aperture value to the CPU 102 in response to the request. The CPU 102 acquires the blur shape-related information corresponding to the acquired aperture value based on the acquired correspondence relationship information (correspondence relationship information stored in the memory 101).

[0049] The CPU 102 outputs the aperture value and the blur shape-related information acquired by the above method. For example, the CPU 102 outputs the aperture value and the blur shape-related information from the output terminal 105 to the outside, or records the aperture value and the blur shape-related information in the recording medium 106 in association with the captured image (frame). Here, the CPU 102 may store or output the aperture value and the blur shape-related information in a manufacturer-specific region of existing metadata or an existing protocol, such as RDD-18 of EXIF ​​or SMPTE. The CPU 102 may record or output the aperture value and the blur shape-related information using a manufacturer-specific standard (for example, a manufacturer-specific communication standard).The CPU 102 may record or output an aperture value defined by a resolution or format suitable for specifying blur separately from the aperture value stored in the existing metadata. For example, an F-number may be an F-number of a resolution of 0.01 obtained by multiplying the aperture value by 100, or an F-number in a log format of a 16-bit resolution.

[0050] As described above, the output aperture value is used to determine the blur intensity of the graphic, and the output blur shape-related information is used to determine the blur shape of the graphic. However, an object for applying the output aperture value and the output blur shape-related information varies depending on a method of VFX shooting. In the in-camera method, the aperture value and the blur shape-related information are applied to the graphic to be displayed on the display device that is one of the shooting targets, and in the post-production method, the aperture value and the blur shape-related information are applied to the graphic to be synthesized with the shot image.

[0051] The in-camera method will be described in detail. The CPU 102 outputs the aperture value and blur shape-related information to the CG generation device connected to the output terminal 105 in real time (by sequentially processing the input captured image). The CG generation device generates a graphic based on the aperture value and blur shape-related information acquired from the camera main body 100 (the CPU 102) and outputs the generated graphic to the display device. After the graphic without blur is generated, blur based on the aperture value and blur shape-related information can be applied to the graphic, or a graphic with blur based on the aperture value and blur shape-related information can be generated without generating the graphic without blur.The method of applying blur is not particularly limited, but blur is applied to each region, for example, through filter processing using a filter based on the aperture value and blur shape information. The display device displays the graphic generated by the CG generation device. After that, the camera main body 100 captures an image at a camera angle including the graphic displayed on the display device and the real object, thereby obtaining a captured image that is a natural synthetic image (synthetic image without discomfort) in which a blur of the real object and a blur of the graphic match.Note that a device that generates a graphic and a device that outputs a graphic to a display device (a device that controls a display on the display device) may be different. As the CG generation device, a CG generation system including a plurality of devices may be used.

[0052] The post-production process will be described in more detail. The CPU 102 records the captured image, the aperture value, and the blur shape-related information in association with each other on the recording medium 106. The captured image file, the aperture value file, and the blur shape-related information file may be the same or different from each other. The captured image, the aperture value, and the blur shape-related information may be recorded on different media. Afterward, the recording medium 106 is removed from the camera main body 100 and inserted into the CG synthesis device, and the captured image, the aperture value, and the blur shape-related information are transferred to the CG synthesis device. The CG synthesis device generates a graphic based on the aperture value and the blur shape-related information.As with the in-camera method, after generating a graphic with no blur, a blur based on the aperture value and blur shape information can be applied to the graphic. A graphic with blur based on the aperture value and blur shape information can be generated without generating a graphic with no blur. The method of applying blur is not particularly limited, but blur is applied, for example, by filter processing using a filter based on the aperture value and blur shape information for each region. Then, the CG synthesizer synthesizes the generated graphic with the captured image. As a result, a natural synthetic image (synthetic image without discomfort) can be obtained in which a blur of the real object and a blur of the graphic are matched.

[0053] Finally, a time of the respective processing in a VFX recording of the in-camera process is determined with reference to the Fig. 13 to 15 described. Fig. 13 is a schematic diagram illustrating an example of the timing of communication between the camera main body 100 (communication port 104) and the lens unit 200 (communication port 203) and the timing of the camera main body 100 (the imaging element 103) capturing images. The imaging element 103 captures images at a specific time interval according to a set frame rate.

[0054] In Fig. 13, control of the iris 204 occurs during a period of frame 2 due to a change in the brightness of an object or a user operation. Specifically, after the start of the period of frame 2, the CPU 102 commands the LPU 202 to start driving the iris 204. Then, the LPU 202 controls a motor that controls the iris 204 according to an instruction from the CPU 102 and changes an opening diameter of the iris 204. When the control of the iris 204 is completed, the LPU 202 notifies the CPU 102 that the control of the iris 204 is completed. As notification of completion of the control of the iris 204, notification of the changed aperture value or the like may be performed. The notification may be started before the control of the iris 204 is completed, so that the notification is not delayed due to a communication time.For example, a time may be predicted at which the control of the shutter 204 is completed, and the notification may be started at a time approximately the communication time before the predicted time. In . Fig. 13, the CPU 102 receives the notification of the completion of the control of the aperture 204 during a period of frame 3. Therefore, a captured image corresponding to the changed aperture value is obtained as the captured image of frame 3 and subsequent frames.

[0055] In Fig. 13, a time t1 is required to control the aperture 204. It is noted that the time t1 in Fig. 13 is an example, and a time required for driving the diaphragm 204 may be longer or shorter than the time t1. An upper limit of a driving speed of the diaphragm 204 may be determined in consideration of noise during shooting, and the time required for driving the diaphragm 204 (change in aperture diameter) may depend on a driving amount (change amount) of the diaphragm 204. The time t1 is a time from when the CPU 102 instructs the start of driving the diaphragm 204 until the notification of completion of driving the diaphragm 204 is received, however, the time required for driving the diaphragm 204 may be a time until completion of driving the diaphragm 204 or a time since the start of driving the diaphragm 204.

[0056] Fig. Figure 14 is a schematic diagram illustrating an example of the timing of communication between the camera main body 100 (output terminal 105) and the CG generation device. Transmission and reception of the captured image and the aperture value are omitted. Fig. 14, the blur shape-related information is changed in a time period of frame 1. The CPU 102 notifies the CG generation device of the changed blur shape-related information, and the CG generation device generates (updates) the graphic based on the changed blur shape-related information. Then, the CG generation device outputs the generated graphic to the display device, and the display device displays the graphic. Fig. 14 starts displaying the graphic based on the changed blur shape related information during a period of frame 5. Therefore, the captured image including the graphic corresponding to the changed blur shape related information is obtained as the captured image of frame 5 and subsequent frames.

[0057] In Fig. 14, a time t2 is required from the output of the blur shape related information by the CPU 102 until the display device starts displaying the graphic after applying the blur shape related information.

[0058] Generally, time t1 and time t2 are different. Although the time depends on a computing power of the CG generation device, a relatively long time is generally required to generate a graphic, and time t2 is longer than time t1. Therefore, when the CPU 102 outputs an aperture drive start instruction and the blur shape-related information at the same time, a timing at which the changed aperture value is applied to the captured image (real object) does not coincide with a timing at which the changed blur shape-related information is applied to the graphic.In a period during which only the changed aperture value or the changed blur shape related information is / are applied, an unnatural synthetic image (synthetic image with discomfort) in which the blur shape of the graphic and the blur shape of the real object do not match is obtained as the captured image.

[0059] Therefore, in the first embodiment, the CPU 102 controls a timing of instructing the drive start of the diaphragm 204 and / or a timing of outputting the blur shape related information based on the times t1 and t2. Fig. Figure 15 illustrates an example of different timings controlled based on times t1 and t2. Since time t2 is longer than time t1, CPU 102 instructs Fig. 15, the control start of the aperture 204 after outputting the blur shape-related information. Specifically, the CPU 102 instructs the control start of the aperture 204 at a time when a time (t2 - t1) has passed after outputting the blur shape-related information. The CPU 102 knows the changed aperture value in advance when instructing the control start of the aperture 204 and outputs the blur shape-related information corresponding to the changed aperture value. Then, the CPU 102 issues an instruction to control the aperture 204 with a control amount based on the changed aperture value as the control start instruction of the aperture 204. Thus, the timing at which the changed aperture value is applied to the captured image (real object) and the timing at which the changed blur shape-related information is applied to the graphic can be brought close to each other (coincided).As a result, it is possible to shorten (eliminate) a period of time during which an unnatural captured image (unnatural synthetic image) is obtained.

[0060] The CPU 102 can control the timing at which the camera main body 100 (the imaging element 103) performs a capture of each frame based on the times t1 and t2 so that the aperture value and blur shape-related information are not changed mid-frame. The CPU 102 controls the timing of outputting the blur shape-related information and / or the timing of instructing the start of driving the aperture 204 and / or the timing of capturing based on the times t1 and t2. Information about the times t1 and t2 may or may not be acquired through communication. The user can measure the times t1 and t2 and input the times t1 and t2 to the camera main body 100. Second embodiment

[0061] A second embodiment of the present invention will be described below. In the first embodiment, the present invention is applied to an imaging device in which a lens unit is interchangeable (interchangeable-lens camera). In the second embodiment, the present invention is applied to an imaging device in which a lens unit is not interchangeable (lens-integrated camera). Note that common parts with the first embodiment will not be described again.

[0062] Fig. Figure 16 shows a block diagram of the structure of a camera 1400 according to the second embodiment. The camera 1400 includes a memory 1401, a CPU 1402, an imaging element 1403, an output terminal 1404, a recording medium 1405, a diaphragm 1406, and a lens group 1407. The memory 1401, the CPU 1402, the imaging element 1403, the output terminal 1404, and the recording medium 1405 have functions similar to those of the memory 101, the CPU 102, the imaging element 103, the output terminal 105, and the recording medium 106 in Fig. 1. The aperture 1406 and the lens group 1407 have functions similar to the functions of the aperture 204 and the lens group 205. Furthermore, the memory 1401 has functions similar to at least part of the function of the memory 201, and the CPU 1402 further has functions similar to at least part of the function of the LPU 202.

[0063] Since the camera 1400 has a structure in which the lens unit and the camera main body are integrated, the aperture 1406 and the lens group 1407 are directly connected to the CPU 1402, and the CPU 1402 transmits a control signal directly to the aperture 1406 and the lens group 1407. Therefore, the CPU 1402 can know an aperture value of the aperture 1406 in real time. Correspondence information indicating a correspondence relationship between the aperture value and the blur shape-related information is stored in a non-volatile region of the memory 1401, and the CPU 1402 can acquire the blur shape-related information corresponding to the aperture value to be output from the memory 1401.The CPU 1402 may acquire the blur shape related information from the non-volatile region of the memory 1401, or may load the correspondence relationship information from the non-volatile region of the memory 1401 into a volatile region and acquire the blur shape related information from the volatile region. Third embodiment

[0064] A third embodiment of the present invention will be described below. In the third embodiment, the present invention is applied to a lens unit detachable from an imaging device. Note that common portions with the first embodiment will not be described again.

[0065] Fig. Fig. 17 shows a block diagram of the structure of a lens unit 1500 according to the third embodiment. The lens unit 1500 includes a memory 1501, an LPU 1502, an operating element 1503, a communication port 1504, a diaphragm 1505, a lens group 1506, and an output port 1507. The memory 1501, the LPU 1502, the communication port 1504, the diaphragm 1505, the lens group 1506, and the output port 1507 have functions similar to those of the memory 201, the LPU 202, the communication port 203, the diaphragm 204, the lens group 205, and the output port 1507 in Fig. 1. The memory 1501 further has functions similar to at least part of the function of the memory 101, and the LPU 1502 further has functions similar to at least part of the function of the CPU 102.

[0066] The operating element 1503 is an operating element that can receive an operation to change the opening diameter (aperture value) of the aperture 1505, and is, for example, an aperture ring or an interactive control panel. The LPU 1502 controls the aperture 1505 according to the operation on the operating element 1503 and acquires the changed aperture value. The LPU 1502 can be instructed to control the aperture 1505 by the camera main body to which the lens unit 1500 is attached via the communication port 1504. Here, too, the LPU 1502 controls the aperture 1505 according to the instruction from the camera main body and acquires the changed aperture value. The LPU 1502 can acquire a captured image from the camera main body to which the lens unit 1500 is attached via the communication port 1504.

[0067] Correspondence information indicating a correspondence between the aperture value and the blur shape-related information is stored in a non-volatile region of the memory 1501, and the LPU 1502 can acquire the blur shape-related information corresponding to the aperture value and output it from the memory 1501. The LPU 1502 can acquire the blur shape-related information from the non-volatile region of the memory 1501, or can load the correspondence information from the non-volatile region of the memory 1501 into a volatile region and acquire the blur shape-related information from the volatile region. The LPU 1502 outputs the acquired captured image, the acquired aperture value, the acquired blur shape-related information, and the like to the outside from the output terminal 1507 or the communication terminal 1504.

[0068] It should be noted that the various control types described above may represent processing performed by a piece of hardware (e.g., a processor or circuit) or in another way. Processing may be distributed among a plurality of pieces of hardware (e.g., a plurality of processors, a plurality of circuits, or a combination of one or more processors and one or more circuits), thereby executing control of the entire device.

[0069] The above processor is also a processor in the broad sense and includes general-purpose processors and dedicated processors. Examples of general-purpose processors include a central processing unit (CPU), a microprocessing unit (MPU), a digital signal processor (DSP), etc. Examples of dedicated processors include a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), etc. Examples of PLDs include a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), etc.

[0070] The above-described embodiment (including modification examples) is merely an example. Any structure obtained by appropriately modifying or changing some configuration of the embodiment within the scope of the subject matter of the present invention is also included in the present invention. The present invention also includes other configurations obtained by appropriately combining various features of the embodiment.

[0071] In the above-described embodiments, the present invention is applied to a camera or a lens unit, but the present invention can be applied to any electronic device (information processing device) that can output information useful for VFX shooting.

[0072] According to the present invention, it is possible to obtain a more natural image (image with less discomfort) than a synthetic image obtained by synthesizing a real object with a graphic. Further examples

[0073] Embodiments of the present invention may also be implemented by a computer of a system or apparatus that retrieves and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a “non-transitory computer-readable storage medium”) for performing the functions of one or more of the embodiments described above, or that includes one or more circuits (e.g., an application-specific integrated circuit (ASIC)) for performing the functions of one or more of the embodiments described above,and by a method performed by the computer of the system or device, for example, by reading and executing the computer-executable instructions from the storage medium to perform the functions of one or more of the above-described embodiments and / or controlling the one or more circuits to perform the functions of one or more of the above-described embodiments. The computer may comprise one or more processors (e.g., central processing unit (CPU),Microprocessing Unit (MPU)) and may include a network of separate computers or separate processors for reading and executing the computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, a hard disk and / or random access memory (RAM) and / or read-only memory (ROM) and / or distributed computing system memory and / or an optical disk (such as a compact disk (CD), digital versatile disk (DVD), or Blu-ray Disk (BD)™) and / or a flash memory device and / or a memory card, and the like.

[0074] Although the invention has been described with reference to embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but is defined by the scope of the following claims. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2004-227332 A

[0003] JP 2021-532649 A

[0003]

Claims

[1] Information processing device with a procurement unit configured to procure an aperture value of a lens of an imaging device, and an output unit configured to output the aperture value acquired during capturing a captured image of the imaging device and blur shape-related information regarding a shape of blur occurring in the captured image. [2] The information processing apparatus according to claim 1, wherein the acquisition unit is configured to acquire the aperture value in real time during capturing the captured image, and the output unit is configured to output the aperture value and the blur shape related information in real time during capturing the captured image. [3] The information processing apparatus according to claim 1 or 2, wherein the output unit is further configured to output the captured image of the imaging apparatus. [4] Information processing apparatus according to any one of claims 1 to 3, further comprising a storage unit having stored therein correspondence relationship information indicating a correspondence relationship between an aperture value and blur shape related information, wherein the output unit is configured to obtain blur shape related information corresponding to the aperture value to be output based on the correspondence relationship information and output the blur shape related information. [5] Information processing apparatus according to claim 4, wherein the storage unit stores a plurality of pieces of correspondence relationship information each corresponding to a plurality of lenses, the information processing device further comprises a second acquisition unit configured to acquire identification information of the lens of the imaging device, and the output unit acquires the blur shape related information corresponding to the aperture value to be output based on the correspondence relationship information corresponding to the lens of the imaging device, and outputs the blur shape related information. [6] The information processing apparatus according to claim 4, further comprising a third acquiring unit configured to acquire the correspondence relationship information from the outside. [7] The information processing apparatus according to any one of claims 1 to 6, wherein the blur shape related information indicates the shape of a blur. [8] The information processing apparatus according to claim 7, wherein the blur shape related information further indicates a similarity of the shape of the blur compared to a perfect circle. [9] The information processing apparatus according to claim 7, wherein the blur shape related information further indicates a degree of collapse of the shape of the blur compared with a perfect circle or a regular polygon. [10] The information processing apparatus according to any one of claims 1 to 9, wherein the output unit is configured to output information related to the aperture value and the blur shape to the outside. [11] An information processing apparatus according to any one of claims 1 to 10, wherein the output unit is arranged to record information related to the aperture value and the blur shape in a recording medium provided in the information processing apparatus. [12] An information processing apparatus according to any one of claims 1 to 11, wherein the blur shape related information is applied to a graphic to be synthesized with the captured image. [13] An information processing apparatus according to any one of claims 1 to 11, wherein the blur shape related information is applied to a graphic displayed on a display device to be captured by the imaging apparatus. [14] Information processing apparatus according to claim 13, further comprising an instruction unit configured to instruct a control start of a diaphragm, and a control unit, where the control unit for controlling a timing at which the output unit outputs the blur shape-related information, and / or a timing at which the instruction unit instructs the start of driving the diaphragm, and / or a timing at which the imaging device performs a recording, based on a first time, which is a time required to control the aperture of the lens, and a second time which is a time from output of the blur shape related information by the output unit to start of display of the graphic by the display device after the blur shape related information has been applied. [15] The information processing apparatus according to claim 14, wherein the control unit is configured to control at least one of the timing at which the output unit outputs the blur shape-related information and the timing at which the instruction unit instructs the start of driving the diaphragm, such that the instruction unit instructs the start of driving the diaphragm after the output unit outputs the blur shape-related information. [16] The information processing apparatus according to any one of claims 1 to 14, wherein the information processing apparatus is the imaging apparatus whose lens is interchangeable. [17] The information processing apparatus according to any one of claims 1 to 14, wherein the information processing apparatus is the imaging apparatus whose lens is not interchangeable. [18] The information processing apparatus according to any one of claims 1 to 14, wherein the information processing device is the lens detachable from the imaging device. [19] The information processing apparatus according to any one of claims 1 to 18, wherein the output unit is configured to output metadata storing information related to the aperture value and the blur shape in a manufacturer-specific region. [20] The information processing apparatus according to any one of claims 1 to 19, wherein the output unit is configured to output the aperture value and the blur shape related information using a manufacturer-specific standard. [21] The information processing apparatus according to any one of claims 1 to 12, wherein the aperture value is a value in a log format of 16-bit resolution. [22] Information processing methods with a step of obtaining an aperture value of a lens of an imaging device and a step of outputting the aperture value acquired during capturing a captured image of the imaging device and blur shape related information related to a shape of blur occurring in the captured image. [23] The information processing method according to claim 22, wherein the aperture value is acquired in real time during capturing the captured image, and the output of the aperture value and the blur shape related information is performed in real time during capturing the captured image. [24] A computer-readable medium storing a program for causing a computer to operate as each unit of the information processing apparatus according to any one of claims 1 to 21.

Images