Image processing system, imaging system, control method, and program

Abstract

This technology provides an error amount for the virtual defocus amount of a virtual space image, taking into account the error in the amount of defocus that may occur when photographing in real space. [Solution] An image processing system is provided, comprising: generation means for generating a virtual space including virtual objects; first generation means for generating a virtual space image corresponding to a predetermined shooting range in the virtual space using information defining a focus state for virtual shooting in the virtual space; calculation means for calculating a first virtual defocus amount in the virtual space image; and second generation means for generating a second virtual defocus amount by adding an error amount based on one or more parameters that affect the error in the defocus amount that occurs in shooting in the real space to the first virtual defocus amount.

Description

Technical Field

[0006] , , , , , , ,

[0005]

[0001] The present invention relates to an image processing system, an imaging system, a control method, and a program.

Background Art

[0002] Conventionally, it is known to generate a virtual image that reproduces the real space, or a virtual image that fuses the real space and the virtual space, and display or shoot a virtual video in the virtual space.

[0003] For example, Patent Document 1 discloses adding noise caused by at least one of the characteristics specific to the imaging unit in the real space and the image processing in the real space to the video in the virtual space.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technique of Patent Document 1, as the video in the virtual space, a video with noise added to match the real space is generated. However, the error in the defocus amount caused by noise or the like in the real space is not considered. Therefore, in the technique of Patent Document 1, there is a problem that focus adjustment is performed in the virtual space using a defocus amount in which the error in the real space is not considered, and the focus adjustment in the virtual space deviates from the focus adjustment in the real space.

[0006] This invention has been made in view of the above circumstances, and aims to provide a technique for adding an error amount to the virtual defocus amount of a virtual space image, taking into account the error in the amount of defocus that may occur in the shooting of real space. [Means for solving the problem]

[0007] To solve the above problems, the present invention provides A generation means for generating a virtual space containing virtual objects, A first generation means generates a virtual space image corresponding to a predetermined shooting range in the virtual space using information defining the focus state for virtual shooting in the virtual space, A calculation means for calculating a first virtual defocus amount in the virtual space image, A second generation means generates a second virtual defocus amount by adding an error amount based on one or more parameters that affect the error in the defocus amount that occurs in real space photography to the first virtual defocus amount, The present invention provides an image processing system characterized by comprising the following features. [Effects of the Invention]

[0008] According to the present invention, it is possible to assign an error amount to the virtual defocus amount of a virtual space image, taking into account the error in the amount of defocus that may occur in the shooting of real space.

[0009] Further features and advantages of the present invention will become clearer from the accompanying drawings and the description of the embodiments for carrying out the invention below. [Brief explanation of the drawing]

[0010] [Figure 1] A diagram showing the configuration of the imaging system 10 (image processing system). [Figure 2] A diagram showing the configuration of camera 100 (imaging device). [Figure 3] A diagram illustrating the pixel arrangement of the image sensor 107. [Figure 4] Plan view and cross-sectional view of pixels of the imaging device 107. [Figure 5] Diagram for explaining the focus detection area of the imaging device 107. [Figure 6] Block diagram showing an example of the hardware configuration of the external arithmetic unit 1000. [Figure 7] Block diagram showing the functional configuration of the external arithmetic unit 1000. [Figure 8] Flowchart showing the processing flow for causing the camera 100 to perform real space imaging and virtual space imaging. [Figure 9] Flowchart showing the details of the real space imaging process (S10 in FIG. 8). [Figure 10] Flowchart of the imaging subroutine (S300 in FIG. 9). [Figure 11] Flowchart of the subroutine of the subject tracking AF process (S400 in FIG. 9). [Figure 12] Flowchart of the subroutine of the subject detection / tracking process (S402 in FIG. 11). [Figure 13] Flowchart showing the details of the virtual space imaging process (S1000 in FIG. 8). [Figure 14] Flowchart of the subroutine of the process for generating and outputting the video of the virtual space according to the first embodiment (S2000 in FIG. 13). [Figure 15] Flowchart of the subroutine of the virtual subject tracking process (S4000 in FIG. 13). [Figure 16] Flowchart of the virtual imaging subroutine (S5000 in FIG. 13). [Figure 17] Diagram for explaining the information regarding the camera / lens information storage device 2000, the camera / lens, and the external arithmetic unit 1000. [Figure 18] Diagram for explaining the virtual imaging reflecting the operation information. [Figure 19] Flowchart of the subroutine of the acquisition of the subject imaging difficulty information (S4100 in FIG. 15). [Figure 20]A flowchart of the subroutine for the virtual defocus amount processing (S4200 in Figure 15). [Figure 21] A diagram explaining framing corrections. [Figure 22] A diagram explaining the correction related to zooming. [Figure 23] A diagram explaining focusing corrections. [Figure 24] This diagram explains the defocus error information acquired by S4202. [Figure 25] This figure shows examples of virtual defocus amounts with and without an error applied to the virtual defocus amount. [Figure 26] A flowchart of the subroutine for viewpoint movement processing (S3000 in Figure 13). [Figure 27] This figure illustrates a specific example of the viewpoint movement processing subroutine described in Figure 26. [Figure 28] A flowchart explaining camera operation during virtual shooting. [Figure 29] A flowchart showing the image playback and evaluation method after shooting with camera 100. [Figure 30] This diagram illustrates the display of the captured image with a defocus map superimposed, which is performed in S1111 based on the evaluation results of S1110. [Figure 31] This section describes an example of displaying the degree of focus based on the evaluation results obtained from the processing shown in Figure 29 for a series of images acquired through continuous shooting. [Figure 32] This diagram illustrates the display of information regarding the photographer's framing technique in captured images, based on the evaluation results of the degree of focus. [Figure 33] A table showing examples of configurable items and evaluation conditions that are evaluated in S1110 regarding proposed configuration changes. [Figure 34] A flowchart of the subroutine for the process of generating and outputting video in a virtual space (S2000 in Figure 13) according to the second embodiment. [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] [First Embodiment] ● System Configuration Figure 1 shows the configuration of an imaging system 10 (image processing system) including a camera 100 (imaging device), an external computing device 1000, and a camera / lens information storage device 2000. The camera 100 has the function of capturing images of subjects existing in real space, capturing images of subjects existing in virtual space, and displaying the captured images. The camera 100 also functions as a tactile sensation reproduction device.

[0013] The external computing device 1000 is connected to the camera 100 by wired or wireless means to enable the exchange of information. The external computing device 1000 includes a virtual space reproduction device 1100 and a virtual image generation device 1200.

[0014] The virtual space reproduction device 1100 places the subject as a virtual object whose position and shape change moment by moment within the set virtual space (background space). The virtual image generation device 1200 acquires camera and lens setting information, control information, operation information of operating components, and position information including shooting direction from the camera 100. The virtual image generation device 1200 also uses the information obtained from the camera 100 to acquire related information from the camera / lens information storage device 2000.

[0015] The camera / lens information storage device 2000 may be a server on a network such as the cloud, or it may be located within the external computing device 1000.

[0016] The virtual image generation device 1200 uses the information acquired as described above to generate (capture) an image from the virtual space constructed by the virtual space reproduction device 1100. The image generated here may be a two-dimensional image or a three-dimensional image containing information that allows for stereoscopic display. In Figure 1, the reproduction of the virtual space and the generation of images are configured to be performed by the external processing unit 1000, but these functions may also be configured to be performed internally by the camera 100.

[0017] Figure 2 shows the configuration of camera 100 (imaging device). In Figure 2, the first lens group 101 is positioned furthest to the subject (front) of the imaging optical system, which is the imaging optical system, and is held so as to be movable in the optical axis direction. The aperture 102 adjusts the amount of light by adjusting its aperture diameter. The second lens group 103 moves in the optical axis direction together with the aperture 102 and performs magnification (zoom) together with the first lens group 101 which moves in the optical axis direction.

[0018] The third lens group 105 (focusing lens) moves along the optical axis to adjust the focus. The optical low-pass filter 108 is an optical element that reduces false colors and moiré in the captured image. The imaging optical system is composed of the first lens group 101, aperture 102, second lens group 103, third lens group 105, and optical low-pass filter 108.

[0019] The zoom actuator 111 rotates a cam cylinder (not shown) around the optical axis, causing a cam on the cam cylinder to move the first lens group 101 and the second lens group 103 in the optical axis direction, thereby changing the magnification. The aperture actuator 112 drives a plurality of light-shielding vanes (not shown) in the opening and closing direction to adjust the light intensity of the aperture 102. The focus actuator 114 moves the third lens group 105 in the optical axis direction to adjust the focus.

[0020] The focus drive circuit 126 drives the focus actuator 114 in response to a focus drive command from the camera CPU 121, moving the third lens group 105 in the optical axis direction. The aperture drive circuit 128 drives the aperture actuator 112 in response to an aperture drive command from the camera CPU 121. The zoom drive circuit 129 drives the zoom actuator 111 in response to the user's zoom operation.

[0021] In this embodiment, the interchangeable lens, which includes the imaging optical system, actuators 111, 112, 114, and drive circuits 126, 128, 129, is configured to be detachably attached to the camera body using a mount portion M that enables electrical and mechanical connections. However, the imaging optical system, actuators 111, 112, 114, and drive circuits 126, 128, 129 may also be provided integrally with the camera body, which includes the image sensor 107.

[0022] The electronic flash 115 has a light-emitting element such as a xenon tube or LED, and emits light to illuminate the subject. The AF assist light-emitting unit 116 has a light-emitting element such as an LED, and projects an image of a mask having a predetermined aperture pattern onto the subject via a projection lens, thereby improving focus detection performance for dark or low-contrast subjects. The electronic flash control circuit 122 controls the electronic flash 115 to light up in synchronization with the imaging operation. The assist light control circuit 123 controls the AF assist light-emitting unit 116 to light up in synchronization with the focus detection operation.

[0023] The camera CPU 121 controls various functions in the camera 100. The camera CPU 121 includes an arithmetic unit, ROM, RAM, A / D converter, D / A converter, and communication interface circuitry. The camera CPU 121 drives various circuits within the camera 100 according to computer programs stored in the ROM, and controls a series of operations such as autofocus, imaging, image processing, and recording. The camera CPU 121 also functions as an image processing device.

[0024] The image sensor 107 consists of a two-dimensional CMOS photosensor containing multiple pixels and its peripheral circuits, and is positioned on the imaging plane of the imaging optical system. The image sensor 107 converts the subject image formed by the imaging optical system into a digital signal. The image sensor drive circuit 124 controls the operation of the image sensor 107 and performs A / D conversion on the analog signal generated by the photoelectric conversion to transmit the digital signal to the camera CPU 121.

[0025] The shutter 106 has a focal-plane shutter configuration and drives the focal-plane shutter based on instructions from the camera CPU 121 and commands from the shutter drive circuit built into the shutter 106. When the signal from the image sensor 107 is read out, the focal-plane shutter blocks light from the image sensor 107. Also, when exposure is taking place, the focal-plane shutter opens and the photographic light beam is directed to the image sensor 107.

[0026] The image processing unit 125 applies predetermined image processing to the image data stored in the RAM of the camera CPU 121. The image processing applied by the image processing unit 125 includes, but is not limited to, so-called development processing such as white balance adjustment, color interpolation (demosaic) processing, and gamma correction processing, as well as signal format conversion processing and scaling processing. Furthermore, the image processing unit 125 determines the main subject based on the posture information of the subject and the position information of objects specific to the scene (hereinafter referred to as specific objects). The image processing unit 125 may use the result of the determination processing for other image processing (for example, white balance adjustment processing). The image processing unit 125 stores the processed image data, the joint positions of each subject, the position and size information of specific objects, the center of gravity of the subject determined to be the main subject, and the position information of the face and pupils in the RAM of the camera CPU 121.

[0027] The display unit 131 is equipped with a display element such as an LCD and displays information related to the camera 100's imaging mode, a preview image before imaging, a confirmation image after imaging, an indicator of the focus detection area, and a focused image. The operation switch group 132 includes a main (power) switch, a release (shooting trigger) switch, a zoom operation switch, a shooting mode selection switch, etc., and is operated by the user. The flash memory 133 records the captured images. The flash memory 133 is detachable from the camera 100.

[0028] The subject detection unit 140 performs subject detection based on dictionary data generated by machine learning. In this embodiment, the subject detection unit 140 uses dictionary data for each subject in order to detect multiple types of subjects. Each dictionary data is, for example, data in which the characteristics of the corresponding subject are registered. The subject detection unit 140 performs subject detection by sequentially switching between the dictionary data for each subject. The dictionary data for each subject is stored in the dictionary data storage unit (ROM in the camera CPU 121). Therefore, multiple dictionary data are stored in the dictionary data storage unit. The camera CPU 121 determines which dictionary data to use for subject detection from among the multiple dictionary data based on the pre-set subject priority and imaging device settings.

[0029] The video input unit 141 receives the generated image when shooting (image generation) is performed in the virtual space. The camera CPU 121 processes the input image by displaying it on the display unit 131 or storing it in the flash memory 133. The information output unit 142 outputs various information to the external processing unit 1000 when shooting is performed in the virtual space. The camera operation information output includes release operations for issuing shooting commands, lens zooming, and focus operations. The camera setting information output includes setting information related to the mode for continuous shooting, autofocus, metering, exposure condition settings, image generation, and lens control. The camera control information output includes information related to correction values ​​and thresholds used in various algorithms used for shooting and image generation. The information output unit 142 also outputs information indicating the position and shooting direction of the camera 100. Details will be described later.

[0030] Examples of dictionary data for subject detection include dictionary data for detecting "people," dictionary data for detecting "animals," and dictionary data for detecting "vehicles." Furthermore, dictionary data for detecting "the whole person" and dictionary data for detecting "the person's face" may be stored separately in the dictionary data storage unit.

[0031] In this embodiment, the subject detection unit 140 is composed of a machine-learned Convolutional Neural Network (CNN) and estimates the position of subjects included in the image data. The subject detection unit 140 may be implemented using a graphics processing unit (GPU) or a circuit specialized for estimation processing by CNN.

[0032] Machine learning of a CNN can be performed using any method. For example, a designated computer, such as a server, may perform machine learning of the CNN, and the camera 100 may acquire the trained CNN from the designated computer. For example, the designated computer may perform supervised learning using training image data as input and the position of subjects corresponding to the training image data as training data, thereby training the CNN of the subject detection unit 140. As a result, a trained CNN is generated. The CNN training may be performed by the camera 100 or the image processing device described above.

[0033] Next, the pixel arrangement of the image sensor 107 will be explained using Figure 3. Figure 3 shows the pixel arrangement of the image sensor 107 in a range of 4 pixel columns × 4 pixel rows, viewed from the optical axis direction (z direction).

[0034] Each pixel unit 200 contains four imaging pixels arranged in a 2x2 grid. By arranging a large number of pixel units 200 on the image sensor 107, photoelectric conversion of a two-dimensional subject image can be performed. Within each pixel unit 200, an imaging pixel 200R with spectral sensitivity for red (R) is located in the upper left, and imaging pixels 200G with spectral sensitivity for green (G) are located in the upper right and lower left. Furthermore, an imaging pixel 200B with spectral sensitivity for blue (B) is located in the lower right. Each imaging pixel also contains a first focus detection pixel 201 and a second focus detection pixel 202, which are divided in the horizontal direction (x direction).

[0035] In the image sensor 107 of this embodiment, the pixel pitch P of the imaging pixels is 4 μm, and the number of imaging pixels N is approximately 20.75 million pixels (5575 horizontal columns × 3725 vertical rows). The pixel pitch PAF of the focus detection pixels is 2 μm, and the number of focus detection pixels NAF is approximately 41.5 million pixels (11150 horizontal columns × 3725 vertical rows).

[0036] In this embodiment, the case in which each imaging pixel is divided into two horizontally is described, but it may also be divided vertically. Furthermore, although the image sensor 107 in this embodiment has a plurality of imaging pixels, each including a first and a second focus detection pixel, the imaging pixels and the first and second focus detection pixels may be provided as separate pixels. For example, the first and second focus detection pixels may be discretely arranged within a plurality of imaging pixels.

[0037] Figure 4(a) shows one imaging pixel (200R, 200G, 200B) as viewed from the light-receiving side (+z direction) of the image sensor 107. Figure 4(b) shows the cross-section aa of the imaging pixel in Figure 4(a) as viewed from the -y direction. As shown in Figure 4(b), one imaging pixel is provided with one microlens 305 for focusing incident light.

[0038] Furthermore, the imaging pixel is provided with photoelectric conversion units 301 and 302, which are divided into N sections (2 sections in this embodiment) in the x direction. The photoelectric conversion units 301 and 302 correspond to the first focus detection pixel 201 and the second focus detection pixel 202, respectively. The centers of gravity of the photoelectric conversion units 301 and 302 are eccentric to the -x side and the +x side, respectively, with respect to the optical axis of the microlens 305.

[0039] A color filter 306 of R, G, or B is provided between the microlens 305 and the photoelectric conversion units 301 and 302 in each imaging pixel. The spectral transmittance of the color filter may be changed for each photoelectric conversion unit, or the color filter may be omitted.

[0040] Light incident on the imaging pixel from the imaging optical system is focused by the microlens 305, spectrally separated by the color filter 306, and then received by the photoelectric conversion units 301 and 302, where it is photoelectrically converted. The camera 100 having the image sensor 107 shown in Figures 3 and 4 can perform so-called phase-difference focus detection, which detects the phase difference from a pair of signal sequences obtained by dividing the light beam passing through the imaging optical system, using known techniques (for example, the technique described in Patent Document 2). Phase-difference focus detection makes it possible to detect the amount of defocus in a predetermined area within the imaging range, including its direction.

[0041] Next, using Figure 5, we will explain the focus detection region of the image sensor 107, which is the region from which a pair of signal sequences for detecting phase difference is acquired. In Figure 5, A(n,m) represents the nth focus detection region in the x-direction and the mth focus detection region in the y-direction, out of a total of nine focus detection regions (three each in the x-direction and y-direction) set in the effective pixel region 300 of the image sensor 107. A pair of signal sequences is generated from multiple pixels contained in the focus detection region A(n,m). I(n,m) represents an index that displays the position of the focus detection region A(n,m) on the display unit 131. Note that the nine focus detection regions shown in Figure 5 are merely examples, and the number, position, and size of the focus detection regions are not limited. For example, one or more regions may be set as focus detection regions within a predetermined range centered on a position specified by the user or the position of the subject detected by the subject detector. In this embodiment, the focus detection regions are arranged to obtain a higher resolution focus detection result when acquiring the defocus map described later. For example, a total of 9600 focus detection regions are arranged on the image sensor with 120 horizontal divisions and 80 vertical divisions.

[0042] Figure 6 is a block diagram showing an example of the hardware configuration of the external computing device 1000. The external computing device 1000 includes a CPU 1001, a ROM 1002, a RAM 1003, a storage unit 1004, an input interface 1005, an output interface 1006, and a system bus 1007. The input interface 1005 is connected to the camera 100 and the camera / lens information storage device 2000. The output interface 1006 is connected to the camera 100.

[0043] The CPU 1001 is a processor that comprehensively controls each component of the external arithmetic unit 1000. The ROM 1002 is memory that stores programs and other data used for processing within the external arithmetic unit 1000. The RAM 1003 is memory that functions as the main memory and work area of ​​the CPU 1001. The CPU 1001 uses the RAM 1003 as a work area and executes programs stored in the ROM 1002 to perform various processes described later.

[0044] The storage unit 1004 is a storage device that stores image data used for processing by the external computing device 1000, as well as parameters (i.e., setting values) for said processing. The storage unit 1004 can be an HDD, optical disc drive, flash memory, or the like.

[0045] The input interface 1005 is, for example, a serial bus interface such as USB or IEEE1394. The external computing device 1000 can acquire the various types of information described above from the camera 100 via the input interface 1005. The output interface 1006 is, for example, a video output terminal such as DVI or HDMI (registered trademark). The external computing device 1000 can output image data processed by the external computing device 1000 to the display 131 of the camera 100 via the output interface 1006. The external computing device 1000 can also output images for recording to the flash memory 133 of the camera 100 via the output interface 1006. Note that the external computing device 1000 may include components other than those described above.

[0046] Next, the virtual image generation process performed by the external computing device 1000 in Figure 6 will be explained using Figure 7. Figure 7 is a block diagram showing the functional configuration of the external computing device 1000. In this embodiment, the CPU 1001 functions as each block shown in Figure 7 by executing the program stored in the ROM 1002. However, the CPU 1001 does not need to perform all functions, and processing circuits that perform each function may be provided in each part of the external computing device 1000.

[0047] First, let's describe the virtual space reproduction device 1100. The foreground object acquisition unit 1102 acquires 3D objects of foreground subjects, such as performers on stage, stored in the foreground object storage unit 1101. A 3D object is 3D shape data that describes information indicating shape and color, and is composed of textured mesh models or 3D point clouds with colored points. Note that 3D objects do not necessarily have color. Various 3D objects can be stored in the foreground object storage unit 1101, such as people of different races, genders, and ages, various animals, and moving objects such as cars. The foreground object acquisition unit 1102 is not limited to acquiring one foreground object, but may acquire multiple objects. The 3D object also has subject information such as velocity, acceleration, angular velocity, angular acceleration, size, and contrast. In addition, the 3D object may be generated from captured images of subjects that the photographer wants to photograph in virtual space photography using a pre-trained model that estimates a 3D model for images, and size and contrast information may also be stored. Furthermore, by using multiple time-series images, information on the velocity, acceleration, angular velocity, and angular acceleration of the 3D object can also be generated and stored.

[0048] Alternatively, a 3D object may be generated and stored from images captured using multiple imaging devices with different viewpoints. Multiple imaging devices capture the imaging area from multiple directions. This imaging area could be, for example, an indoor imaging studio or a stage where a play is performed. The multiple imaging devices are installed at different positions surrounding this imaging area and perform imaging synchronously. Note that the multiple imaging devices do not need to be installed around the entire circumference of the imaging area; depending on the limitations of the installation location, they may be installed only in a part of the imaging area. The number of imaging devices can be set in various ways; for example, if the imaging area is a soccer field, about 30 imaging devices may be installed around the field. Also, imaging devices with different functions, such as telephoto cameras and wide-angle cameras, may be installed.

[0049] For each of the multiple imaging devices, a parameter set may be described for each imaging device, including parameters representing the three-dimensional position, parameters representing the orientation of the imaging device in the pan, tilt, and roll directions, and the size of the imaging device's field of view (angle of view) and resolution. The information included in the parameter set is calculated in advance using a known camera calibration procedure and stored in a suitable storage device (e.g., foreground object storage unit 1101). That is, points in multiple images based on imaging by the multiple imaging devices are associated and calculated by geometric calculation. Note that the content of the information included in the parameter set is not limited to the above. For example, there may be multiple parameter sets corresponding to multiple frames that constitute a video of the imaging device, and the information may indicate the position and orientation of the imaging device at each of multiple consecutive time points.

[0050] The foreground object acquisition unit 1102 generates a 3D object of a person, such as a stage performer, which is a foreground subject, based on images from multiple viewpoints and a parameter set received from the imaging device, for example, according to the method described in Patent Document 3.

[0051] Similarly, the background object acquisition unit 1105 acquires a 3D object as a space for arranging foreground objects such as a stage or stadium, which are stored in the background object storage unit 1104. The objects stored in the background object storage unit 1104 can be 3D objects of various spaces, such as a large concert hall or soccer stadium, or a small indoor room. The background object may be created using design data such as CAD, or shape and color data scanned with a laser scanner, or it may be generated from a set of images from multiple viewpoints using computer vision techniques such as Structure from Motion.

[0052] The object compositing unit 1103 places the foreground object within the space of the acquired background object. Therefore, the virtual space includes a virtual object that contains both the foreground object and the background object. The information about the foreground object acquired by the object compositing unit 1103 may include a 3D model of the subject at multiple times, corresponding to the shape and color of the subject at multiple times. During placement, the foreground object is positioned so that it does not float relative to the ground included in the background object, except for interference between objects or actions such as jumping. The foreground object may be placed according to the object placement information (position, orientation) of the background object, or it may be placed based on instructions from an external source such as a user.

[0053] Next, the virtual image generation device 1200 will be described. The viewpoint information acquisition unit 1201 acquires virtual viewpoint parameters, including the position and direction (pan, tilt, roll) of the virtual viewpoint in the virtual space. The virtual viewpoint parameters may be set as initial values, registered values, previous history positions, etc., in the virtual space, or they may be set by the user.

[0054] The camera lens information acquisition unit 1202 acquires information about the camera and lens to be used for virtual shooting from the camera / lens information storage device 2000 or the camera 100. Details of the information will be described later. The camera lens information update unit 1203 acquires and updates the camera lens information, which is updated over time, as needed.

[0055] The operation information acquisition unit 1205 acquires operation information for the camera and lens from the camera 100. Details of the information will be described later.

[0056] Viewpoint information, camera lens information, and operation information are input to the image correction amount calculation unit 1206. The image correction amount calculation unit 1206 calculates the image correction amount. The image correction amount calculation unit 1206 calculates the image correction amount using information obtained from the subject shooting difficulty calculation unit 1261 and the photographer's intention extraction unit 1262. Details of the process will be described later.

[0057] The display image generation unit 1204 uses the foreground and background object information, virtual viewpoint information, and camera lens information obtained from the object composition unit 1103 to render and generate a virtual image. The generated virtual image is output to the camera 100 and displayed on the camera 100's display unit 131. The generated virtual image is also recorded in the camera 100's flash memory 133 and the external processing unit 1000's storage unit 1004.

[0058] ● Shooting and processing Figure 8 is a flowchart showing the processing flow for causing the camera 100 of this embodiment to perform real-space and virtual-space imaging. Specifically, Figure 8 shows the process for causing the camera 100 to perform operations from before image capture to displaying the image on the display unit 131 to capture a still image. The camera CPU 121, which is a computer, executes this process according to the computer program. In the following description, S means step.

[0059] In S1, the camera CPU 121 starts displaying a menu for settings on the display unit 131, as well as live view images of the real or virtual space. The generation of the live view images to be played back will be described later. Upon initial startup or user operation, the display unit 131 displays a menu setting screen in which the user can select whether to shoot in the real or virtual space. The displayed content may be determined based on the history of the previous startup. If shooting in the real or virtual space has already been set, the previously started live view display will continue.

[0060] In S2, the camera CPU 121 determines whether or not to perform virtual shooting based on user instructions or previous history. If the answer in S2 is Yes, the process proceeds to S1000, and the camera CPU 121 performs virtual space shooting. On the other hand, if the answer in S2 is No, the process proceeds to S10, and the camera CPU 121 performs real space shooting. After completing the processes in S10 or S1000, the process proceeds to S3, where the camera CPU 121 determines whether or not the main switch included in the operation switch group 132 has been turned off. If the main switch has been turned off, the camera CPU 121 terminates the process in this flowchart; otherwise, the process returns to S1.

[0061] ● Real-world space image processing Figure 9 is a flowchart detailing the real-world space capture process (S10 in Figure 8). The camera CPU 121, which is a computer, executes this process according to the computer program. In the following description, S stands for step.

[0062] In S11, the camera CPU 121 drives the image sensor 107 using the image sensor drive circuit 124 and acquires imaging data from the image sensor 107. Subsequently, the camera CPU 121 acquires a pair of focus detection signals from the pair of focus detection pixels included in each of the focus detection regions shown in Figure 5 from the acquired imaging data. The camera CPU 121 also generates an imaging signal by adding the pair of focus detection signals of all effective pixels of the image sensor 107, and has the image processing unit 125 perform image processing on the imaging signal (imaging data) to acquire image data. If imaging pixels and focus detection pixels are provided separately, the camera CPU 121 performs interpolation processing on the focus detection pixels to acquire image data.

[0063] In S12, the camera CPU 121 instructs the image processing unit 125 to generate a live view image from the image data obtained in S11, and displays this on the display unit 131. The live view image is a scaled-down image matched to the resolution of the display unit 131, allowing the user to adjust the imaging composition, exposure conditions, etc., while viewing it. Therefore, the camera CPU 121 adjusts the exposure based on the metering values ​​obtained from the image data and displays it on the display unit 131. Exposure adjustment is achieved by appropriately adjusting the exposure time, opening and closing the aperture of the photographic lens, and adjusting the gain for the output of the image sensor 107.

[0064] In S13, the camera CPU 121 determines whether the switch Sw1, which instructs the start of the image preparation operation, has been turned on by half-pressing the release switch included in the operation switch group 132. If Sw1 is not turned on, the camera CPU 121 repeats the determination in S13 to monitor the timing when Sw1 will be turned on. On the other hand, if Sw1 is turned on, the camera CPU 121 proceeds to S400 and performs subject-tracking autofocus (AF) processing. Here, it performs processing to detect the subject area from the obtained imaging signal and focus detection signal (subject detection processing), processing to set the focus detection area, and predictive AF processing to suppress the effect of the time lag between the focus detection processing and the image capture processing. Details will be described later.

[0065] In S15, the camera CPU 121 determines whether the switch Sw2, which instructs the start of the imaging operation, has been turned on by fully pressing the release switch. If Sw2 is not turned on, the camera CPU 121 returns to S13. On the other hand, if Sw2 is turned on, the camera CPU 121 proceeds to S300 and executes the imaging subroutine. Details of the imaging subroutine will be described later. Once the imaging subroutine finishes, the processing of this flowchart ends.

[0066] In this embodiment, the system is configured to perform subject detection processing and AF processing after Sw1 is detected as being turned on in S13, but the timing of these processes is not limited to this. By performing the subject tracking AF processing in S400 before Sw1 is turned on, it is possible to eliminate the need for the photographer to perform preparatory actions before shooting.

[0067] Next, we will explain the details of the imaging subroutine (S300 in Figure 9) executed by the camera CPU 121 using the flowchart shown in Figure 10.

[0068] In step S301, the camera CPU 121 performs exposure control processing and determines the imaging conditions (shutter speed, aperture value, image sensitivity, etc.). This exposure control processing can be performed using brightness information obtained from the image data of the live view image. The camera CPU 121 then transmits the determined aperture value to the aperture drive circuit 128 to drive the aperture 102. The camera CPU 121 also transmits the determined shutter speed to the shutter 106 to open the focal plane shutter. Furthermore, the camera CPU 121 causes the image sensor 107 to accumulate charge during the exposure period via the image sensor drive circuit 124.

[0069] In S302, the camera CPU 121 instructs the image sensor drive circuit 124 to read out all pixels of the imaging signal from the image sensor 107 for still image capture. The camera CPU 121 also instructs the image sensor drive circuit 124 to read out one of the pair of focus detection signals from the focus detection area (focus target area) within the image sensor 107. The focus detection signal read out at this time is used to detect the focus state of the image during image playback, which will be described later. The other focus detection signal can be obtained by subtracting one of the pair of focus detection signals from the imaging signal.

[0070] In S303, the camera CPU 121 instructs the image processing unit 125 to perform defective pixel correction processing on the imaging data read in S302 and converted by A / D.

[0071] In S304, the camera CPU 121 instructs the image processing unit 125 to perform image processing (demosaic (color interpolation) processing, white balance processing, gamma correction (tone correction) processing, color conversion processing, edge enhancement processing, etc.) and encoding processing on the captured data after the defective pixel correction processing.

[0072] In S305, the camera CPU 121 records the still image data obtained as image data through the image processing and encoding process in S304, and the focus detection signal read out in S302, as an image data file in the flash memory 133.

[0073] In S306, the camera CPU 121 records camera characteristic information, which is characteristic information of the camera 100, in the flash memory 133 and the memory within the camera CPU 121, corresponding to the still image data recorded in S305. The camera characteristic information includes, for example, the following information. • Imaging conditions (aperture value, shutter speed, ISO sensitivity, etc.) • Information regarding image processing performed by the image processing unit 125 • Information regarding the light-receiving sensitivity distribution of the imaging pixels and focus detection pixels of the image sensor 107. • Information regarding vignetting of the imaging light beam within Camera 100 • Information on the distance from the mounting surface of the imaging optical system in camera 100 to the image sensor 107. • Information regarding manufacturing tolerances for Camera 100

[0074] Information regarding the light sensitivity distribution of the imaging pixels and focus detection pixels of the image sensor 107 (hereinafter simply referred to as light sensitivity distribution information) is information regarding the sensitivity of the image sensor 107 according to the distance (position) on the optical axis from the image sensor 107. Since this light sensitivity distribution information depends on the microlens 305 and the photoelectric conversion units 301 and 302, it may also be information regarding these. Furthermore, the light sensitivity distribution information may also be information regarding the change in sensitivity with respect to the angle of incidence of light.

[0075] In S307, the camera CPU 121 records lens characteristic information as characteristic information of the imaging optical system in the flash memory 133 and the memory within the camera CPU 121, corresponding to the still image data recorded in S305. The lens characteristic information includes, for example, information about the exit pupil, information about the frame such as the lens barrel that emits the light beam, information about the focal length and F number at the time of imaging, information about aberrations of the imaging optical system, information about manufacturing errors of the imaging optical system, and information about the position (subject distance) of the focusing lens (third lens group 105) at the time of imaging.

[0076] In S308, the camera CPU 121 records image-related information, which is information related to still image data, in the flash memory 133 and the memory within the camera CPU 121. Image-related information includes, for example, information related to the focus detection operation before image capture, information related to the movement of the subject, and information related to the focus detection accuracy.

[0077] In S309, the camera CPU 121 displays a preview of the captured image on the display unit 131. This allows the user to easily check the captured image.

[0078] Once processing S309 is complete, the camera CPU 121 terminates the imaging subroutine.

[0079] Next, using Figure 11, we will explain the subroutine for subject tracking AF processing (S400 in Figure 9) executed by the camera CPU 121.

[0080] In S401, the camera CPU 121 calculates the amount of image shift between pairs of focus detection signals obtained in each of the multiple focus detection regions acquired in S12, and calculates the amount of defocus for each focus detection region from the amount of image shift. In this embodiment, the group of focus detection results obtained from a total of 9600 points arranged on the image sensor 107 in 120 horizontal divisions and 80 vertical divisions is called a defocus map.

[0081] In S402, the camera CPU 121 performs subject detection processing and tracking processing. Subject detection processing is performed by the subject detection unit 140 described above. Depending on the state of the obtained image, subject detection may not be possible. In such cases, the subject detection unit 140 performs tracking processing using other methods such as template matching to estimate the position of the subject. Details will be described later.

[0082] In S403, the camera CPU 121 sets the focus detection area using the subject detection area information obtained in S402. The camera CPU 121 acquires information such as the subject's position, size, and reliability as the subject detection area information obtained as the output of the subject detection and tracking process performed in S402. For setting the focus detection area, it is sufficient to select a focus detection result that indicates a subject that is highly reliable and relatively close in distance from the results of the focus detection area within the area set as the subject detection area. Alternatively, for setting the focus detection area, a focus detection area may be placed again within the area set as the subject detection area, image data and focus detection signals may be acquired again, and the focus detection result may be selected in the same way.

[0083] In S404, the camera CPU 121 acquires the focus detection result for the set focus detection area. The focus detection result acquired here may be selected from the focus detection results calculated in S401 to be close to the desired area, or the amount of defocus may be calculated using a newly set focus detection signal corresponding to the set focus detection area. Furthermore, the focus detection area for which the amount of defocus is calculated is not limited to one, but may be arranged in multiple locations around it.

[0084] In S405, the camera CPU 121 performs predictive AF processing using the defocus amount obtained in S404 and multiple defocus amounts, which are time-series data of past focus detection timings. This processing is necessary when there is a time lag between the timing of focus detection and the timing of image exposure. It predicts the position of the subject in the optical axis direction at the time of image exposure, which is a predetermined time after the timing of focus detection, and performs AF control.

[0085] The position of the subject on the image plane is predicted by performing multivariate analysis (e.g., least squares method) using historical data of the subject's image plane position and time, and obtaining the equation of the prediction curve. By substituting the timing of exposure of the captured image into the obtained prediction curve equation, the predicted position of the subject on the image plane can be calculated. Furthermore, the position may be predicted not only in the optical axis direction but also in three dimensions. If the screen is considered as XY and the optical axis direction as Z, and the vectors are in the XYZ directions, the position of the subject at the time of exposure of the captured image may be predicted from the time-series data of the subject's XY position obtained in the subject detection and tracking process in S402 and the Z-direction position obtained from the defocus amount obtained in S405. Furthermore, the position of the subject may be predicted from the time-series data of the joint positions of the person who is the subject. With the above prediction, even if the ball or person is hidden or part of the person's joint positions become invisible during the process, the positions of each can be estimated.

[0086] Prediction is performed not only on the main subject but also on multiple detected subjects. By performing predictive AF processing on multiple subjects, when the main subject changes, there is no need to re-store the history of the defocus amount of the new main subject. Therefore, predictive AF can be continued without any time loss. In S405, the predictive AF processing result is used to calculate the amount of drive of the focus lens, and the focus actuator 114 is driven in accordance with the focus drive command from the camera CPU 121, and the third lens group 105 is moved in the optical axis direction to perform focus adjustment processing.

[0087] Once the processing in S406 is complete, the camera CPU 121 terminates the subject tracking AF processing subroutine and proceeds to S15 in Figure 9.

[0088] Next, using Figure 12, we will explain the subroutine for subject detection and tracking (S402 in Figure 11) executed by the camera CPU 121.

[0089] In S421, the camera CPU 121 sets dictionary data according to the type of subject to be detected, based on the data detected from the image data acquired in S12. Based on the pre-set subject priority and the settings of the camera 100, the camera CPU 121 selects the dictionary data to be used in this process from multiple dictionary data stored in the dictionary data storage unit. For example, multiple dictionary data may be stored, classifying subjects into categories such as "people," "vehicles," and "animals." In this embodiment, one or more dictionary data may be selected. If one dictionary data is selected, it becomes possible to repeatedly detect subjects that can be detected by that one dictionary data at a high frequency. On the other hand, if multiple dictionary data are selected, subjects can be detected sequentially by setting the dictionary data sequentially according to the priority of the subjects to be detected.

[0090] In S422, the subject detection unit 140 uses the image data read in S12 as the input image and performs subject detection using the dictionary data set in S421. At this time, the subject detection unit 140 outputs information such as the position, size, and confidence level of the detected subject. At this time, the camera CPU 121 may display the above information output by the subject detection unit 140 on the display unit 131. In S422, the subject detection unit 140 detects multiple regions of the subject hierarchically from the image data. For example, if "person" or "animal" is set as the dictionary data, the subject detection unit 140 detects multiple organs such as the "whole body" region, the "face" region, and the "eyes" region. Local areas such as the eyes and face of a person are areas where the focus and exposure state should be adjusted as a subject, but they may not be detectable due to surrounding obstacles or the orientation of the face. Even in such cases, the subject is robustly detected by performing whole-body detection, so the system is configured to detect subjects hierarchically. Similarly, if "vehicle" such as a motorcycle is set as dictionary data, the system is configured to detect the driver, the entire vehicle including the vehicle body, and the helmet (head) as a local area in a hierarchical manner.

[0091] In S423, the camera CPU 121 performs known template matching processing using the subject detection area obtained in S422 as a template. The camera CPU 121 uses multiple images obtained in S12 to search for similar areas in the most recently obtained image, using the subject detection area obtained in past images as a template. Any of the information used for template matching may be used, such as brightness information, color histogram information, or feature point information such as corners and edges. Various matching methods and template update methods can be considered, but any known method may be used. The tracking process performed in S423 is performed to achieve stable subject detection and tracking processing by detecting areas similar to past subject detection data from the most recently obtained image data when no subject was detected in S422.

[0092] Once the processing in S423 is complete, the camera CPU 121 terminates the subject detection and tracking subroutine and proceeds to S403 in Figure 11.

[0093] ●Virtual space imaging processing Figure 13 is a flowchart detailing the virtual space shooting process (S1000 in Figure 8). The virtual space shooting process is the process of extracting information from a virtual space that changes over time and generating an image for a specific moment. In virtual shooting in a virtual space, a physical shooting optical system or image sensor is not required, but for the sake of simplicity, the same terminology as shooting in real space will be used. For example, image generation in a virtual space is expressed as shooting or imaging. More specifically, Figure 13 shows the process of causing the camera 100 to perform operations from before imaging to still image capture, where the image in the virtual space is displayed as a live view image on the display unit 131. The camera CPU 121 and CPU 1001, which are computers, execute this process according to the computer program.

[0094] In S1001, settings related to virtual shooting, such as the virtual shooting space where the shooting will take place and the equipment to be used, are configured. In the virtual shooting space configuration, as explained in Figure 7, foreground objects are placed in appropriate positions relative to background objects. Information regarding the position and shape of foreground objects that change over time is also acquired. There may be one or more foreground objects to be placed.

[0095] Furthermore, in the virtual shooting settings, the camera CPU 121 outputs the model information of the equipment being operated to the external computing device 1000 as settings for the camera and lens to be used in the virtual shooting. If a different model of equipment is used for the virtual shooting than the one actually being operated, the operator sets a unique symbol for that model. The camera CPU 121 outputs the set information to the external computing device 1000. This allows the operator (photographer) to experience shooting with cameras and lenses they do not actually own. For example, in the real world, they can operate a camera equipped with a short focal length, a so-called wide-angle lens, while in the virtual world, they can experience shooting with a long focal length telephoto lens. Operating equipment different from the actual equipment is not limited to lenses; it can also be a camera, or both. This enables a more flexible shooting experience that is not limited by the weight or size of the equipment. Similarly, by using a camera that the photographer does not actually own, they can experience the new functions of that camera, the improved performance due to newly implemented algorithms, and so on.

[0096] Furthermore, the virtual shooting settings allow you to set initial values ​​for the viewpoint position and direction (the camera's position and direction in the virtual space) when virtual shooting begins. You should set an appropriate distance from the foreground object type and other information mentioned above as the initial value. Alternatively, you can set a pre-configured shooting position within the background object as the initial value.

[0097] Next, in S1002, the CPU 1001 issues an initialization command to the camera 100 for the camera drive unit. Details will be described later.

[0098] In step S1003, the CPU 1001 begins acquiring camera information, lens information, and camera and lens operation information from the camera 100 and the camera / lens information storage device 2000.

[0099] Here, referring to the table in Figure 17, we will describe the information regarding the camera / lens information storage device 2000, the camera / lens, and the external processing unit 1000. The camera / lens information storage device 2000 stores camera information and lens information. The camera / lens information storage device 2000 also acquires and stores information from the camera / lens.

[0100] Camera information includes the display resolution, recorded video resolution, and image sensor size, which are previously acquired from camera 100. It also includes camera settings (distance frame mode, autofocus (AF) mode (one-shot or servo, etc.), continuous shooting settings, and shooting difficulty settings set by the photographer). Furthermore, it includes the AF algorithm, camera algorithm information (automatic exposure (AE) and continuous shooting drive sequence, etc.), and camera detection information (temperature, etc.). It also includes image sensor characteristic information (S / N information for each ISO sensitivity, etc.). Finally, it includes focus-related correction information (image sensor signal characteristic correction, shading correction value representing light intensity unevenness, defocus conversion coefficient that converts image shift amount into defocus amount, information on best focus position correction that corrects the discrepancy between focus detection result and best image plane position, and defocus error information, etc.). Finally, it includes general information (camera / lens model name, and firmware version of various algorithms, etc.).

[0101] The lens information includes the range, current value, and resolution of the focal length, as well as the range, increment, and current value of the F-number. It also includes focus information (such as the drive range and current position of the focus lens), focus control information regarding the control characteristics of the focus drive, and sensitivity for converting the focus lens drive into image plane movement. Furthermore, the lens information includes image stabilization information regarding the range, current value, and correction resolution of the image stabilization, as well as image stabilization control information regarding the control characteristics of the image stabilization. Finally, the lens information includes aperture control information regarding the control characteristics of the aperture drive, lens frame information (position, diameter) regarding vignetting, peripheral light falloff information, distance information regarding the focus lens position and distance, and information regarding the point image distribution function.

[0102] Cameras and lenses contain operational information generated by the photographer's actions in operating the camera / lens body.

[0103] The operation information includes information related to framing, zooming, focusing, shutter release, and other button operations. The camera CPU 121 transmits this operation information to the external processing unit 1000 and incorporates it into the virtual image generation.

[0104] The external processing unit 1000 acquires camera information, lens information, and operation information, and generates display images, recorded images, subject information which is shooting difficulty information, virtual defocus amount, and various shooting-related information.

[0105] The lens information acquired includes, for example, focal length, f-number, information on the configurable range and current position of the focus lens, the mechanical controllability of the lens, the amount of image plane movement (sensitivity) associated with the movement of the focus lens, lens frame information (position, diameter) related to vignetting, peripheral light falloff information, and shooting distance (distance to the subject in focus) information.

[0106] The camera information acquired also includes, for example, general information (model name, firmware version), resolution of EVF images and still images, size of the image sensor, setting of the AF frame to set the AF range, and camera setting information (setting of AF mode (one-shot or servo AF, etc.), and setting of continuous shooting mode such as continuous shooting speed, etc.). The camera setting information also includes difficulty level information related to shooting set by the photographer (shooting difficulty setting). Furthermore, as correction values ​​for the signal used for autofocus focus detection, the camera includes correction values ​​for signal characteristics that depend on the characteristics of the image sensor 107, shading correction values ​​that represent unevenness in light intensity, a defocus conversion coefficient that converts the phase difference of a pair of signals into a defocus amount, and a best focus correction value that corrects the discrepancy between the focus detection result and the best image plane position. In addition, the camera information includes characteristic information of the image sensor 107, such as signal S / N information for each ISO sensitivity, various algorithm information such as continuous shooting sequence and metering when shooting with the camera, and autofocus-related algorithm information such as AF frame selection and predictive AF. Some of this information changes with camera operation; therefore, information that may change is acquired periodically from S1003 onwards.

[0107] Furthermore, camera and lens operation information includes, for example, information regarding the amount and speed of panning, zooming, and focusing operations performed by the operator, as well as information regarding button press operations such as shutter release, which are used to instruct shooting.

[0108] Returning to Figure 13, in S2000, CPU 1001 generates video of the virtual space based on the settings made so far and outputs it to camera 100. Details of this process will be described later.

[0109] In S1005, the camera CPU 121 acquires the image output in S2000 and displays it on the display unit 131. The displayed image is updated thereafter, for example, at 60fps. Using camera and lens operation information, the display unit 131 updates with images showing different ranges of the virtual space in response to camera panning and zooming operations.

[0110] In S1006, the CPU 1001 determines whether the user is in a mode where they move their viewpoint in the virtual space they are observing through the display 131 (viewpoint movement mode). If it is in viewpoint movement mode, the process proceeds to S3000; otherwise, the process proceeds to S1007.

[0111] In S3000, CPU1001 performs viewpoint movement processing, which determines the camera's position and shooting direction in the virtual space. Details will be described later. Once S3000 finishes, processing returns to S2000.

[0112] In S1007, the camera CPU 121, similar to S13 in Figure 8, determines whether the switch Sw1, which instructs the start of the image preparation operation, has been turned on by a half-press operation of the release switch included in the operation switch group 132. If Sw1 is not turned on, the camera CPU 121 returns to S2000 and repeats the determination to monitor the timing when Sw1 will be turned on. On the other hand, if Sw1 is turned on, the camera CPU 121 proceeds to S4000 and performs virtual subject tracking processing.

[0113] The S4000 performs various corrections that affect the generated image in response to the photographer's actions and the movement of the subject, enabling shooting of the subject, which is at least part of the foreground objects. Details of this process will be described later.

[0114] In S1008, the camera CPU 121, similar to S15 in Figure 8, determines whether the switch Sw2, which instructs the start of the imaging operation, has been turned on by fully pressing the release switch. If Sw2 is not turned on, the camera CPU 121 returns to S2000. On the other hand, if Sw2 is turned on, the process proceeds to S5000, and the CPU 1001 executes the virtual imaging subroutine. Details of the virtual imaging subroutine will be described later. When the virtual imaging subroutine finishes, the virtual space imaging process in Figure 13 ends.

[0115] ● Subroutine for generating and outputting images in a virtual space. Using Figure 14, we will explain the subroutine (S2000 in Figure 13) that generates and outputs virtual space images, which is executed by the external computing device 1000.

[0116] In S2001, the foreground object acquisition unit 1102 acquires foreground objects. At this time, the photographer selects the type of subject to be photographed (e.g., person, animal, vehicle, etc.) using the virtual space reproduction device 1100. Next, the photographer selects the shape and color of the subject, and also selects what kind of movement (speed, direction of movement, etc.) the subject should have. The user interface for selection may display the information stored in the foreground object storage unit 1101 of the virtual space reproduction device 1100 on the camera's display 131, allowing the photographer to operate and select. As described above, foreground objects, which are 3D models of the subject, are acquired by multiple methods.

[0117] In S2002, the background object acquisition unit 1105 acquires background objects. As described above, background objects, which are 3D models other than the subject, are acquired by multiple methods.

[0118] In S2003, the object compositing unit 1103 performs object compositing. Object compositing is the compositing of the foreground object and background object mentioned above. The object compositing unit 1103 performs object compositing by determining how to arrange the background object in 3D space and where to place the foreground object in 3D space relative to the background object. First, the object compositing unit 1103 arranges the background object in 3D space, and the photographer selects where to place the foreground object in 3D space. The foreground object is positioned so that it can only be placed in positions that are possible relative to the background object (for example, outside the interior of the background object) based on the 3D model of the background object and the coordinates placed in 3D space. The photographer selects the position to place the foreground object from the 3D space of the background object. Object compositing is then performed.

[0119] In S2004, the camera lens information acquisition unit 1202 acquires camera / lens information. The camera / lens information acquired here is information for generating and outputting images in the virtual space, which will be described later. Specifically, camera information acquired includes the display resolution of the display unit 131 for display, the size and number of pixels of the camera's image sensor, etc. Lens information acquired includes the range and current value of the focal length, the range and current value of the aperture, the focus lens range and current value, and information regarding peripheral illumination falloff and point image distribution function, etc.

[0120] In S2005, the viewpoint information acquisition unit 1201 acquires viewpoint position information. Virtual viewpoint information in 3D space is acquired in order to generate the virtual image described later. The virtual viewpoint information may be a predetermined value as an initial value, or it may be a virtual viewpoint that has been changed by the viewpoint movement process in S3000 described later.

[0121] In S2006, the operation information acquisition unit 1205 acquires camera / lens operation information. Operation information includes information related to framing, zooming, focusing, shutter release, and other button operations.

[0122] In S2007, the image correction amount calculation unit 1206 acquires the image correction amount. The image correction amount is the correction amount related to framing, zooming, and focus. Details will be explained later in the virtual subject tracking processing subroutine of S4000. The image correction amount calculated by the virtual subject tracking processing subroutine is stored in RAM 1003 and acquired in S2007. Before the execution of the virtual subject tracking processing subroutine, a predetermined initial value for the image correction amount is acquired.

[0123] In S2008, the display image generation unit 1204 generates a display image (virtual space image) in a virtual space. The display image generation unit 1204 renders an image based on the aforementioned foreground and background objects placed in three-dimensional space, as well as viewpoint position information. Based on the focal length from the aforementioned lens information, the image sensor size and display unit resolution from the camera information, camera settings, and framing and zooming information from the operation information, the range to be displayed (shooting range in virtual shooting) is determined, and the range to be displayed in the virtual space can be determined by further modifying the range with an image correction amount. Furthermore, a display image with changed aperture value and defocus amount is generated from the aperture value, peripheral light falloff information and point image distribution function information from the lens information, as well as the focus lens position information and image correction amount related to focusing. The display image is not recorded, unlike the recorded image described later. Therefore, the display image range can be correctly determined, and by not using some information such as focus lens position information and peripheral light falloff information, the display image can be generated simply with less information compared to the generation of recorded image.

[0124] In S2009, the display image generation unit 1204 outputs the display video generated in S2008. The output display video is transmitted from the external processing unit 1000 to the camera 100 and displayed on the display unit 131.

[0125] In S2010, CPU 1001 is responsible for saving video-related information. Video-related information includes subject information, shooting-related information, virtual defocus amount, and AF log information. Video-related information is temporarily stored in RAM 1003 of external processing unit 1000 and recorded as video-related information in the virtual shooting subroutine described later. Details will be described later.

[0126] This completes the S2000 subroutine. In this embodiment, the external computing device 1000 generated and output the virtual space image, but the camera 100 may also generate and output the virtual space image.

[0127] ●Subroutine for virtual subject tracking processing Using Figure 15, we will explain the subroutine for virtual subject tracking processing (S4000 in Figure 13).

[0128] Among the various corrections described later, the framing correction is performed when the photographer's framing is off and the subject they wanted to photograph is outside the frame or cut off, so that the subject is properly contained within the camera's displayed field of view.

[0129] Zooming correction addresses situations where the photographer's zoom (lens focal length) is inaccurate, causing the subject to extend beyond the frame or appear too small. The correction ensures the subject is displayed at an appropriate size within the frame. Furthermore, zooming correction includes not only single-timing correction but also timing-based correction, such as when zooming to capture an approaching subject while maintaining a consistent size within the frame. This correction can be applied to achieve smooth focal length changes in cases where the photographer's zoom would result in jerky images.

[0130] Focusing corrections, for example, in autofocus mode, correct for blurred focus caused by fast-moving or rapidly changing subjects based on the camera's tracking algorithm (tracking limit performance), resulting in an image that is in focus with reduced blur. In manual focus mode, it corrects for focus blur caused by the photographer's focusing operation. Furthermore, focusing corrections also include corrections for phenomena such as the focus lens moving towards the background area due to the photographer's framing being off, resulting in blurred focus.

[0131] In S4001, the image correction amount calculation unit 1206 acquires camera / lens information from the camera lens information acquisition unit 1202. The camera / lens information acquired here is for determining whether correction is ON / OFF, acquiring subject difficulty information, and calculating the correction amount, as described later. Specifically, camera information includes camera settings related to correction, such as the shooting difficulty setting set by the photographer. Lens information includes information on focal length, focus lens position, and ON / OFF setting of the image stabilization switch.

[0132] In S4001, the image correction amount calculation unit 1206 uses the acquired camera / lens information to obtain setting information related to the correction to be performed in S4002. The setting information related to the correction includes, for example, setting information such as the ON / OFF status of correction settings within the camera, mode settings such as game difficulty settings, and ON / OFF settings for the image stabilization switch on the lens.

[0133] In S4003, the image correction amount calculation unit 1206 detects framing and acquires information such as whether the camera is panning, in which direction and at what speed.

[0134] In S4004, the image correction amount calculation unit 1206 detects zooming and acquires information such as whether the zoom lens is being operated and at what speed in the Tele / Wide direction.

[0135] In the S4005, the image correction amount calculation unit 1206 detects focusing and acquires information such as whether the focus ring is being operated and at what speed in the direction of near focus or infinity focus.

[0136] Furthermore, detection in S4003-S4005 includes not only manual operations by the photographer, but also automatic operations performed by the camera (such as auto-framing, auto-zoom, and auto-focus).

[0137] In S4006, the image correction amount calculation unit 1206 sets the subject area. Here, it determines which foreground object in the image generated by the object synthesis unit 1103 will be the main subject, and simultaneously sets the area to be AF'd. Furthermore, once the main subject is determined, information regarding the subject's velocity and acceleration, angular velocity and angular acceleration, size of the subject, contrast value of the subject, and distance between the subject and the photographer can be obtained from the foreground object storage unit 1101. There are various methods for setting the subject area, and in this embodiment, it is possible to set it in three-dimensional space. On the other hand, in a camera shooting in real space, it is set by the photographer's framing and the detection result of the subject detection unit in the image (2D) space. In the virtual space shooting of this embodiment, the method for setting the subject area should be as described above, by combining information about the foreground objects, information about objects that are closer, and information about objects closer to the center of the shooting range. Alternatively, the main subject may be detected from the obtained image, similar to when shooting in real space. This makes it possible to reproduce the performance of the camera when shooting in real space more accurately.

[0138] In S4007, the image correction amount calculation unit 1206 determines whether to turn correction ON or OFF based on the various information acquired in S4002 to S4006. For example, in S4002, if the information regarding correction within the camera is ON, the correction is turned ON; if it is OFF, the correction is turned OFF. Alternatively, the image correction amount calculation unit 1206 may use the photographer's intent extraction unit 1262 to extract and determine the photographer's intent from the framing information detected in S4003, such as which subject the photographer is aiming at, whether they are tracking it with framing, or whether they are trying to switch the framing to a different subject. If it is determined that the photographer is tracking the intended subject with framing, the image correction amount calculation unit 1206 turns the correction ON. This allows the photographer's framing mistakes to be covered by the correction. If it is determined that the photographer is trying to switch the framing to a different subject, the image correction amount calculation unit 1206 turns the correction OFF, allowing the photographer to frame (fit into the field of view) the other subject as intended.

[0139] If the zooming and focusing information detected by S4004 and S4005 indicates that manual operation such as manual zoom or manual focus is in progress, the image correction amount calculation unit 1206 determines that the photographer's intent is strong and turns the correction OFF. Alternatively, when using automatic functions such as auto zoom or autofocus, the image correction amount calculation unit 1206 determines that the photographer's intent is weak and turns the correction ON.

[0140] As described above, by turning off correction when it is determined that the photographer's intention is strong, it is possible to provide shooting results and a shooting experience that is close to the photographer's intuitive feel. Therefore, even if correction is turned on when the intention is weak, good images can be obtained through correction without impairing the photographer's shooting experience. In addition, the camera itself has a defined correction capability value, and there is a method of determining whether to turn on correction only if the camera's correction capability value exceeds the subject shooting difficulty (sometimes simply called "subject difficulty"), as described later.

[0141] In the S4100, the image correction amount calculation unit 1206 acquires subject shooting difficulty information using the subject shooting difficulty calculation unit 1261. By calculating various correction amounts described later using the subject difficulty information (for example, the higher the subject difficulty, the smaller the correction amount), it becomes more difficult to keep the subject in the frame or maintain focus for more difficult subjects. On the other hand, by setting a large correction amount for easy subjects, even if the photographer makes a big mistake, a good shooting result can be obtained through correction, thus increasing the success rate of shooting easy subjects. With easy subjects, it is often not a situation where the photographer is concentrating on shooting or enjoying the shooting experience itself, so even if the correction amount is large, the number of failed photos can be reduced without impairing the shooting experience. On the other hand, if the correction amount is large for difficult subjects, it may impair the richness of the shooting experience, so in this embodiment, the correction is small. This is also true for camera shooting in real space, so adjusting the correction amount according to the subject difficulty leads to providing a more realistic shooting experience.

[0142] Here, using Figure 19, we will explain the subroutine for acquiring subject shooting difficulty information (S4100 in Figure 15).

[0143] In S4101, the subject shooting difficulty calculation unit 1261 acquires the subject's velocity and acceleration information from the foreground object storage unit 1101.

[0144] In S4102, the subject shooting difficulty calculation unit 1261 obtains the subject's angular velocity and angular acceleration information from the foreground object storage unit 1101. This information may be time-specific information or fixed, defined information such as maximum velocity and maximum acceleration. The larger these values ​​are, the higher the subject shooting difficulty calculated in S4106.

[0145] In S4103, the subject shooting difficulty calculation unit 1261 obtains subject size information from the foreground object storage unit 1101.

[0146] In S4104, the subject shooting difficulty calculation unit 1261 obtains the contrast value of the subject from the foreground object storage unit 1101. The lower the contrast value, the higher the difficulty of shooting the subject.

[0147] In S4105, the subject shooting difficulty calculation unit 1261 obtains the distance between the subject and the photographer (subject distance) from the information from the foreground object storage unit 1101 and the viewpoint information acquisition unit 1201. By combining the subject distance with the zooming (focal length) information obtained in S4001 and S4004, and the subject size information obtained in S4103, the subject size on the image sensor is determined. The smaller the subject size on the image sensor, the higher the difficulty of photographing the subject. Differences in the part of the subject (e.g., eyes or face) also affect the difficulty of photographing the subject.

[0148] In S4106, the subject shooting difficulty calculation unit 1261 calculates the subject shooting difficulty from the information acquired in S4101 to S4105. The subject shooting difficulty defined here may be defined as a single piece of information encompassing all elements. Alternatively, the subject shooting difficulty may be defined as multiple types of information (framing difficulty, zooming difficulty, focusing difficulty, etc.) corresponding to the framing correction, zooming correction, and focusing correction described later. Furthermore, the subject shooting difficulty may be calculated from these various pieces of information, or the difficulty itself may be stored in the foreground object storage unit 1101. In addition, although this embodiment describes the subject shooting difficulty as being calculated (the subject shooting difficulty changes) each time the subject's speed or distance changes, it may also be defined as always being fixed.

[0149] Returning to Figure 15, in S4009, the image correction amount calculation unit 1206 calculates the virtual defocus amount. The calculation method will be explained below. It is assumed that the distance between objects (or each region) in the virtual space and the user's viewpoint is stored in the form of a map. It is also assumed that there is a table for each lens where distance and focus position correspond. From this table, the focus position value corresponding to each distance is the defocus amount with respect to a certain reference focus position (for example, the infinity position). By offsetting the defocus amount with respect to a certain reference focus position so that the defocus amount at the in-focus position becomes 0, the virtual defocus amount with respect to the in-focus position can be calculated.

[0150] From the above, by calculating the virtual defocus amount for each region from the distance and focus position corresponding to each region, a virtual defocus map, which is a general term for the virtual defocus amounts of each region, can be generated.

[0151] The calculated virtual defocus amount may be a defocus map calculated from multiple regions, as explained using Figure 5, or it may be a single output for a region of the subject (for example, a single part such as a face). In the former case, a process is performed to select one region from multiple regions.

[0152] In the S4200, the image correction amount calculation unit 1206 performs processing on the defocus amount. Details will be described later using Figure 20.

[0153] In S4011, the image correction amount calculation unit 1206 calculates the focus drive amount. As the focus drive amount, the value obtained by converting the virtual defocus amount, to which an error amount has been added in S4200, into a focus lens drive amount may be used as is. Alternatively, the future subject position may be predicted from the subject position in multiple past frames, and the focus drive amount may be set for that predicted position. Various methods of prediction are conceivable, but any known method can be used. In shooting in a virtual space, since the driving of a physical focus lens is not required, the focus drive can be performed without taking time, and the desired focus state can be switched instantaneously. However, in this embodiment, the intention is to provide the photographer with an experience similar to shooting in a real space by shooting in a virtual space using a camera capable of shooting in a real space. For this reason, the process of changing the focus state during shooting (for example, focusing on a subject that is out of focus) is performed over a predetermined time. The time spent on focus drive may be set to match the function / performance of the camera and lens actually being used using camera / lens information, or it may be set assuming a virtual camera and lens.

[0154] In steps S4012 to S4014, the image correction amount calculation unit 1206 calculates various correction amounts. Specifically, in S4012, the image correction amount calculation unit 1206 calculates correction amounts related to framing.

[0155] Here, we will explain framing correction using Figure 21. Assuming there are two subjects, Subject A and Subject B, and Subject A is defined as being more difficult to photograph, the maximum framing correction amount will be smaller for Subject A according to the difficulty of photography (maximum framing correction amount A < maximum framing correction amount B). Also, in Figure 21, the dotted rectangular area represents the area actually framed by the photographer, and the solid rectangular area represents the framed area after applying framing correction.

[0156] As shown in the middle section of Figure 21, the photographer's framing is off relative to subject A, but by applying a correction that falls within the maximum framing correction amount A, the corrected framing area is able to include subject A within the frame. On the other hand, in the lower section of Figure 21, the photographer's framing is even more off relative to subject A, and even with the maximum framing correction amount A applied, subject A could not be included within the frame. Next, for subject B, as shown in the middle section of Figure 21, the photographer's framing deviation is small, so, similar to subject A, the corrected framing area is able to include subject B within the frame. In the lower section of Figure 21, the deviation is larger, but since it falls within the maximum framing correction amount B, unlike the case of subject A, the corrected framing area is able to include subject B within the frame. In this way, by changing the amount of framing correction according to the difficulty of photographing the subject, it is possible to provide a realistic shooting experience where subjects that are more difficult to photograph are harder to frame.

[0157] In S4013, the image correction amount calculation unit 1206 calculates the correction amount related to zooming.

[0158] Here, we will explain the correction related to zooming using Figure 22. If we define that there are two subjects, Subject C and Subject D, and Subject C is defined as being more difficult to photograph, then the maximum correction amount for zooming will be smaller for Subject C according to the difficulty of photography (maximum zooming correction amount C < maximum zooming correction amount D). Also, in Figure 22, the dotted rectangular area shows the field of view that the photographer is actually adjusting by zooming, and the solid rectangular area shows the field of view after applying the zooming correction.

[0159] In the middle section of Figure 22, the photographer's zoom is off relative to subject C, but by applying a correction that falls within the maximum zoom correction amount C, the corrected zoom field of view is able to keep subject C within the frame. On the other hand, in the lower section of Figure 22, the photographer's zoom is even more off relative to subject C, and even with the maximum zoom correction amount C applied, subject A could not be kept within the frame. Next, regarding subject D, as shown in the middle section of Figure 22, the photographer's zoom deviation is small, so, similar to subject C, the corrected zoom field of view is able to keep subject D within the frame. In the lower section of Figure 22, the deviation is larger, but since it falls within the maximum zoom correction amount D, the corrected zoom field of view is able to keep subject D within the frame. In this way, by changing the amount of zoom correction according to the difficulty of photographing the subject, it is possible to provide a realistic shooting experience where it is more difficult to keep subjects that are difficult to photograph within the frame while zooming.

[0160] In S4014, the image correction amount calculation unit 1206 calculates the correction amount related to focusing.

[0161] Here, we will explain focusing correction using Figure 23. If we have two subjects, E and F, and define subject E as being more difficult to photograph, then the maximum focusing correction amount will be smaller for subject E according to the difficulty of photography (maximum focusing correction amount E < maximum focusing correction amount F).

[0162] In Figure 23, Graph 2301 shows how subject E approaches the photographer from a distance as time progresses. Graph 2302 shows how subject F approaches the photographer from a distance as time progresses. The solid lines show the trajectory of each subject's position, the dotted lines show the trajectory of the focus being moved to focus on the subject (the trajectory of the actual focus position), and the dashed lines show the trajectory after focus correction. If the solid line, which represents the subject's position, coincides with the dashed or dotted line, it indicates that an image in focus on the subject can be taken.

[0163] The reason why the subject position and the actual focus position may be misaligned is largely due to the photographer's focusing operation when using manual focus. On the other hand, when using autofocus, it is largely due to the results of the camera's tracking algorithm (tracking limit performance), and examples of causes include the subject's speed being fast and the speed changing significantly. Therefore, Figure 23 illustrates an example where the autofocus tracking limit is reached as time passes, causing the subject position and the actual focus position to be misaligned.

[0164] In Graph 2301, the actual focus position is shifted relative to the subject E, but the corrected focus position matches the subject position up to the range within the maximum focusing correction amount E. However, as time passes and the subject gets closer, the amount of shift in the actual focus position increases, and eventually the maximum focusing correction amount E is insufficient, making it impossible to focus.

[0165] On the other hand, in Graph 2302, even if the actual focus position shifts significantly relative to the subject F, it remains within the range of the maximum focusing correction amount F, making it possible to capture an image with the subject F in focus until the very end.

[0166] Furthermore, as mentioned above, focusing corrections also include blurring caused by manual focus operation and blurring due to mis-measurement of background areas due to framing errors. The correction for these can be the same amount as the correction value for blurring due to the tracking limit performance, or a different amount. Also, considering that these blurs occur simultaneously, a combination of correction amounts for each cause may be applied. In this way, by changing the focusing correction amount according to the difficulty of photographing the subject, it becomes possible to provide a realistic shooting experience where it is more difficult to focus on subjects that are more difficult to photograph.

[0167] In calculating the various correction amounts in S4012 to S4014 described above, the calculations may be performed using past recorded image information and past video correction amounts stored in the memory unit 1004, as long as SW1 detection in the virtual space shooting process shown in Figure 13 continues. Doing so allows for continuity in the correction results between images and reduces the sense of incongruity as a series of recorded images.

[0168] In S4015, CPU1001 performs focus drive (adjustment of the focus state for virtual shooting). Here, the focus drive amount calculated in S4011 and the focus correction amount calculated in S4014 are reflected in the drive.

[0169] As described above, by changing the amount of correction effect applied to the recorded image based on the photographer's operation information, subject information, and camera information, it is possible to take photos in a virtual space without compromising the shooting experience itself.

[0170] In this embodiment, we have described an example where the amount of image correction decreases as the difficulty of photographing the subject increases. However, by conversely increasing the amount of image correction as the difficulty of photographing the subject increases, it is possible to take successful photographs (such as photographs in which the subject is within the frame or in focus) at a certain level regardless of the difficulty of photographing the subject. Furthermore, whether to decrease or increase the amount of image correction as the difficulty of photographing the subject increases can be switched in the camera settings.

[0171] Furthermore, the virtual defocus amount calculation and focus drive processes performed in the S4000's virtual subject tracking can be operated using the AF algorithm stored in the Camera 100's ROM, or they can be operated using the AF algorithm of another camera.

[0172] ●Subroutine for processing virtual defocus amount Next, the subroutine for virtual defocus amount processing (S4200 in Figure 15) will be explained using Figure 20. In this subroutine, an error amount based on one or more parameters that affect the error in the defocus amount that occurs in real space shooting is added to the virtual defocus amount (first virtual defocus amount) calculated in S4009. This generates a virtual defocus amount (second virtual defocus amount) that has an error similar to that in real space. The one or more parameters include at least one of the following: camera information (e.g., image sensor characteristic information), lens information, shooting setting information, and the contrast value of the subject. Details of how to obtain (calculate) the error amount will be explained in S4202.

[0173] Ideally, in virtual space photography, the amount of virtual defocus is calculated from a known subject distance, so calculation errors beyond those caused by the number of significant digits in each value do not occur. On the other hand, in real-world photography, errors occur steadily due to the characteristics of the image sensor, as well as errors that differ with each image. In order to reproduce the same focus behavior in virtual space photography as in real-world photography, it is necessary to introduce the errors that occur in real-world photography into virtual space photography as well.

[0174] In this embodiment, based on the above, an error is added to the virtual defocus amount calculated in S4009, which does not include the error, in order to simulate the shooting of real space.

[0175] In S4201, the CPU 1001 acquires camera information for the virtual image generated by the virtual image generation device 1200. The camera information includes the resolution of the recorded video, the image sensor size, AF frame mode, AF algorithm, S / N information for each ISO sensitivity, and focus-related correction information such as defocus conversion coefficients, focus position correction information, and defocus error information. The CPU 1001 also acquires lens information. The lens information includes the focal length of the lens, resolution, F-number, focus lens information, focus drive control information, sensitivity for converting focus lens drive into image plane movement amount, image stabilization control information, aperture control information, lens frame information, peripheral light falloff information, and distance information related to the focus lens position and distance. The CPU 1001 acquires the camera information and lens information from the camera / lens information storage device 2000 and stores it in the external processing unit 1000 as association information with the virtual image.

[0176] In the S4202, the CPU 1001 retrieves the defocus error information (error amount) stored in the RAM 1003. The defocus error information will be described later.

[0177] In S4203, CPU1001 adds the defocus error amount obtained in S4202 to the virtual defocus amount calculated in S4009. If the virtual defocus amount has been calculated for multiple focus detection regions, the defocus error amount is added to all of the focus detection regions.

[0178] Here, we will explain the defocus error information acquired in S4202 using Figure 24. Figure 24 is a graph showing the relationship between the contrast value of the subject acquired in S4006 and the amount of error expected to occur in the virtual defocus amount. In general, when photographing in real space, if there are few patterns on the subject and the contrast is low, the amount of error included in the detected defocus amount will be large.

[0179] In Figure 24, the horizontal axis shows the contrast of the subject, and the vertical axis shows the amount of error. The straight line 24101 indicates that the greater the contrast of the subject (to the right on the horizontal axis), the smaller the amount of error. This relationship between contrast and error changes depending on the signal-to-noise ratio (SNR) of the image sensor's pixel and readout circuits, the number of pixels used for the focus detection signal, and the gain applied to the signal, which is set by the ISO sensitivity. Therefore, in this embodiment, the relationship shown in Figure 24 is stored for each mode in which the SNR of the image sensor changes, the specifications of the focus detection signal, and the ISO sensitivity. The relationship between contrast and error can be stored discretely as a table. Alternatively, the relationship between contrast and error can be represented as a function, and stored in the form of the coefficients of the function.

[0180] Furthermore, the amount of error changes because the defocus conversion coefficient, which is camera information, changes depending on the lens information, such as the F-number and lens frame information. In this embodiment, the amount of error calculated using the above-mentioned lens information and camera information is multiplied by a predetermined coefficient. The predetermined coefficient only needs to be stored as a table of ratios to a reference value. For example, a table is stored that contains the defocus conversion coefficient and the coefficient multiplied by the above-mentioned amount of error, using the F-number and lens frame information as indices, and the amount of error is calculated according to the shooting conditions.

[0181] Figures 25(a) and 25(b) show examples of virtual defocus amounts with and without error applied to the virtual defocus amount, superimposed on a map relative to the main subject.

[0182] Figure 25(a) corresponds to the case where no error is added to the virtual defocus amount. In the virtual defocus map 25102, the area around the head of person 25106 within the AF frame 25101 of the virtual image is included in the in-focus area 25103 (hatched on a horizontal and vertical grid). Figure 25(b) corresponds to the case where an error is added to the virtual defocus amount. The area around the head of person 25106 within the AF frame 25101 includes the in-focus area 25103, the front focus position 25104 (hatched on a diagonal grid), and the rear focus position 25105 (hatched with black dots).

[0183] In Figure 25(a), all AF frames were in focus, whereas in Figure 25(b), due to the addition of errors, AF frames for front focus and rear focus are created in addition to the in-focus area. In this way, when capturing a virtual image, an error corresponding to the change in camera / lens information is added to the virtual defocus amount, and by applying the algorithm used when selecting AF frames, it is possible to obtain defocus detection results similar to those obtained when shooting in the real world. In this embodiment, for the sake of clarity, an example of adding a defocus error on the defocus map is shown, but a defocus error may also be added to a single AF frame. Since this error affects predictive AF, a similar effect can be expected. As a result, even when shooting in a virtual space, the same focus adjustment behavior as when shooting in the real world can be reproduced, and it is possible to evaluate the performance of products and check new functions before purchasing cameras and lenses.

[0184] In this embodiment, an error is added to the virtual defocus amount in order to bring the virtual space shooting results closer to the results of shooting in the real space, but adding an error is not essential. If it is not necessary to bring the results closer to the results of shooting in the real space, it is possible to consider not adding an error, or to switch on or off depending on the situation. If no error is added, the calculation of the focus drive amount in S4011 in Figure 15 is performed based on the virtual defocus amount calculated in S4009. Therefore, the CPU 1001 can switch whether or not to add an error, thereby performing a process (switching process) to switch whether to adjust the focus state based on the virtual defocus amount with an error added (second virtual defocus amount) or based on the virtual defocus amount without an error added (first virtual defocus amount).

[0185] ●Virtual imaging subroutine Next, using Figure 16, we will explain the virtual imaging subroutine (S5000 in Figure 13) executed by the external computing device 1000.

[0186] In S5001, CPU1001 outputs the set F-number and the time SW2 was detected. The use of this information will be explained later in the actual camera operation in conjunction with the virtual shooting operation shown in Figure 28.

[0187] In S5002, CPU1001 acquires camera / lens information. Camera information includes the resolution for recording, as well as the size and number of pixels of the camera's image sensor. Lens information includes the range and current value of the focal length, the range and current value of the F-number, the focus lens range and current value, as well as information on peripheral light falloff and point image distribution function.

[0188] In S5003, CPU1001 obtains the aforementioned image correction amount.

[0189] In S5004, CPU 1001 generates recorded video of the virtual space. CPU 1001 renders an image from the aforementioned foreground and background objects placed in 3D space and viewpoint position information. CPU 1001 determines the range to be recorded video from the focal length from the aforementioned lens information, the image sensor size and resolution from the camera information, and the camera settings. Furthermore, CPU 1001 generates recorded video from the F-number from the lens information, peripheral light falloff information, point image distribution function information, and focus lens position information. Recorded video differs from the aforementioned display video in that it is recorded video. Therefore, CPU 1001 correctly determines the range of the recorded video and generates recorded video using various optical information such as focus lens position information, peripheral light falloff information, and point image distribution function information. Unlike the display video, recorded video does not need to be displayed to the photographer in real time. Therefore, the generation of recorded video can be delayed compared to the generation of display video. For this reason, CPU 1001 can generate recorded video using detailed data regarding camera information and lens information.

[0190] In S5005, the CPU 1001 records video footage of the virtual space. The recorded video footage is stored in the memory unit 1004 of the external computing device 1000. Alternatively, the CPU 1001 may transfer the video footage generated in S5004 to the camera 100, and the camera 100 may record the video footage in the flash memory 133.

[0191] In S5006, the CPU 1001 records various video-related information. Video-related information includes subject information (shooting difficulty information), shooting-related information, and virtual defocus amount. The video-related information is recorded in the memory unit 1004 of the external processing unit 1000. Alternatively, the CPU 1001 may transfer the previous video-related information to the camera 100, and the camera 100 may record the video-related information in the flash memory 133. With this, the virtual shooting subroutine terminates.

[0192] ● Virtual shooting that reflects operation information Figure 18 illustrates virtual imaging that reflects the operation information. Figure 18(a) is an example of zooming operation, and Figure 18(b) is an example of framing operation.

[0193] In Figure 18(a), the static virtual space display image generated by the display image generation unit 1204 is displayed on the camera 100's display unit 131 as the virtual space display image 18003. When the photographer rotates the lens zoom ring (arrow 18001) and significantly changes the focal length, the display image generation unit 1204 acquires the resulting focal length as operation information. Then, when generating the virtual space display image, the display image generation unit 1204 changes the range based on the operation information to generate a virtual space display image 18004 that reflects the focal length result from the photographer's zooming operation, and displays it on the display unit 131.

[0194] In this embodiment, the display image generation unit 1204 generates a display image in the virtual space using camera / lens information. Therefore, it is possible to generate a display image in the virtual space within a focal length range that cannot be operated by the lens being operated by the photographer. For example, even if the photographer is operating a lens with a short focal length, by generating a display image in the virtual space using lens information of a telephoto lens with a long focal length, it becomes possible to shoot as if using a telephoto lens with a long focal length. Generally, lenses with long focal lengths used for shooting in the real world are large, heavy, and expensive, but in the virtual space shooting of this embodiment, it is possible to provide a shooting experience without such constraints.

[0195] In Figure 18(b), the static virtual space display image generated by the display image generation unit 1204 is displayed on the camera 100's display unit 131 as the virtual space display image 18006. In Figure 18(b), the photographer performs a camera / lens framing operation, moving the camera lens in the direction of the arrow 18005, which is horizontal. In this case, the display image generation unit 1204 acquires framing information, which is the position of the camera / lens due to the framing operation, as operation information. Then, when generating the virtual space display image, the display image generation unit 1204 changes the range based on the operation information to generate a virtual space display image 18007 that reflects the focal length result from the photographer's framing operation, and displays it on the display unit 131.

[0196] As a result, it is possible to generate a virtual display image that reflects the photographer's operation information using virtual still image capture.

[0197] ● Subroutine for viewpoint movement processing Next, the subroutine for viewpoint movement processing (S3000 in Figure 13) will be explained using Figure 26. Viewpoint movement processing refers to the process of changing the viewpoint position (camera position in the virtual space) when taking images in the virtual space. This process is performed by CPU 1001.

[0198] In S3001, the CPU 1001 adjusts the depth of field and field of view of the displayed image when the viewpoint is moved. When moving the viewpoint, it is desirable to be able to easily see a wider range of subjects and to be able to easily see the distance at which the focus is on. This is because, as will be described later, the destination of the viewpoint is set from the field of view and the object at the distance at which the focus is on, as displayed on the display 131. In this embodiment, in S3001, the CPU 1001 widens the field of view to a pre-set field of view and adjusts the range of the distance at which the focus is on to a pre-set depth of field. Note that the processing in S3001 is intended to facilitate the operation of moving the viewpoint and may be omitted.

[0199] In S3002, CPU1001 displays images of the virtual space based on the settings made in S3001. CPU1001 displays images from the viewpoint position used for pre-existing virtual space captures or from the viewpoint position set as the initial position.

[0200] In S3003, the CPU 1001 adjusts the focus and the indicator direction. First, in adjusting the focus, as explained in S3001, the state in focus is displayed within a predetermined distance range within the shooting range, and the operator performs the same focus adjustment operation as during shooting. Specifically, with one focus detection indicator (I(n,m)) as explained in Figure 5 displayed, the operator adjusts the focus on an object at the position where the viewpoint is to be moved within the shooting range by half-pressing the release switch included in the operation switch group 132. This allows the operator to see the distance at which the image is in focus on the display unit 131. The CPU 1001 also sets the distance at which the image is in focus as the viewpoint movement distance.

[0201] Regarding the setting of the viewpoint shift distance, although the method described is the same as for autofocus adjustment, it may also be done by manually operating the focus lens (third lens group 105) of the imaging optical system, in other words, in the same way as manual focus operation. By rotating the focus ring (not shown) provided on the imaging optical system, the in-focus distance can be adjusted to infinity or near, and the viewpoint shift distance can be adjusted to the distance intended by the operator.

[0202] Furthermore, adjusting the indicator direction is done to set the direction in which the viewpoint will be moved. The operator changes the position of the indicator (I(n,m)) used for focus detection on the screen of the display 131, or moves the camera 100 by panning, etc., and aligns the indicator with the direction in which the viewpoint will be moved. This allows the photographer to set the direction of viewpoint movement while checking the image of the virtual space displayed on the display 131.

[0203] In S3004, the CPU 1001 determines whether or not there is an instruction to move the viewpoint. If the viewpoint position change button included in the operation switch group 132 is pressed, the process proceeds to S3005. If the viewpoint position change button is not pressed, the process returns to S3003, and the operator continues to adjust the focus and indicator direction.

[0204] In S3005, the CPU 1001 determines whether or not viewpoint movement is possible. When the operator presses the viewpoint position change button to instruct viewpoint movement, the direction and distance of viewpoint movement are determined. If the viewpoint after movement is below the ground (underground) of a background object, inside a foreground object, or if the camera after viewpoint movement interferes with other objects, the CPU 1001 determines that viewpoint movement is not possible. Also, if the distance at which the viewpoint is in focus when the viewpoint position change button is pressed is infinity, the CPU 1001 determines that viewpoint movement to infinity is not possible. If the viewpoint movement distance is farther than a predetermined distance, the CPU 1001 may reset the viewpoint movement distance to a predetermined distance set in advance as the maximum value.

[0205] In S3006, CPU 1001 determines whether viewpoint movement is possible based on the viewpoint movement feasibility determination in S3005. If viewpoint movement is possible, the process proceeds to S3007. On the other hand, if it is determined that viewpoint movement is not possible, the process proceeds to S3008.

[0206] In S3007, CPU1001 performs viewpoint movement according to the viewpoint movement distance and direction set in S3003.

[0207] In S3008, the CPU 1001 notifies the operator via the display 131 that viewpoint movement is not possible. The CPU 1001 may notify only that viewpoint movement is not possible, or it may also notify the operator of the reason, such as interference with an object or that the set movement distance is too far.

[0208] In S3009, CPU1001 determines whether or not it has received an instruction to terminate the viewpoint movement mode. If it has received an instruction to terminate, this subroutine terminates. If there is no instruction to terminate the viewpoint movement mode, processing returns to S3003.

[0209] Next, using Figure 27, a specific example of the viewpoint movement processing subroutine described in Figure 26 will be explained. Figure 27 shows an example of the display on the display unit 131 in viewpoint movement mode. Figure 27(a) shows an example of setting viewpoint movement in viewpoint movement mode in a virtual space where a person and a dog are placed as foreground objects. The foreground objects 27003 of the person and dog are displayed on the display screen 27001 of the display unit 131, and the camera is positioned at a viewpoint from the upper left of the person, as is the state before viewpoint movement. 27002 is one of the indicators (I(n,m)) described in Figure 5, and indicates the direction in which the viewpoint moves on the display screen. 27005 shows the viewpoint movement distance along with the range in which viewpoint movement is possible. In Figure 27(a), it is possible to move from 0.45m to 10m, and the current adjustment distance is 1m.

[0210] In the example in Figure 27(a), the object is in focus at a distance of 1m from the current viewpoint (camera position). However, the figure does not show the in-focus and out-of-focus distances, and it shows a state where everything from very close to infinity is in focus. Indicator 27002 is movable within the display screen 27001. By panning the camera, etc., indicator 27002 can be superimposed on an object such as a dog, and the focus can be adjusted to that distance, i.e., the viewpoint shift distance can be set. Sub-display screen 27004 displays a preview image of the foreground object 27003 as observed from the viewpoint after the viewpoint shift that is currently set. The direction of the image to preview from the viewpoint after the viewpoint shift may be set automatically based on the position information of the foreground object, or it may be set by the operator using an operation switch. In addition, a rectangular frame may be shown within sub-display screen 27004 to indicate the shooting range corresponding to the focal length of the lens being used.

[0211] In this way, by using the screen displayed on the display screen 27001 and the indicator 27002 to set the viewpoint movement distance and direction, the operator can intuitively and easily move the viewpoint using the same operation as when taking a picture.

[0212] Figure 27(b) illustrates a modified version of viewpoint movement. Figure 27(b) shows a method in which a viewpoint movement target is placed and displayed as the target position on the display screen, making it easier for the operator to move the viewpoint.

[0213] Figure 27(b) shows how the viewpoint movement targets (target positions) are displayed on the screen in viewpoint movement mode. The viewpoint movement targets 27006 are displayed in a grid pattern on the ground, which is part of the background object of the display screen 27001. At the intersections of the grid, the intersection of the 2nd row and 1st column is shown as viewpoint movement target 27006(2,1), and the intersection of the 4th row and 4th column is shown as viewpoint movement target 27006(4,4). The operator can set the viewpoint movement distance by moving the indicator 27002 closer to the viewpoint movement target 27006 that is close to the distance to which they want to move their viewpoint, and selecting one of the intersections. The viewpoint movement targets may be displayed superimposed on the foreground object, or they may be displayed together with the numerical value of the distance of the viewpoint movement target. By displaying the viewpoint movement targets in this way, the operator can set the viewpoint movement distance more easily.

[0214] ●Feedback of the virtual shooting operation through the operation of the camera drive unit Next, Figure 28 will be used to explain the camera's operation during virtual shooting. In real-world shooting, the operation of the shutter and lens provides tactile feedback to the photographer, such as vibration and sound, which improves the quality of the shooting experience. On the other hand, as mentioned above, in virtual shooting, virtual subject tracking and virtual shooting may be achieved by operating the release switch of the camera 100's operation switch group 132. In this case, there is no drive unit necessary for image generation, and there is no need to drive the real shutter or lens. However, this may lead to a decrease in the quality of the shooting experience. In this embodiment, by driving the camera 100's drive unit in synchronization with the virtual shooting operation, tactile feedback such as vibration and sound is provided to the photographer, realizing a more realistic shooting experience.

[0215] Figure 28 is a flowchart illustrating the camera operation during virtual shooting. Each process is executed by the camera CPU 121.

[0216] In S1201, the camera CPU 121 receives an initialization instruction for the camera drive unit from the CPU 1001 (S1002 in Figure 13) and performs the initialization of the camera drive unit. The camera CPU 121 drives the shutter to match the camera settings in the virtual space. For example, if shooting is about to begin, the camera CPU 121 drives the shutter to the open state. In addition, the zoom actuator 111, aperture actuator 112, and focus actuator 114 are driven according to the angle of view, depth of field, focal length and aperture value corresponding to the distance at which the image is in focus, and the focus lens position at the start of virtual shooting. In this embodiment, the initial position of the camera 100's drive unit is described as being set by an instruction from the CPU 1001, but the position information of each drive unit of the camera 100 may be output to the external computing device 1000, and the external computing device 1000 may determine the initial state.

[0217] In S1202, the camera CPU 121 outputs camera information, lens information, and operation information, corresponding to S1003 in Figure 13. The content of the information is as described above.

[0218] In S1203, the camera CPU 121 acquires the image generated in S2000 in Figure 13 and displays it on the display unit 131.

[0219] In S1204, the camera CPU 121 monitors whether a focus drive instruction corresponding to S4015 in Figure 15 is input to the camera 100. If no focus drive instruction is input, the process proceeds to S1206; if a focus drive instruction is input, the process proceeds to S1205.

[0220] In step S1205, the camera CPU 121 drives the focus lens (third lens group 105) using the focus actuator 114 in accordance with the focus drive instruction.

[0221] In S1206, the camera CPU 121 monitors whether the aperture value input to the camera 100 in S5001 (Figure 16) differs from the current setting. If there is no change in the aperture value, the process proceeds to S1208; if there is a change in the aperture value, the process proceeds to S1207.

[0222] In S1207, the camera CPU 121 drives the aperture 102 using the aperture actuator 112 according to the change in aperture value.

[0223] In S1208, the camera CPU 121 monitors whether the detection time of SW2 is input to the camera 100 in S5001 of Figure 16. If the detection time of SW2 is not input, the process proceeds to S1210. If the detection time of SW2 is input (i.e., still image capture is performed), the process proceeds to S1209.

[0224] In S1209, the camera CPU 121 drives the shutter 106 in the same way as when taking pictures in real space, after a predetermined time has elapsed since the detection time of the input SW2.

[0225] In S1210, the camera CPU 121 determines whether the main switch included in the operation switch group 132 is turned off or not. If the main switch is turned off, the camera CPU 121 terminates the processing in this flowchart; otherwise, it returns to processing in S1204.

[0226] As described above, in virtual space photography, by driving the focus lens, aperture, and shutter, the photographer can receive feedback such as vibrations and sounds from the operation of the camera, resulting in a more realistic shooting experience.

[0227] In this embodiment, the drive unit of the camera 100 is configured to be driven in response to operation instructions related to shooting in the virtual space. However, depending on the operation instructions that are given, the camera 100 may not be able to be driven. For example, this may occur if the operation instructions are for a continuous shooting speed that can be driven by the camera 100 or if the operation instructions are for driving a focus lens with a longer distance than the lens that is attached. In such cases, the drive unit of the camera 100 may be prohibited from being driven based on the operation instructions that have been given, or the operation instructions may be edited to change them to instructions that can be driven by the drive unit of the camera 100, and then driven. For example, if the operation instructions are for a speed that can be driven by the camera 100 or higher, it is conceivable to skip operation instructions at regular intervals and drive the shutter. Furthermore, regarding the driving of aperture and focus, it is conceivable to compare the specifications of the lens used in the virtual space with the specifications of the lens attached in the real space, standardize them to match the drive range, and then standardize the amount of drive in the same way when an operation instruction is given.

[0228] Furthermore, while this embodiment describes the feedback of the user's perception of operation, the sounds and vibrations generated during this process may also be recorded. For example, blur corresponding to the amount of camera vibration may be added to still images, or the resulting sounds may be recorded in the video. This makes it possible to achieve virtual shooting that is closer to shooting in the real world.

[0229] ●Replay and evaluation method of shooting results Figure 29 is a flowchart showing the image playback and evaluation method after shooting with the camera 100 of this embodiment. Specifically, the image defocus map is displayed and the degree of focus of a series of consecutive images is calculated during playback of images captured in the real space, playback of images captured in a virtual space, and under conditions different from those used during actual shooting. This allows for the display of the causes of poor focus and the display of the best settings to improve them. In the following description, S means step.

[0230] In S1101, the camera CPU 121 selects the image to be played back from the flash memory 133. By operating the control switch group 132 that the photographer uses to instruct playback, the most recently captured image or the previously played image can be displayed. After that, the photographer operates the control switch group 132 to play back the desired image.

[0231] In S1102, the camera CPU 121 determines whether the image to be played back was taken in a virtual space or in a real space. If the camera CPU 121 determines that the image to be played back was taken in a virtual space, it proceeds to S1103; if it determines that the image to be played back was taken in a real space, it proceeds to S1104.

[0232] In S1103, the camera CPU 121 plays back the captured image from the virtual space stored in the memory unit 1004 of the external computing device 1000 on the display 131.

[0233] In S1104, the camera CPU 121 plays back the captured images of the real world stored in the flash memory 133 on the display 131.

[0234] In S1105, the camera CPU 121 decides whether to perform an image capture evaluation of the image played back in S1103 or S1104. If an image capture evaluation is performed, the process proceeds to S1106; otherwise, the process in this flowchart ends.

[0235] In S1106, the camera CPU 121 acquires shooting-related information when the image to be played back was taken. Shooting-related information refers to the camera settings and various lens information set at the time of shooting. Shooting-related information includes lens and camera settings at the time of shooting, such as focal length, f-number, continuous shooting mode, AF mode, subject detection AF tracking setting, AF frame setting, and shutter method. Shooting-related information is used when evaluating the focus state of the image, as described later, and can be any information that affects the focus state. Shooting-related information may be stored in the flash memory 133 or the memory unit 1004, or it may be attached to the image being played back as metadata.

[0236] In the S1107, the camera CPU 121 acquires AF log information attached as metadata to the playback image. The AF log information includes defocus information when the playback image was taken, AF frame setting information, tracking information (the subject detection AF function focuses on the set detected object, such as a person, animal, or vehicle, which is automatically detected using a set algorithm), servo AF characteristics (setting the focus priority by assigning various servo AF parameters), action recognition information (information on the subject's posture, and information on prioritizing subject recognition when the subject performs a specific action), and shutter method information (you can select a shutter mode, such as a mechanical shutter mode that drives the mechanical shutter or an electronic shutter mode that determines the exposure time using only the image sensor without using a mechanical shutter, and check the setting information for continuous shooting frame rate, such as 30, 20, or 10 frames per second for the electronic shutter).

[0237] In S1108, the camera CPU 121 sets one or more images, including the image being played back, as an evaluation image group. The evaluation image group can be set by selecting images taken at a time close to the time the image being played back was taken, or by selecting a group of images taken in a single burst of shots. Alternatively, the system can be configured to allow setting the beginning and end of the evaluation image group.

[0238] In S1109, the camera CPU 121 sets the evaluation sequence. The camera CPU 121 (or the external computing device CPU 1001 in the case of virtual space imaging) determines the equipment to be used and various algorithms, and decides what kind of evaluation to perform. Details will be described later.

[0239] In S1110, the camera CPU 121 performs evaluations under each setting condition. The camera CPU 121 acquires various information from S1106 to S1109 as described above and performs evaluations under the conditions of various setting contents. As a result, it is possible to calculate the amount of defocus in the captured image from the focus control result, which is different from that at the time of shooting, for the image being played back, and to evaluate the amount of focus deviation based on a threshold determined from the amount of defocus, thereby calculating the degree of focus.

[0240] In calculating the degree of focus, the camera CPU 121 simultaneously performs image analysis to analyze the causes of good or bad focus. Furthermore, the camera CPU 121 calculates the amount of defocus from the focus control results, which differ from those during shooting, and creates a defocus map on the captured image, which is then superimposed and displayed.

[0241] This allows us to evaluate the difference in focus state between images taken with the same conditions as when the image was acquired and images taken with different conditions.

[0242] In S1111, the camera CPU 121 displays the evaluation results performed in S1110 on the display unit 131 or an external display such as a PC. The method of displaying the evaluation results is not limited to one format. The display methods will be described later. By displaying the evaluation results, the photographer can check the cause of poor focus, which can lead to improvements in shooting technique, such as correcting the shooting method or changing the shooting settings.

[0243] In S1112, the camera CPU 121 presents the best settings. Based on the evaluation results from S1111, the camera CPU 121 presents the best settings based on the evaluation results regarding the degree of focus in the image. An explanation of the presented content and an example of the display will be described later. The photographer confirms and changes the best settings displayed on the display unit 131, but a menu can also be provided to decide whether to automatically change the settings using the evaluation results before the evaluation is performed in the playback image settings in S1101. By selecting automatic change, the camera settings can be automatically changed to the best settings based on the evaluation results.

[0244] Furthermore, similar processing can be performed in parallel during shooting. For example, during continuous shooting, the system can automatically switch to the optimal setting for the fourth shot based on the results of evaluating the first three shots, and continue continuous shooting without requiring confirmation from the photographer.

[0245] ● Defocus map display Next, using Figure 30, we will explain the display of the captured image with a defocus map superimposed, which is performed in S1111 based on the evaluation results of S1110. Figure 30(a) shows an example of a scene of a person skiing.

[0246] Figure 30(b) shows a defocus map 30001 superimposed on the captured image, based on calculations of the focus control result from the shooting-related metadata attached to the captured image. This defocus map displays on the captured image whether the focus position is in the positive direction (front focus) or the negative direction (back focus) relative to 0 for each 10x8 grid-like block.

[0247] The vertical line pattern frame 30002 displayed within each block indicates that the focus position is near 0, meaning the subject is in focus.

[0248] The diagonal line pattern frame 30003 displayed within each block indicates a positive defocus amount, suggesting a tendency towards front focus.

[0249] The dotted frame 30004 displayed within each block indicates a negative defocus level, suggesting a tendency towards back focus.

[0250] The block diagram of a person skiing is displayed as a frame with roughly vertical lines (30002), indicating that the focus is on the subject.

[0251] Figure 30(b) is just one example; the defocus map does not need to be a 10x8 grid and may be displayed in finer detail. In Figure 30(b), the amount of defocus is shown for the area that roughly encompasses the main subject, but it is not limited to this area and the amount of defocus may be shown for the entire captured image.

[0252] Figure 30(c) shows a superimposed defocus map 30001 based on calculations of focus control results from the shooting-related metadata attached to the captured image taken with the camera (product name CA) and lens (product name LA) combination used by the photographer.

[0253] AF frame 30000 is the result of shooting under conditions where only one point is used as the focus detection area based on the camera's AF frame setting information. The evaluation result shows that when the AF frame covers half of the subject's face, the contrast of the background subject affects the focus, resulting in the focus being further away from the main subject and a deterioration in the degree of focus. The diagram shows that the amount of defocus at the focus position of each block is indicated by a diagonal line, indicating a tendency for front focus on the right side of the subject.

[0254] Figure 30(d) shows the defocus result when AF frame 30000 is changed from the single-point AF described above to a wide-area AF with a broader AF frame. Figure 30(d) corresponds to the evaluation sequence settings performed in S1109 for images taken with the camera (product name CA) and lens (product name LA) combination used by the photographer as described above.

[0255] In the S1110, the amount of defocus is calculated based on the single-point AF frame setting information of the actual captured image and the focus control information when the setting is changed to a wide-area AF frame. The camera CPU 121 displays the result as a defocus map 30001, as shown in Figures 30(c) and 30(d). By showing the change in the defocus map 30001 due to the change in the AF frame setting, it is possible to compare the amount of defocus with that of single-point AF and that of area-expanded AF.

[0256] Figure 30(d) shows that the vertical line pattern frame 30002 is heavily superimposed on the subject, indicating that the background does not affect the image and a focused image of the subject can be obtained. In the example shown in Figure 30, it can be confirmed that, depending on the photographer's framing skills, shooting with a wide-area AF frame setting rather than a single-point AF can yield better defocus results.

[0257] In addition, the camera CPU 121 may obtain the necessary AF information from virtual camera information of a different camera (product name CB) than the original camera (product name CA) and existing lens information. Then, the camera CPU 121 can compare the AF performance difference between camera (product name CA) and (product name CB) by rewriting the focus-related information of the captured image with the focus control information of camera (product name CB). This allows for evaluation of the degree of performance improvement for each shooting scene, especially when performance improvements are expected with new products. This evaluation result can be used when considering purchasing new products.

[0258] Similarly, with respect to lenses, the camera CPU 121 may acquire lens information for a different virtual lens (product name LB) from the original lens (product name LA), which is different from the lens used in the playback image. The camera CPU 121 can then acquire a virtual defocus amount for the playback image, combining different focal lengths and f-numbers. As a result, the defocus information of the virtual lens (product name LB) can be compared with the defocus information of the image taken using the original lens (product name LA) to display and verify the performance when the lens is changed.

[0259] These displays may be shown on the display device of an external PC instead of the display 131 of the camera 100. Regarding the method of comparing the defocus map images after the change, the images before and after the change may be arranged side by side, or only the defocus amount of the defocus map may be changed. The defocus amounts in FIG. 30 are shown by classifying the focus position into three stages: near the in-focus position, front pin, and rear pin, but they may be classified more finely or the defocus amount may be shown in units of mm or the like.

[0260] The photographer can confirm the performance difference by changing the focus control result based on the information of a camera or lens different from the camera and lens used for shooting. Also, since the performance can be confirmed before purchasing a desired camera or lens, etc., a camera or lens that meets the photographer's requirements can be selected.

[0261] In FIG. 30, the shooting in the real space is described, but the same processing may be applied not only to the real space but also to the images obtained by shooting in the virtual space using the external computing device 1000.

[0262] ● Degree of focus An example of displaying the degree of focus based on the evaluation result by the process of FIG . 29 for a series of images obtained by continuous shooting will be described using FIG. 31.

[0263] Figure 31 shows how photographer 31003 continuously photographs a series of scenes of a subject skiing, and how image 31001 is obtained. The images used here may be taken in real space or in virtual space. The camera CPU 121 calculates the amount of defocus based on the evaluation results of the series of images taken (obtained by processing in S1110). If the calculated amount of defocus is within a predetermined threshold range centered on 0, the camera CPU 121 determines that the degree of focus is "○", and otherwise determines that the degree of focus is "×". The camera CPU 121 makes a "○" or "×" judgment for each image and displays the percentage of "○" images in the entire series of images obtained in continuous shooting on screen 31002.

[0264] Figure 31 shows an example where the percentage of images with a "○" (in focus) rating was 70%. In this case, the camera CPU 121 determines that there is room for improvement and displays a "△" (in focus) next to the percentage "70%" on screen 31002. If the percentage of images with a "○" rating is 80% or higher, the camera CPU 121 may display a "○" on screen 31002 to clearly indicate the degree of focus to the photographer.

[0265] Furthermore, the display of symbols can be freely configured, such as displaying an "×" on screen 31002 if the percentage of focus is "○" is 60% or less, or a configuration can be adopted that displays only the percentage of focus without displaying any symbols.

[0266] In the above explanation, the focus determination result for each image is given as either "○" or "×," but the method for determining focus is just one example and can be freely determined. For example, the camera CPU 121 may display the dispersion using the defocus calculation unit mm. The display method can also be freely set without detailed configuration. In this example, the series of focus determination results are displayed on the camera's display unit 131, but as another example, they could be viewed on a display device such as a PC.

[0267] ●Example of setting changes Figure 33 is a table showing examples of the modifiable items and evaluation conditions for the proposed configuration changes, which are evaluated in S1110. Examples of modifiable items are shown horizontally.

[0268] The items written in the white frame are examples of camera information settings, showing AF frame settings, subject detection AF tracking, AF mode (one-shot AF or servo AF), servo AF characteristics related to various servo AF parameters, and shutter type. The diagonal frame is an example of lens information settings, showing the lens and focal length.

[0269] Figure 33 shows the initial settings, which are the shooting settings for image acquisition. Recommended settings 1 and 2 are examples of evaluation sequences set in S1109 in Figure 29. The number of evaluation sequences is not limited to two. By evaluating in S1110 using all combinations of changeable parameters, the best settings (evaluation sequence) that maximize the degree of focus can be found. On the other hand, since this increases the computational load, it is also possible to reduce the computational load by changing only the parameters that are effective in improving the degree of focus from the initial settings.

[0270] Using the recommended settings in Figure 33, we will explain settings that are effective in improving the degree of focus. With the default settings set by the photographer, the following situations can be considered as factors that cause the AF frame to move away from the subject. With the combination of single-point AF and tracking "off", the AF frame 32001 (Figure 32) visible in the viewfinder is fixed. Therefore, the photographer needs to keep the AF frame aligned with the subject, making it difficult to frame due to unexpected subject movements. In recommended setting 1, even with single-point AF, subject detection is performed by turning on and using tracking AF. Therefore, the AF frame 32002 can automatically capture and track the subject. Therefore, the photographer only needs to overlap the subject with the AF frame and start tracking at the start of shooting, and can concentrate only on getting the subject into the frame, thus reducing the difficulty of framing. Settings related to the AF frame and subject detection AF tracking can be changed and evaluated for both images taken in real space and images taken in virtual space. By using the image and the defocus map information acquired at the time of image acquisition, a new AF frame can be selected by applying an algorithm for the AF frame settings to be applied after the change. Furthermore, a new subject detection area can be set by applying tracking algorithms and other settings to the image after the change.

[0271] Similarly, in Recommended Setting 2, the Expanded Area AF setting is selected to widen the AF frame area compared to single-point AF. This setting makes framing easier without using tracking.

[0272] Regarding shutter methods, with electronic front curtain shutters, the shutter curtain is driven with each release, resulting in blackouts. These blackouts cause the photographer to momentarily lose sight of the subject, and the display update rate also decreases, making framing difficult. Electronic shutters do not use a shutter curtain, so the display update rate does not decrease during continuous shooting, and blackouts (completely black screen) do not occur. Therefore, especially when continuously shooting moving subjects, electronic shutters can reduce the difficulty of framing without losing sight of the subject. Regarding shutter method settings, for images captured in real space, it is possible to change the frame rate in the direction of slowing down (dropping images), but changing it in the direction of speeding it up is difficult due to the lack of information. For images captured in virtual space, it is possible to regenerate images in the shooting environment, so it is possible to change the frame rate in the direction of speeding it up. Therefore, when evaluating images captured in real space, if the frame rate is slow, such as when using an electronic front curtain shutter, it may be possible to recommend the use of an electronic shutter to the photographer using other information such as the speed of the subject being photographed.

[0273] Regarding the lens focal length, the photographer is assumed to be using a 70mm-200mm zoom lens. During shooting, the focal length is set to 200mm, resulting in a narrow field of view relative to the subject. This makes framing difficult, as unexpected movements or fast sliding movements of the subject are more likely to cause them to go out of frame. By widening the field of view, there is more leeway in the field of view to accommodate unexpected movements or fast sliding movements of the subject, thus reducing the risk of them going out of frame. Therefore, setting the focal length to the wider end, 70mm, can lower the difficulty of framing. Regarding the setting of focal length, it is possible to change the field of view in the direction of narrowing the field of view for images taken in real space, but it is difficult to change it in the direction of widening the field of view because there is no information (image). For images taken in virtual space, it is possible to regenerate the image in the shooting environment, so it is possible to change the field of view in the direction of widening the field of view. Therefore, when evaluating images taken in real space, if the shooting was done with a long focal length, it may be possible to recommend to the photographer the use of a wider-angle lens, using other information such as the speed of the subject being photographed.

[0274] As mentioned above, by considering factors that reduce the degree of focus and changing the settings, it is possible to find recommended settings that improve the degree of focus more efficiently and reduce the computational load. The setting method in Figure 33 is just one example, and settings may be set based on various considerations. Analysis results of the shooting environment, such as whether the subject is human or animal, or whether it is a scene where multiple subjects intersect, can also be utilized. Furthermore, the recommended settings may be narrowed down by determining the photographer's skill from the shooting history entered in advance and the movement of the subject and the photographer's framing during shooting.

[0275] ●Explanation of information display regarding shooting settings Next, using Figure 32, we will explain the display of information about the photographer's framing technique in the captured image, which is performed based on the evaluation results of the degree of focus described above.

[0276] Figure 32(a) shows a blurry image that, based on the evaluation results of a series of consecutively captured images (obtained in S1111), has a large amount of defocus and is judged to have a poor focus rating. It is assumed that the framing did not keep up with the movement of the subject skiing at high speed, and the AF frame 32001 was outside the subject's face, resulting in a poor focus rating. It is presumed that the shutter type, AF frame selection, and angle of view due to the lens's focal length, based on the camera settings set by the photographer, made framing difficult. In such situations, the camera displays a suggestion for the best settings based on the image evaluation during image playback, and displays this suggestion on the camera's display 131 or a display device such as a PC. The suggestion and display of the best settings will be explained in Figure 32(b).

[0277] Figure 32(b) shows an example where, after evaluating various shooting sequences, the best settings are suggested to the photographer.

[0278] In the evaluation of each setting condition in S1110, the system calculates many different types of conditions by changing various combinations of information and setting conditions from S1106 to S1109, and then searches for the setting condition that yields the highest degree of focus. For example, in Figure 32(a), the AF frame 32001 is outside the subject, but in the evaluation of S1110, the camera CPU 121 calculates the degree of focus when the AF frame is widened from the AF frame setting in the AF log information of S1107. The camera CPU 121 checks the degree of focus when the tracking information of subject detection AF is added without changing the AF mode. In addition, the camera CPU 121 calculates the degree of focus by swapping various conditions and selects the setting combination that yields the highest degree of focus.

[0279] In the example shown in Figure 32, it is assumed that the following conditions resulted in an increase in the degree of focus, as evaluated in S1110. (1) Keep the AF frame as 1 point and change the subject detection AF tracking to ON. (2) The shutter type will be changed to an electronic shutter. (3) Change the angle of view to wide angle

[0280] Based on the evaluation results of S1110 described above, the best information can be proposed to the photographer. Display 32003 displays the best information for the above-mentioned factors.

[0281] FIG. 32(c) shows a display 32004 for allowing the photographer to select whether to change the tracking regarding the subject detection of the shooting-related information of the best setting proposal of FIG. 32(b) described above. The photographer can check the display 32004 and make a setting that can improve the focusing degree.

[0282] Next, although not shown in the figure, the camera CPU 121 similarly displays information for allowing the photographer to select whether to change the shutter method. In the case of shooting in the real space, since the lens angle of view cannot be selected, the camera CPU 121 advises the change of the focal length by displaying information to the photographer. In the case of shooting in the virtual space, since the focal length can be virtually changed, similar to FIG. 32(c), the camera CPU 121 displays a display regarding the change and prompts the photographer to make a selection.

[0283] Note that the camera CPU 121 may display various settings together for selection as a display method for these setting changes. Also, the camera CPU 121 may display only the focusing degree and collectively change to the settings for which the focusing degree selected by the photographer is calculated.

[0284] Here, the display for prompting the photographer to change the settings has been described, but the above settings may be automatically changed by the camera.

[0285] FIG. 32(d) shows the evaluation results of the shooting reflecting the setting change in FIG. 32(c). The AF frame 32001 has changed to a dotted line which is the frame of the tracking AF, and subject detection is being performed. As a result, by constantly following the subject, the photographer can focus on framing.

[0286] By changing the shutter system to an electronic shutter, there is no blackout, and the subject remains visible in the viewfinder at all times, reducing the likelihood of losing sight of the subject. The lens has been changed to a wide-angle 70mm, which provides more leeway in the sense of the subject and angle of view, thus reducing instances of the subject going out of frame. Figure 32(d) shows that, as a result of changing to the best settings, the degree of focus has improved to 85%, a significant improvement from the previous 70%.

[0287] By evaluating a series of captured images and quantifying the degree of focus, photographers can understand their framing skills. Analyzing the causes of poor focus and displaying the optimal settings can help improve photographers' framing techniques.

[0288] Figure 32 illustrates real-world space photography, but, as with Figure 31, the optimal settings may be displayed and the shooting settings may be changed or automatically changed by evaluating the degree of focus from the results obtained from photography in a virtual space environment using an external computing device 1000, not just in real space.

[0289] ●Summary of the first embodiment As described above, according to the first embodiment, the imaging system 10, which includes an image processing system, generates a virtual space containing virtual objects. The imaging system 10 also generates a virtual space image corresponding to a predetermined shooting range in the virtual space using information that defines the focus state for virtual shooting in the virtual space (for example, S2009 in Figure 14). The imaging system 10 also calculates a virtual defocus amount (first virtual defocus amount) in the virtual space image (for example, S4009 in Figure 15). The imaging system 10 then generates a second virtual defocus amount by adding an error amount based on one or more parameters that affect the error in the defocus amount that occurs in shooting in the real space to the first virtual defocus amount (for example, Figure 20).

[0290] Therefore, according to the first embodiment, it is possible to add an error amount to the virtual defocus amount of the virtual space image, taking into account the error in the amount of defocus that may occur in the shooting of the real space.

[0291] ● Variation In the first embodiment, a configuration was described in which the detection of the focus region is achieved by region detection based on machine learning. However, the method for detecting the focus region is not limited to this. For example, the focus region can be set using the aspect ratio of the subject detection region, the size of the subject detection region, or depth information of the subject using a defocus map.

[0292] [Second Embodiment] Next, a second embodiment will be described. In this embodiment, in the virtual space image generation and output processing, images captured in the real space are also used to generate and output images in the virtual space. The basic configuration of the imaging system 10 in this embodiment is the same as in the first embodiment, but there is a difference in part of the process of generating and outputting images in the virtual space (S2000 in Figure 13). The following will mainly describe the differences from the first embodiment.

[0293] Figure 34 is a flowchart of the subroutine for the process (S2000 in Figure 13) that generates and outputs video of a virtual space, according to the second embodiment.

[0294] In S3501, CPU1001 acquires images of the real world. These images may be those captured in the aforementioned real-world imaging process, or they may be images that have been captured in advance.

[0295] In S3502, the foreground object acquisition unit 1102 acquires and synthesizes foreground objects. The foreground object acquisition unit 1102 may use a trained model that estimates a 3D model for an image to generate a 3D model of the subject from a captured image in real space and synthesize it with the 3D model of the subject in the virtual space described above. For example, the foreground object acquisition unit 1102 may acquire the face as a foreground object from a captured image in real space, and other parts such as the torso may be acquired from the foreground object storage unit 1101 of the virtual space reproduction device 1100. Alternatively, the foreground object acquisition unit 1102 may acquire the 3D model of the subject from the captured image and the 3D model of the subject in the virtual space described above alternately in a time series and display them at different timings as a display image in the virtual space described later. Furthermore, the foreground object acquisition unit 1102 may acquire foreground objects in the virtual space and synthesize them with a captured image in real space at the stage of generating the display image in the virtual space in S3505 described later.

[0296] In S3503, the background object acquisition unit 1105 acquires and combines background objects. Similar to the foreground object acquisition in S3502, the background object acquisition unit 1105 may generate a 3D model from the captured image and use it as a background object, or it may combine it with the background object in the virtual space, separating the regions. Alternatively, the background object acquisition unit 1105 may separate the background object from the captured image and the background object in the virtual space in chronological order. Furthermore, the background object acquisition unit 1105 may acquire the background object in the virtual space and combine it with the captured image in the real space during the virtual space display image generation stage in S3505, which will be described later.

[0297] In S3505, the display image generation unit 1204 generates a display image in the virtual space. If the captured image has already been composited with the foreground and background objects, S3505 performs the same processing as in S2008 of the first embodiment. Otherwise, the display image generation unit 1204 generates a display image by aligning and compositing the captured image in the real space with the display image generated by the foreground and background objects in the virtual space. For example, the display image generation unit 1204 cuts out only the face portion of the subject in the captured image and composites it with the display image in the virtual space.

[0298] Furthermore, if the range and viewpoint position differ due to differences in the field of view of the captured image, the display image generation unit 1204 generates a 3D model from the captured image using a pre-trained model that estimates a 3D model for the image in advance. Then, the display image generation unit 1204 generates a display image by changing the range and viewpoint position, and generates a display image in the virtual space that includes the captured image by dividing the virtual space display image into regions and combining them.

[0299] Thus, in the video generation and output processing of virtual space, it is possible to generate and output virtual space video using captured images from the real world. Furthermore, the technology described in the first embodiment can be similarly applied to the virtual space according to the second embodiment.

[0300] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0301] [summary] The embodiments described above disclose, but are not limited to, the inventions shown in at least the following items. (Item 1) A generation means for generating a virtual space containing virtual objects, A first generation means generates a virtual space image corresponding to a predetermined shooting range in the virtual space using information defining the focus state for virtual shooting in the virtual space, A calculation means for calculating a first virtual defocus amount in the virtual space image, A second generation means generates a second virtual defocus amount by adding an error amount based on one or more parameters that affect the error in the defocus amount that occurs in real space photography to the first virtual defocus amount, An image processing system characterized by comprising the following features. (Item 2) The calculation means calculates the first virtual defocus amount for the region of the subject in the virtual space image. The image processing system described in item 1, characterized by the features described herein. (Item 3) The one or more parameters mentioned above include the contrast value of the subject. The image processing system described in item 2, characterized by the features described herein. (Item 4) The one or more parameters mentioned above include at least one of camera information, lens information, and shooting setting information. An image processing system according to any one of items 1 to 3, characterized in that it is the same as described in item 1 to 3. (Item 5) A storage means for storing one or more of the aforementioned parameters, An image processing system according to any one of items 1 to 4, further comprising the above. (Item 6) Adjustment means for adjusting the focus state based on the second virtual defocus amount, An image processing system according to any one of items 1 to 5, further comprising the following: (Item 7) A switching means that switches whether the adjustment means adjusts the focus state based on the second virtual defocus amount or the first virtual defocus amount. The image processing system according to item 6, further comprising the following: (Item 8) An image processing system as described in any one of items 1 through 7, Imaging device and An imaging system characterized by comprising the following features. (Item 9) A control method performed by an image processing system, A generation process that generates a virtual space containing virtual objects, A first generation step of generating a virtual space image corresponding to a predetermined shooting range in the virtual space using information defining the focus state for virtual shooting in the virtual space, A calculation step for calculating a first virtual defocus amount in the virtual space image, A second generation step involves generating a second virtual defocus amount by adding an error amount based on one or more parameters that affect the error in the defocus amount that occurs in real space photography to the first virtual defocus amount, A control method characterized by comprising: (Item 10) A program for causing a computer to function as one of the means of an image processing system as described in any one of items 1 through 7.

[0302] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0303] 10...Imaging system, 100...Camera, 1000...External computing device, 1100...Virtual space reproduction device, 1200...Virtual image generation device, 2000...Camera / lens information storage device

Claims

1. A generation means for generating a virtual space containing virtual objects, A first generation means generates a virtual space image corresponding to a predetermined shooting range in the virtual space using information defining the focus state for virtual shooting in the virtual space, A calculation means for calculating a first virtual defocus amount in the virtual space image, A second generation means generates a second virtual defocus amount by adding an error amount based on one or more parameters that affect the error in the defocus amount that occurs in the real space to the first virtual defocus amount, An image processing system characterized by comprising the following features.

2. The calculation means calculates the first virtual defocus amount for the region of the subject in the virtual space image. The image processing system according to feature 1.

3. The one or more parameters mentioned above include the contrast value of the subject. The image processing system according to claim 2, characterized in that it is as described above.

4. The one or more parameters mentioned above include at least one of camera information, lens information, and shooting setting information. The image processing system according to feature 1.

5. A storage means for storing one or more of the above parameters, The image processing system according to claim 1, further comprising the following:

6. Adjustment means for adjusting the focus state based on the second virtual defocus amount, The image processing system according to claim 1, further comprising the following:

7. A switching means that switches whether the adjustment means adjusts the focus state based on the second virtual defocus amount or the first virtual defocus amount. The image processing system according to claim 6, further comprising the above.

8. An image processing system according to any one of claims 1 to 7, Imaging device and An imaging system characterized by comprising the following features.

9. A control method performed by an image processing system, A generation process that generates a virtual space containing virtual objects, A first generation step of generating a virtual space image corresponding to a predetermined shooting range in the virtual space using information defining the focus state for virtual shooting in the virtual space, A calculation step for calculating a first virtual defocus amount in the virtual space image, A second generation step involves generating a second virtual defocus amount by adding an error amount based on one or more parameters that affect the error in the defocus amount that occurs in the real space to the first virtual defocus amount, A control method characterized by comprising:

10. A program for causing a computer to function as one of the means of the image processing system described in any one of claims 1 to 7.

Images