Image processing method, image processing device, and image processing system

Abstract

An image processing method according to the present technology controls display of a virtual object to be composited with a captured image captured by an imaging device on the basis of a control condition.

Description

Image processing method, image processing device, and image processing system

[0001] The present technology relates to an image processing method, an image processing device, and an image processing system, and in particular to a technology for superimposing a CG image on a captured image.

[0002] For example, as disclosed in Patent Document 1, a system has been proposed in which an image captured by a camera and a CG image generated by 3DCG (Three Dimensional Computer Graphics) processing are combined in real time, and the combined image (so-called AR image) is displayed on a camera monitor. In this system, the combined image can be displayed on the camera monitor in real time, so that the cameraman can check the combined image in real time and can operate the camera while viewing the combined image.

[0003] JP 2011-35638 A

[0004] In recent years, with the advancement of CG processing technology, it has become difficult to distinguish between real objects depicted in captured images and virtual objects depicted in CG images. Therefore, with the technology described in Patent Document 1, it is difficult to distinguish between real objects and virtual objects, which may confuse the photographer (user).

[0005] The present technology has been made in view of the above circumstances, and aims to improve the convenience of composite images for users when capturing images.

[0006] The image processing method according to the present technology controls the display of a virtual object to be combined with an image captured by an imaging device based on a control condition. According to the image processing method, it is possible to control the display of the virtual object or change the display mode of the virtual object depending on whether or not a predetermined control condition is satisfied.

[0007] 1 is a diagram showing an overview of an image processing system according to an embodiment. FIG. 1 is a diagram illustrating a captured image, a CG image, and a composite image. FIG. 2 is a diagram illustrating the configuration of an imaging device. FIG. 3 is a block diagram illustrating an example configuration of an image processing device. FIG. 4 is a diagram illustrating the functional configuration of an image processing device. FIG. 5 is a diagram illustrating modes. FIG. 6 is a diagram illustrating a display mode of a virtual object. FIG. 7 is a diagram illustrating a display mode of a virtual object. FIG. 8 is a diagram illustrating display and non-display of a virtual object in manual mode. FIG. 9 is a diagram illustrating control of the display mode of a virtual object in manual mode. FIG. 10 is a diagram illustrating a change in transparency of a virtual object in manual mode. FIG. 11 is a diagram illustrating an example of control conditions in a camera position and orientation mode. FIG. 12 is a diagram illustrating another example of control conditions in a camera position and orientation mode. FIG. 13 is a diagram illustrating a change in transparency of a virtual object when a change in the position or orientation of the imaging device is set as a control condition in the camera position and orientation mode. FIG. 14 is a diagram illustrating display control of a virtual object when a distance between the imaging device and the virtual object is within a predetermined threshold as a control condition. FIG. 15 is a diagram illustrating a change in transparency of a virtual object when a distance between the imaging device and the virtual object is within a predetermined threshold as a control condition. FIG. 16 is a diagram illustrating depth of field. FIG. 17 is a diagram illustrating control of display and non-display of a virtual object in an imaging control mode. 10 is a diagram showing a change in transparency of a virtual object when a range other than the depth of field is determined as the display control range in the imaging control mode. FIG. 11 is a diagram showing an example of control conditions in the virtual object mode. FIG. 12 is a diagram showing display control of a virtual object when the control condition is set that a virtual object has entered a control switching area. FIG. 13 is a diagram showing a change in transparency of a virtual object when the control condition is set that a virtual object has entered a control switching area. FIG. 14 is a diagram explaining display control of a virtual object in the real object mode. FIG. 15 is a diagram explaining a change in transparency of a virtual object in the real object mode. FIG. 16 is a diagram showing an example of an icon indicating a mode. FIG. 17 is a diagram showing an example of an icon indicating a display mode. FIG. 18 is a diagram showing an example of an icon indicating the passage of time in a change in transparency. FIG. 19 is a diagram showing an icon during special processing.FIG. 1 is a diagram showing an example of a composite image for a monitor on which an icon is displayed. FIG. 2 is a flowchart showing a processing flow for display control of a virtual object 6. FIG. 3 is a flowchart showing a processing flow for display control for each set mode. FIG. 4 is a diagram showing a modified example of the configuration of an image processing system. FIG. 5 is a diagram showing a modified example of the configuration of an image processing system.

[0008] Hereinafter, with reference to the accompanying drawings, embodiments of a sensor device according to the present technology will be described in the following order. <1. Overview of Image Processing System> <2. Description of Modes> <3. Icons Displayed in a Composite Image for Monitor> <4. Processing Flow for Controlling Display of Virtual Objects> <5. Modifications> <6. Summary of Embodiments> <7. The Present Technology> Note that, in the present technology, "imaging" not only refers to imaging accompanied by recording of image data, but also includes imaging for displaying an image on a display unit without recording image data, such as a so-called through image or live view image. "Image" refers to a moving image. Furthermore, "image" does not only refer to a state in which the image is displayed on a display unit, but may also refer to image data that is not displayed on a display unit. "Real object" refers not only to an object (subject) captured by an imaging device, but also includes the image of the subject captured in the image. "Virtual object" refers not only to a virtual object generated by 3DCG processing, but also includes a virtual object captured in a CG image generated by rendering.

[0009] <1. Overview of Image Processing System> [1.1. Overall Configuration of Image Processing System] Fig. 1 is a diagram showing an overview of an image processing system 1 according to an embodiment. Fig. 2 is a diagram explaining a captured image, a CG image, and a composite image. As shown in Fig. 1, the image processing system 1 according to an embodiment is configured to include an image capturing device 2 and an image processing device 3.

[0010] The imaging device 2 captures an image of a real object 5 (a person in this case) as a subject in response to an operation by a cameraman 4, thereby generating a captured image (moving image) 101 as shown in FIG. 2A.

[0011] The image processing device 3 is configured by, for example, a workstation. The image processing device 3 may be located at the imaging site or at a location different from the imaging site. The image processing device 3 is connected to the imaging device 2 wirelessly or via a wire, and receives the captured image 101 from the imaging device 2 and transmits the composite image 103 to the imaging device 2.

[0012] The image processing device 3 generates a CG image 102 containing a virtual object 6 (here, a car) as shown in Fig. 2B. The image processing device 3 then composites the CG image 102 (virtual object 6) with the captured image 101 to generate a composite image 103 as shown in Fig. 2C. Note that the composite image 103 may also include frames in which the CG image 102 is not composited. The image processing device 3 then transmits the generated composite image 103 to, for example, one or both of the image capture device 2 and the monitor 7.

[0013] When the imaging device 2 receives the composite image 103 from the image processing device 3, it displays the received composite image 103 on the display unit 15. When the monitor 7 receives the composite image 103 from the image processing device 3, it displays the received composite image 103. As a result, in the image processing system 1, it becomes possible for the cameraman 4 to check the composite image 103 as a moving image in almost real time while the imaging device 2 is capturing an image, that is, an image in which the virtual object 6 is superimposed on the captured image 101.

[0014] [1.2. Overall Configuration of Imaging Device] Fig. 3 is a diagram showing the configuration of the imaging device 2. As shown in Fig. 3, the imaging device 2 includes a lens system 11, an imaging unit 12, an image processing unit 13, a recording unit 14, a display unit 15, a camera control unit 16, a memory unit 17, a driver unit 18, a sensor unit 19, a communication unit 20, and an operation unit 21. Note that, hereinafter, all of the units constituting the imaging device 2 except for the display unit 15 will be collectively referred to as a main body unit 2A.

[0015] The lens system 11 includes lenses such as a zoom lens and a focus lens, an aperture mechanism, and the like. The lens system 11 also includes a motor for operating these lenses and the aperture mechanism. The lens system 11 focuses light (incident light) from the real object 5 onto the imaging unit 12. The lens system 11 may be provided integrally with the imaging device 2, or may be configured as an interchangeable lens separate from the imaging device 2.

[0016] The imaging unit 12 includes an image sensor (imaging element), such as a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type. The imaging unit 12 performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc. on the electrical signal obtained by photoelectrically converting light received by the image sensor, and further performs A / D (Analog / Digital) conversion processing. The imaging unit 12 then outputs the captured image signal as digital data to the image processing unit 13.

[0017] The image processing unit 13 is configured as an image processor, for example, a DSP (Digital Signal Processor), etc. The image processing unit 13 performs various signal processing on the captured image signal from the imaging unit 12. The image processing unit 13 performs, for example, pre-processing, synchronization processing, YC generation processing, resolution conversion processing, file formation processing, etc.

[0018] In the pre-processing, the captured image signal from the imaging unit 12 is subjected to clamping processing, which clamps the R, G, and B black levels to a predetermined level, and correction processing between the R, G, and B color channels. In the synchronization processing, a color separation processing is performed so that the image data for each pixel contains all R, G, and B color components. For example, in the case of an image sensor using a Bayer array color filter, demosaic processing is performed as the color separation processing. In the YC generation processing, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data. In the resolution conversion processing, a resolution conversion process is performed on the image data that has been subjected to various signal processing.

[0019] In the file creation process, the image data that has undergone the various processes described above is subjected to compression encoding for recording or communication, formatting, and the generation and addition of metadata to generate files for recording or communication. For example, an image file is generated in the MP4 format used for recording moving images and audio that conforms to MPEG-4. It is also possible to generate an image file as raw image data.

[0020] The recording unit 14 is a memory card (such as a portable flash memory) that is a recording medium that can be attached to or detached from the imaging device 2, or a flash memory or HDD (hard disk drive) that is built into the imaging device 2. The recording unit 14 records the image data output from the image processing unit 13.

[0021] The display unit 15 executes various displays on the display screen based on instructions from the camera control unit 16. For example, the display unit 15 displays a playback image of image data read from the recording unit 14. The display unit 15 also displays a composite image 103 transmitted from the image processing device 3 in accordance with instructions from the camera control unit 16. As a result, the composite image 103 during composition confirmation, video recording, etc. is displayed on the display unit 15 as a so-called through image. The display unit 15 may also execute displays on the screen of various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the camera control unit 16.

[0022] The camera control unit 16 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The memory unit 17 stores information used by the camera control unit 16 for processing. The memory unit 17 collectively refers to, for example, a read-only memory (ROM), a random access memory (RAM), a flash memory, and the like. The memory unit 17 may be a memory area built into the microcomputer chip that serves as the camera control unit 16, or may be configured by a separate memory chip. The camera control unit 16 controls the entire imaging device 2 by executing a program stored in the ROM, flash memory, or the like of the memory unit 17. The camera control unit 16 includes functional units serving as an imaging control unit 16a and a self-position and orientation estimation unit 16b.

[0023] The imaging control unit 16a controls the operations of various units required for imaging, such as controlling the shutter speed of the imaging unit 12, issuing instructions for various signal processing in the image processing unit 13, imaging and recording operations in response to operations by the cameraman 4, playback of recorded image data, and control operations of the lens system 11 such as zooming, focusing, and aperture adjustment. The imaging control unit 16a also calculates the angle of view, aperture value, lens distortion, and focal length of the imaging device 2 as lens information based on the specifications of each lens and aperture mechanism constituting the lens system 11, movement of the focus lens and zoom lens, and operation information such as opening and closing of the aperture mechanism. Note that known methods can be used to calculate this lens information, and therefore a description thereof will be omitted.

[0024] The self-position and orientation estimation unit 16b estimates the position and orientation of the imaging device 2 based on detection information (in this example, triaxial acceleration information and angular velocity information) from the sensor unit 19 and images captured by the imaging unit 12. Examples of position and orientation estimation methods include SLAM (Simultaneous Localization and Mapping) and VIO (Visual Inertial Odometry). For example, VIO is assumed to estimate the position and orientation using a technology such as an INS (Inertial Navigation System) using the output of an IMU (sensor unit 19) that has a higher output rate than the image sensor (imaging unit 12). Note that methods such as VO (Visual Odometry) that do not use IMU detection information may also be employed.

[0025] The RAM in the memory unit 17 is used as a working area for temporarily storing data, programs, etc. when the CPU of the camera control unit 16 processes various data. The ROM and flash memory (non-volatile memory) in the memory unit 17 are used to store an OS (Operating System) for the CPU to control each unit, application programs for various operations, firmware, various setting information, etc. The various setting information includes exposure settings, shutter speed settings, and mode settings as setting information related to imaging operations, white balance settings, color settings, and settings related to image effects as setting information related to image processing, and custom key settings and display settings as setting information related to operability.

[0026] The driver unit 18 is provided with, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for an aperture mechanism motor, etc. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 16, and cause the focus lens or zoom lens to move, open or close the aperture blades of the aperture mechanism, etc.

[0027] The sensor unit 19 collectively refers to various sensors mounted on the imaging device 2. The sensor unit 19 may be, for example, an IMU (Inertial Measurement Unit). The IMU has a triaxial acceleration sensor and an angular velocity sensor, and outputs triaxial acceleration information and angular velocity information.

[0028] The communication unit 20 is capable of performing wired or wireless inter-device communication and network communication, which is communication with an external device via a predetermined communication network such as the Internet. The communication unit 20 transmits the captured image 101 to the image processing device 3, for example, and receives the composite image 103.

[0029] The operation unit 21 collectively represents input devices used by the user (cameraman 4) to input various operations. Specifically, the operation unit 21 includes various operators (keys, dials, touch panels), a touch panel, etc. When the operation unit 21 detects a user operation, a signal corresponding to the input operation is sent to the camera control unit 16.

[0030] [1.3. Configuration of Image Processing Device] Fig. 4 is a block diagram showing an example configuration of the image processing device 3. As shown in Fig. 4, the image processing device 3 includes a CPU 31. The CPU 31 executes various processes in accordance with a program stored in a ROM 32 or a program loaded from a storage unit 39 to a RAM 33. The RAM 33 also stores data necessary for the CPU 31 to execute various processes as appropriate. The CPU 31, the ROM 32, and the RAM 33 are connected to one another via a bus 43.

[0031] A GPU (Graphics Processing Unit) 34 is also connected to the bus 43. The GPU 34 performs 3DCG processing mainly in real time in accordance with drawing commands from the CPU 31.

[0032] An input / output interface (I / F) 35 is also connected to the bus 43. An input unit 36 ​​including controls and operation devices is connected to the input / output interface 35. For example, the input unit 36 ​​may be various controls and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. An operation is detected by the input unit 36, and a signal corresponding to the detected operation is interpreted by the CPU 31.

[0033] A display unit 37, such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 38, such as a speaker, are also connected to the input / output interface 35, either integrally or separately. The display unit 37 is used to display various types of information, and may be configured, for example, by a display device provided in the housing of the image processing device 3 or a separate display device connected to the image processing device 3. The display unit 37 may be the monitor 7.

[0034] The display unit 37 displays images for various image processes, moving images to be processed, etc. on the display screen based on instructions from the CPU 31. The display unit 37 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 31.

[0035] The input / output interface 35 is connected to a storage unit 39 and a communication unit 40, which are configured from an SSD (Solid State Drive) or an HDD (Hard Disk Drive).

[0036] The communication unit 40 is capable of performing wired or wireless inter-device communication, and network communication, which is communication with an external device via a predetermined communication network such as the Internet. In particular, in this embodiment, the communication unit 40 is configured to be able to perform data communication with the communication unit 20 of the imaging device 2.

[0037] A drive 41 is connected to the input / output interface 35 as required, and a removable recording medium 42 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon.

[0038] The drive 41 can read data files such as programs used in various processes from the removable recording medium 42. The read data files are stored in the storage unit 39, and images and sounds contained in the data files are output on the display unit 37 and the audio output unit 38. Furthermore, the computer programs and the like read from the removable recording medium 42 are installed in the storage unit 39 as needed.

[0039] 5 is a diagram showing the functional configuration of the image processing device 3. Note that FIG. 5 also shows the configurations of the imaging device 2 and the monitor 7.

[0040] 5, the CPU 31 and GPU 34 of the image processing device 3 function as a CG generation unit 51, a first composition unit 52, an image recognition unit 53, a condition determination unit 54, an image processing unit 55, and a second composition unit 56. These functional units may be configured to function by only one of the CPU 31 and the GPU 34. Furthermore, one functional unit may be configured to function by both the CPU 31 and the GPU 34.

[0041] In the image processing system 1, the captured image 101 captured by the imaging unit 12 of the imaging device 2 (more precisely, the captured image 101 subjected to image processing by the image processing unit 13) is transmitted to the image processing device 3 via the communication unit 20.

[0042] Furthermore, the imaging device 2 transmits camera information 110 obtained while capturing the captured image 101 to the image processing device 3 via the communication unit 20. The camera information 110 may be metadata assigned to the captured image 101, or may be information independent of the captured image 101. The camera information 110 includes the position, orientation, angle of view, aperture value, lens distortion, focal length, etc. of the imaging device 2. This information is transmitted, for example, for each frame constituting the captured image 101. The position and orientation of the imaging device 2 are estimated for each frame by the self-position and orientation estimation unit 16b. Furthermore, the angle of view, aperture value, lens distortion, and focal length of the imaging device 2 are calculated by the imaging control unit 16a as lens information, as described above.

[0043] When the image processing device 3 receives the captured image 101 and camera information 110, it inputs the captured image 101 to the first synthesis unit 52, the image recognition unit 53, and the second synthesis unit 56, and inputs the camera information 110 to the CG generation unit 51 and the condition determination unit 54.

[0044] The CG generation unit 51 includes a vertex shader unit 51 a, a first pixel shader unit 51 b, and a second pixel shader unit 51 c. However, the CG generation unit 51 is not limited to this, and may have other configurations as long as it can generate the CG image 102 from a 3D model.

[0045] Upon receiving the camera information 110, the vertex shader unit 51a sets the angle of view, aperture value, lens distortion, and focal length of a virtual camera that virtually captures an image in a virtual space represented by a CG model, based on the angle of view, aperture value, lens distortion, and focal length included in the camera information 110. Here, the settings of the lens system of the virtual camera are matched to the settings of the lens system 11 of the imaging device 2.

[0046] Furthermore, the vertex shader unit 51a updates the position and orientation of the virtual camera based on the position and orientation of the image capture device 2 included in the camera information 110. Here, the CG model indicates the position (coordinate position) of the virtual object 6 in a coordinate system based on a predetermined position in the virtual space. Furthermore, the coordinate system of the real space captured by the image capture device 2 is adjusted to match the coordinate system of the virtual space indicated by the CG model. Therefore, by matching the position and orientation of the virtual camera to the position and orientation of the image capture device 2, the imaging range of the imaging device 2 and the imaging range of the virtual camera can be matched. Next, the vertex shader unit 51a performs processing such as converting the shape of the virtual object 6 when viewed from the updated position and orientation of the virtual camera.

[0047] The first pixel shader unit 51b performs processing (shading) to determine the final color for a pixel. The shading processing includes processes such as texture mapping and lighting. In addition, various other processes are performed as fragment-based processing. The first pixel shader unit 51b applies pixel shader processing to generate a CG image 102 in which the virtual object 6 appears. The generated CG image 102 is output to the first composition unit 52.

[0048] The first composition unit 52 composes the captured image 101 transmitted from the imaging device 2 with the CG image 102 generated by the CG generation unit 51, thereby generating a composite image 103 in which the virtual object 6 is superimposed on the captured image 101. Note that, in the present embodiment, an example is described in which the virtual object 6 is located closer to the viewer than the real object 5, but the virtual object 6 may also be located further back than the real object 5. In such a case, the captured image 101 may include information on the distance to the real object 5 for each pixel, and the CG image 102 may include information on the distance to the virtual object 6 for each pixel, and the composite image 103 may be generated so that the pixels on the closer side are preferentially left when combining the images.

[0049] In this way, the image processing system 1 generates in real time a composite image 103 in which the virtual object 6 is superimposed on the captured image 101. The composite image 103 generated here is stored in the storage unit 39 as, for example, the composite image 103 to be ultimately retained.

[0050] As described above, the cameraman 4 captures the real object 5 while checking the captured image displayed on the display unit 15 or the monitor 7. Therefore, if only the captured image 101 without the CG image 102 is displayed on the display unit 15 or the monitor 7, it is difficult to check where and how the CG image 102 is actually displayed, whether the focus of the imaging device 2 is on the virtual object 6, and so on.

[0051] Therefore, in the image processing system 1, a composite image 103 in which a CG image 102 (virtual object 6) is combined with a captured image 101 is displayed as a through image on the display unit 15 and the monitor 7, making it easier for the cameraman 4 to understand the situation during the capture.

[0052] However, if the composite image 103 generated as described above, i.e., the virtual object 6, is superimposed as is on the captured image 101, the virtual object 6 may be displayed over most of the display unit 15, which may make it difficult to understand the surrounding situation. In addition, the cameraman 4 may become confused during image capture because he or she may not be able to distinguish between the real object 5 and the virtual object 6.

[0053] Therefore, the image processing system 1 controls the display of the virtual object 6 based on predetermined control conditions. The control of the display of the virtual object 6 includes control of whether to display or hide the virtual object 6, and control of the display mode of the virtual object 6. In the following, the composite image 103 displayed on the display unit 15 and the monitor 7 will be described as a monitor composite image 104.

[0054] First, a description will be given of the functional units for generating the composite image for monitor 104. However, it goes without saying that the functional units described above also function as functional units for generating the composite image for monitor 104.

[0055] The second pixel shader unit 51c performs processing (shading) to determine the final color for a pixel. Unlike the first pixel shader unit 51b, the second pixel shader unit 51c performs semi-transparency processing for pixels. In the semi-transparency processing, the transparency of each pixel is changed to a specified transparency. Note that only one of the first pixel shader unit 51b and the second pixel shader unit 51c may be provided. In this case, it is sufficient that the one pixel shader unit provided performs both processes.

[0056] The CG image 102 generated by the first pixel shader section 51b and the second pixel shader section 51c is output to the image processing section 55 and the second composition section 56 as appropriate.

[0057] The image recognition unit 53 performs an object recognition process to recognize a predetermined real object 5 from the captured image 101. Possible types of the real object 5 to be recognized include a person, a vehicle, etc. Note that the object recognition process can use a known method, and therefore a description thereof will be omitted here.

[0058] The condition determination unit 54 determines whether or not a control condition, which will be described in detail later, is satisfied based on the camera information 110 received from the imaging device 2 and the recognition result of the real object 5 by the image recognition unit 53. The condition determination unit 54 outputs whether or not the control condition is satisfied to the CG generation unit 51, the image processing unit 55, and the second composition unit 56 as appropriate.

[0059] The image processing unit 55 performs two-dimensional processing of the virtual object 6 (CG image 102) in accordance with a mode, a display mode, etc., which will be described in detail later, based on whether or not the control condition is satisfied. The processing will be described in detail later. After performing the processing on the virtual object 6, the image processing unit 55 outputs the CG image 102, in which the processed virtual object 6 is shown, to the second composition unit 56.

[0060] The second composition unit 56 generates a composite image for monitoring 104 by combining the captured image 101 transmitted from the imaging device 2 with the CG image 102 generated by the CG generation unit 51 or the CG image 102 processed by the image processing unit 55. The generated composite image for monitoring 104 is output to the imaging device 2 and the monitor 7. The display unit 15 of the imaging device 2 and the monitor 7 then display the combined composite image for monitoring 104.

[0061] 6 is a diagram for explaining modes. As shown in Fig. 6, the image processing system 1 has modes for controlling the display of the virtual object 6, including a manual mode, a camera position and orientation mode, an imaging control mode, a virtual object mode, and a real object mode. Note that the image processing system 1 may have any one or more of these modes.

[0062] The manual mode is a mode in which the display of the virtual object 6 is controlled in accordance with, for example, an operation on the operation unit 21 by the cameraman 4 (user). Here, the operation by the cameraman 4 is described, but the operation may also be performed by a person other than the cameraman 4. The camera position and orientation mode is a mode in which control conditions are set based on the position and orientation of the imaging device 2. The imaging control mode is a mode in which control conditions are set based on imaging information (angle of view, aperture, lens distortion, focal length) of the imaging device 2. The virtual object mode is a mode in which control conditions are set based on the virtual object 6. The real object mode is a mode in which control conditions are set based on the real object 5.

[0063] In the image processing system 1, one or more of these five modes are set, and the display of the virtual object 6 is controlled according to the set mode. In each mode, the display of the virtual object 6 can be controlled by controlling whether the virtual object 6 is displayed or not (whether or not to display), controlling the display mode of the virtual object 6, and changing the transparency of the virtual object 6. The display control performed from among these may be determined in advance, may be switched by operating the operation unit 21, or may be determined by attributes added to the virtual object 6.

[0064] [2.1. Display Modes of Virtual Object 6] Here, the display modes of the virtual object 6 include a normal display mode, an outline highlight display mode, a semi-transparent display mode, a mesh display mode, and an outline display mode. Therefore, the display modes of the virtual object 6 will be explained first, and then each mode will be explained in detail.

[0065] 7 and 8 are diagrams showing the display mode of the virtual object 6. For convenience of explanation, the real object 5 is shown together with the virtual object 6 in FIGS.

[0066] 7A , the normal display mode is a display mode in which the virtual object 6 output to the first composition unit 52, i.e., the virtual object 6 that will ultimately be superimposed on the captured image 101, is displayed as is. Therefore, when the virtual object 6 is placed in front of the real object 5 (on the side of the image capture device 2), the portion of the real object 5 that overlaps with the virtual object 6 becomes invisible in the composite image for monitoring 104. This normal display mode allows the cameraman 4 to view the composite image for monitoring 104 that is identical to the composite image 103 that will ultimately be created.

[0067] As shown in FIG. 7B , the contour-enhanced display mode is a display mode in which the contour of the virtual object 6 is emphasized. In the contour-enhanced display mode, the image processing unit 55 extracts the contour of the virtual object 6 and performs processing such as displaying the extracted contour as a thick line or in a different color. Therefore, when the virtual object 6 is placed in front of the real object 5 (toward the image capture device 2), the portion of the real object 5 that overlaps with the virtual object 6 becomes invisible in the monitor composite image 104, as in the normal display mode. In this contour-enhanced display mode, the contour of the virtual object 6 is emphasized, allowing the cameraman 4 to easily identify the virtual object 6. Furthermore, the virtual object 6 and the real object 5 can be easily distinguished from each other.

[0068] 7C , the semi-transparent display mode is a display mode in which a semi-transparent virtual object 6 generated by the first pixel shader unit 51b is displayed. Therefore, even if the virtual object 6 is placed in front of the real object 5 (on the imaging device 2 side), the portion of the real object 5 that overlaps with the virtual object 6 is visible through the virtual object 6 in the composite image for monitor 104. In this semi-transparent display mode, the virtual object 6 is displayed semi-transparently, allowing the cameraman 4 to easily identify the virtual object 6. It also allows the cameraman 4 to confirm the real object 5 behind the virtual object 6.

[0069] As shown in FIG. 8A , the mesh display mode is a display mode in which the virtual object 6 is displayed in a mesh shape. When meshing the virtual object 6, a 3D virtual object 6 may be meshed, or a 2D virtual object 6 may be meshed. When meshing the 2D virtual object 6, horizontal stripes may be used as shown in FIG. 8A , or vertical or diagonal stripes may be used. In the horizontal stripe mesh display mode, the image processing unit 55 divides the 2D virtual object 6 vertically at predetermined widths and performs processing to thin out every other divided portion. In this mesh display mode, the virtual object 6 can be roughly recognized, and since the rear sides appear to be alternately hollow in the vertical direction, it is possible to confirm the real object 5 behind the virtual object 6. Furthermore, since the horizontal stripe mesh display mode processes the 2D virtual object 6, it is also possible to reduce the processing load.

[0070] As shown in FIG. 8B , the contour display mode is a display mode in which only the contour of the virtual object 6 is displayed. In the contour display mode, the image processing unit 55 performs processing to extract only the contour of the virtual object 6. Therefore, even if the virtual object 6 is placed in front of the real object 5, in the composite image for monitor 104, the portion of the real object 5 that overlaps with the virtual object 6 appears completely transparent to the virtual object 6. In this contour display mode, only the contour of the virtual object 6 is displayed, allowing the cameraman 4 to easily identify the position and shape of the virtual object 6. This also allows the virtual object 6 to be easily distinguished from the real object 5. Furthermore, the rear side of the virtual object 6 can be easily confirmed.

[0071] Although the image processing system 1 is configured to provide these five display modes, it is sufficient that any one or more of these display modes is provided.

[0072] [2.2. Manual Mode] In the manual mode, the entire range is determined as the display control range of the virtual object 6. The display control range indicates the range on the captured image 101 where display control of the virtual object 6 is performed. When the entire range is determined as the display control range, the display control of the entire range of the virtual object 6 superimposed on the captured image 101 is changed all at once. Note that the control switching range is automatically determined by the CPU 31 when the mode is set.

[0073] The control conditions include operating a predetermined operation unit 21 and being set in advance in the menu of the imaging device 2 .

[0074] For example, when the control condition is set to be that the predetermined operation unit 21 is operated, the camera control unit 16 of the imaging device 2 transmits information about the operation of the predetermined operation unit 21 to the image processing device 3. In the image processing device 3, the information about the operation of the predetermined operation unit 21 is input to the condition determination unit 54, and the condition determination unit 54 determines whether or not the control condition is met based on the input information.

[0075] 9 is a diagram showing the display and non-display of the virtual object 6 in the manual mode. First, the case where the display and non-display of the virtual object 6 in the manual mode is controlled will be described.

[0076] 9A , it is assumed that a composite image for monitor 104 onto which a virtual object 6 in a normal display mode is composited is displayed on the display unit 15 and the monitor 7. In the following description, unless otherwise specified, it is assumed that the composite image for monitor 104 onto which a virtual object 6 in a normal display mode is composited is displayed first.

[0077] In this state, when the predetermined operation unit 21 is operated, the condition determination unit 54 determines that the control condition is met. If the control condition is met, the condition determination unit 54 notifies the second composition unit 56 that the control condition is met. If the second composition unit 56 is notified that the control condition is met, it does not superimpose the virtual object 6 on the captured image 101, as shown in FIG. 9B . In other words, it outputs the captured image 101 as is to the imaging device 2 and the monitor 7 as a composite image for monitoring 104. As a result, the composite image for monitoring 104 without the virtual object 6 superimposed thereon is displayed on the display unit 15 and the monitor 7.

[0078] 9B , when the predetermined operation unit 21 is no longer operated while the composite image for monitoring 104 without the virtual object 6 superimposed thereon is displayed on the display unit 15 and the monitor 7, the condition determination unit 54 determines that the control condition is no longer satisfied. When the control condition is no longer satisfied, the condition determination unit 54 notifies the second composition unit 56 that the control condition is no longer satisfied. When notified that the control condition is no longer satisfied, the second composition unit 56 superimposes the virtual object 6 on the captured image 101 to generate the composite image for monitoring 104, as shown in FIG. 9A , and outputs the composite image for monitoring 104 to the imaging device 2 and the monitor 7. As a result, the composite image for monitoring 104 with the virtual object 6 superimposed thereon is displayed on the display unit 15 and the monitor 7.

[0079] This allows the image processing system 1 to control the display and non-display of the virtual object 6 at the timing desired by the cameraman 4, making it easier for the cameraman 4 to take pictures.

[0080] 10 is a diagram showing control of the display mode of the virtual object 6 in the manual mode. Next, a case where the display mode of the virtual object 6 in the manual mode is controlled will be described.

[0081] 10A , a composite monitor image 104, onto which a virtual object 6 in a normal display mode is composited, is displayed on the display unit 15 and the monitor 7. In this state, when the predetermined operation unit 21 is operated, the condition determination unit 54 determines that a control condition is satisfied. If the control condition is satisfied, the condition determination unit 54 notifies the CG generation unit 51 or the image processing unit 55 that the control condition is satisfied. The CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the virtual object 6 is displayed in a predetermined changed display mode, and outputs the CG image 102 to the second composition unit 56. Note that the changed display mode may be one of an outline-highlighted display mode, a semi-transparent display mode, a mesh display mode, or an outline display mode, which is predetermined by, for example, the cameraman 4, or is predetermined by an attribute added to the virtual object 6. 10B , the second composition unit 56 superimposes the virtual object 6 in the changed display mode (here, a semi-transparent display mode) on the captured image 101 to generate a composite image for monitoring 104, and outputs the generated composite image for monitoring to the image capturing device 2 and the monitor 7. As a result, the composite image for monitoring 104 on which the virtual object 6 in the changed display mode is superimposed is displayed on the display unit 15 and the monitor 7. Note that, in this case, the CG image 102 may be generated by one or both of the CG generation unit 51 and the image processing unit 55. The same applies in the following description to the case where the CG image 102 is generated by the CG generation unit 51 or the image processing unit 55.

[0082] 10B , when the predetermined operation unit 21 is no longer operated while the monitor composite image 104 on which the virtual object 6 in the post-change display mode is superimposed is displayed on the display unit 15 and the monitor 7, the condition determination unit 54 determines that the control condition is no longer satisfied. If the control condition is no longer satisfied, the condition determination unit 54 notifies the CG generation unit 51 or the image processing unit 55 that the control condition is no longer satisfied. The CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the virtual object 6 has a display mode before the change (i.e., a normal display mode) and outputs the CG image 102 to the second composition unit 56. As shown in FIG. 10A , the second composition unit 56 superimposes the virtual object 6 in the normal display mode on the captured image 101 to generate the monitor composite image 104 and outputs the generated image to the imaging device 2 and the monitor 7. As a result, the monitor composite image 104 on which the virtual object 6 in the post-change display mode is superimposed is displayed on the display unit 15 and the monitor 7.

[0083] This allows the image processing system 1 to control the display mode of the virtual object 6 at the timing desired by the cameraman 4, making it easier for the cameraman 4 to take pictures, just as in the case of controlling the display and non-display of the virtual object 6.

[0084] 11 is a diagram showing changes in the transparency of the virtual object 6 in the manual mode. Next, changes in the transparency of the virtual object 6 in the manual mode will be described.

[0085] 10A , a composite monitor image 104, onto which a virtual object 6 in a normal display mode is composited, is displayed on the display unit 15 and the monitor 7. In this state, when a predetermined operation unit 21 is operated, information indicating the amount of operation of the operation unit 21 is input from the imaging device 2 to the image processing device 3, and the condition determination unit 54 determines that the control condition is met.

[0086] When the control condition is met, the condition determination unit 54 notifies the CG generation unit 51 or the image processing unit 55 that the control condition is met. The CG generation unit 51 or the image processing unit 55 generates a CG image 102 so that the virtual object 6 has a transparency corresponding to the amount of operation of the operation unit 21, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 generates a composite image for monitoring 104 in which the virtual object 6, whose transparency corresponds to the amount of operation of the operation unit 21, is superimposed on the captured image 101.

[0087] As a result, as shown in order from FIG. 10A to FIG. 10D, for example, a composite image for monitoring 104 is generated in which the transparency of the virtual object 6 increases as the amount of operation of the operation unit 21 increases.

[0088] As described above, in the image processing system 1, it is possible to allow the cameraman 4 to view the virtual object 6 with a desired transparency by adjusting the amount of operation of the operation unit 21. Furthermore, by making the transparency of the virtual object 6 variable, it is possible to adjust how the rear side of the virtual object 6 is visible.

[0089] [2.3. Camera Position and Orientation Mode] In the camera position and orientation mode, the entire range or a partial range is determined as the display control range of the virtual object 6. In the camera position and orientation mode, the display control range is determined according to the set control conditions.

[0090] FIG. 12 is a diagram showing an example of control conditions in the camera position and orientation mode. FIG. 13 is a diagram showing another example of control conditions in the camera position and orientation mode. Note that FIG. 12 shows a virtual object 6 virtually placed in real space with a dashed line. The control conditions include a change in the position or orientation of the image capture device 2, a distance between the image capture device 2 and the virtual object 6 being within a predetermined threshold Th as shown in FIG. 12, and the image capture device 2 entering a predetermined control switching area 61 as shown in FIG. 13. Note that the control switching area 61 can be set in advance by the cameraman 4 in a coordinate system of real space (or a coordinate system of virtual space). If the control conditions are a change in the position or orientation of the image capture device 2 and the image capture device 2 entering the predetermined control switching area 61, the entire range is determined as the display control range. If the control condition is a distance between the image capture device 2 and the virtual object 6 being within the threshold Th, the range within the threshold Th is determined as the display control range.

[0091] [2.3.1. When a Change in Position or Attitude of the Image Capturing Device 2 is Set as a Control Condition] When a change in position or attitude of the image capturing device 2 is set as a control condition, the condition determination unit 54 determines whether or not the control condition is met based on the position and attitude of the image capturing device 2 included in the camera information 110. The condition determination unit 54 determines that the control condition is met when the position and attitude of the image capturing device 2 included in the camera information 110 have changed. Furthermore, the condition determination unit 54 determines that the control condition is not met when the position and attitude of the image capturing device 2 included in the camera information 110 have not changed.

[0092] In this case, the image processing system 1 controls the display of the composite image for monitor 104 in the same way as in the manual mode. For example, when controlling the display and non-display of the virtual object 6, the image processing system 1 displays the composite image for monitor 104 with the virtual object 6 superimposed on it on the display unit 15 and the monitor 7 as shown in Fig. 9 if the control condition is not met, and displays the composite image for monitor 104 with the virtual object 6 not superimposed on it on the display unit 15 and the monitor 7 if the control condition is met.

[0093] Furthermore, in the image processing system 1, when controlling the display mode of the virtual object 6, as shown in FIG. 10 , if the control conditions are not met, the monitor composite image 104 on which the virtual object 6 in the normal display mode is superimposed is displayed on the display unit 15 and the monitor 7, and if the control conditions are met, the monitor composite image 104 on which the virtual object 6 in the changed display mode is superimposed is displayed on the display unit 15 and the monitor 7.

[0094] 14 is a diagram showing a change in the transparency of the virtual object 6 when a change in the position or orientation of the image capture device 2 is set as a control condition in the camera position and orientation mode. In the image processing system 1, when the transparency of the virtual object 6 is to be changed, the condition determination unit 54 notifies the CG generation unit 51 or the image processing unit 55 that the control condition is met. The CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the transparency of the virtual object 6 decreases over time, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 generates a composite image for monitoring 104 in which the virtual object 6, whose transparency has decreased over time, is superimposed on the captured image 101.

[0095] As a result, as shown in order from Figure 14A to Figure 14D, for example, a composite image for monitor 104 is displayed on the display unit 15 and the monitor 7, in which the transparency of the virtual object 6 decreases as time passes after the control condition is met.

[0096] In this way, when the position or attitude of the imaging device 2 is changing, that is, when the cameraman 4 is moving the imaging device 2, it is possible to reduce the obstruction of movement caused by the virtual object 6 being displayed on the display unit 15 and the monitor 7 by hiding the virtual object 6 or changing the display mode of the virtual object 6. It is also possible to facilitate the imaging work of the cameraman 4.

[0097] [2.3.2. Case where the control condition is set to be that the imaging device 2 has entered the predetermined control switching area 61] When the control condition is set to be that the imaging device 2 has entered the predetermined control switching area 61, the condition determination unit 54 determines whether or not the control condition is met based on the position and attitude of the imaging device 2 included in the camera information 110. The condition determination unit 54 calculates the position of the imaging device 2, and determines that the control condition is met when the imaging device 2 is inside the control switching area 61. Furthermore, the condition determination unit 54 determines that the control condition is not met when the imaging device 2 is outside the control switching area 61.

[0098] In this case, the image processing system 1 performs display control similar to that performed when the control condition is set to be a change in the position or orientation of the image capture device 2. Note that the display control has been described above, and therefore a description thereof will be omitted here.

[0099] In this way, when the imaging device 2 enters the control switching area 61, that is, when the imaging device 2 enters any position, it is possible to hide the virtual object 6 or change the display mode of the virtual object 6. This makes it possible to display the virtual object 6 in a way that reflects the intention of the cameraman 4, making it easier for the cameraman 4 to take pictures.

[0100] [2.3.3. When the control condition is set such that the distance between the image capture device 2 and the virtual object 6 is within a predetermined threshold value Th] When the control condition is set such that the distance (distance on the coordinate system; the same applies hereinafter) between the image capture device 2 and the virtual object 6 is within a predetermined threshold value Th, the condition determination unit 54 determines whether the control condition is met based on the position and orientation of the image capture device 2 included in the camera information 110. The condition determination unit 54 calculates the positions of the image capture device 2 and the virtual object 6, and determines that the control condition is met when the distance between the image capture device 2 and the virtual object 6 is within the threshold value Th. Furthermore, the condition determination unit 54 determines that the control condition is not met when the distance between the image capture device 2 and the virtual object 6 is not within the threshold value Th.

[0101] 15 is a diagram illustrating display control of a virtual object 6 when a control condition is set that the distance between the image capture device 2 and the virtual object 6 is within a predetermined threshold Th. When the control condition is met, the condition determination unit 54 notifies the CG generation unit 51 or the image processing unit 55 that the control condition is met. When controlling display and non-display of the virtual object 6, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which a portion of the virtual object 6 whose distance from the image capture device 2 is within the threshold Th is hidden, and outputs the CG image 102 to the second composition unit 56. As shown in FIG. 15A , the second composition unit 56 generates a monitor composite image 104 in which a portion of the virtual object 6 whose distance from the image capture device 2 is within the threshold Th is hidden, and outputs the monitor composite image 104 to the image capture device 2 and the monitor 7. As a result, the monitor composite image 104 on which the virtual object 6 is superimposed, in which a portion of the virtual object 6 whose distance from the image capture device 2 is within the threshold Th is hidden, is displayed on the display unit 15 and the monitor 7. In this way, parts whose distance from the imaging device 2 is within the threshold value Th are hidden, so parts that get in the way of the cameraman 4 when he moves can be hidden, reducing the likelihood that the virtual object 6 will get in the way of his movement.

[0102] Furthermore, when controlling the display mode of the virtual object 6, the condition determination unit 54, when a control condition is satisfied, notifies the CG generation unit 51 or the image processing unit 55 that the control condition is satisfied. The CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which a portion of the virtual object 6 whose distance from the image capture device 2 is within the threshold Th is displayed in a changed display mode, and outputs the CG image 102 to the second composition unit 56. As shown in FIG. 15B , the second composition unit 56 generates a monitor composite image 104 in which a portion of the virtual object 6 whose distance from the image capture device 2 is within the threshold Th is displayed in a changed display mode (here, a semi-transparent display mode), and outputs the monitor image 104 to the image capture device 2 and the monitor 7. As a result, the monitor composite image 104 in which the virtual object 6 whose portion whose distance from the image capture device 2 is within the threshold Th is displayed in a different display mode is superimposed on the display unit 15 and the monitor 7. In this way, the portion whose distance from the image capture device 2 is within the threshold Th is displayed in a different display mode, thereby reducing the virtual object 6 from interfering with movement.

[0103] 16 is a diagram showing changes in the transparency of the virtual object 6 when the control condition is set such that the distance between the image capture device 2 and the virtual object 6 is within a predetermined threshold Th. When changing the transparency of the virtual object 6, the condition determination unit 54 notifies the CG generation unit 51 or the image processing unit 55 that the control condition is met. The CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the transparency of the portion of the virtual object 6 whose distance from the image capture device 2 is within the threshold Th decreases over time, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 generates a composite image for monitoring 104 in which the virtual object 6 having the portion whose transparency decreases over time is superimposed on the captured image 101.

[0104] As a result, as shown in Figures 16A to 16D, for example, a composite image 104 for monitoring is displayed on the display unit 15 and the monitor 7, in which the transparency of the portion of the virtual object 6 whose distance from the imaging device 2 is within the threshold value Th decreases as time passes after the control condition is met.

[0105] In this way, by changing the transparency of the portion whose distance from the imaging device 2 is within the threshold Th, it is possible to reduce the obstruction of movement caused by the virtual object 6 being displayed on the display unit 15 and the monitor 7. It also makes it easier for the cameraman 4 to take pictures.

[0106] [2.4. Imaging Control Mode] In the imaging control mode, the entire range or a partial range is determined as the display control range of the virtual object 6. In the imaging control mode, the display control range is determined by the cameraman 4 or the like. As will be described in detail later, the entire range, a range behind or in front of the focus position, or a range other than the depth of field can be selected as the display control range.

[0107] FIG. 17 is a diagram illustrating the depth of field. As shown in FIG. 17, the imaging device 2 can focus on a position where a virtual object 6 that is not actually being imaged is located. In this case, the portion of the virtual object 6 that corresponds to the depth of field indicated by the dashed-dotted line in FIG. 17 (the portion indicated by the solid line in FIG. 17) is in focus, while the portions before and after that portion (the portions indicated by the dashed line in FIG. 17) are out of focus. Therefore, in the imaging control mode, display control of the virtual object 6 is performed based on the focus position (or the depth of field), allowing the cameraman 4 to recognize the focus position.

[0108] The control condition is that the focus position (focal distance) of the imaging device 2 has changed. The condition determination unit 54 determines whether the control condition is met based on the focal distance included in the camera information 110. The condition determination unit 54 determines that the control condition is met if the focal distance included in the camera information 110 has changed. The condition determination unit 54 also determines that the control condition is not met if the focal distance included in the camera information 110 has not changed. If the control condition is met, the condition determination unit 54 notifies the CG generation unit 51 or the image processing unit 55 that the control condition is met.

[0109] [2.4.1. Control of Display and Non-Display of Virtual Object 6 in Imaging Control Mode] FIG. 18 is a diagram illustrating control of display and non-display of the virtual object 6 in the imaging control mode.

[0110] When the display control range is determined to be behind the focus position, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the part of the virtual object 6 behind the focus position is hidden, as shown in FIG. 18A , and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 combines the input CG image 102 with the captured image 101 to generate a composite image for monitoring 104, and outputs the composite image for monitoring to the imaging device 2 and the monitor 7. As a result, the composite image for monitoring 104, in which the virtual object 6 behind the focus position is hidden and superimposed, is displayed on the display unit 15 and the monitor 7. In this way, by hiding the part behind the focus position, it is possible to allow the cameraman 4 to easily recognize the focus position and to allow the cameraman 4 to check only the part in front of the focus position.

[0111] Furthermore, when the area in front of the focus position is determined as the display control range, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the area in front of the focus position of the virtual object 6 is hidden, as shown in FIG. 18B , and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 combines the input CG image 102 with the captured image 101 to generate a composite image for monitoring 104, and outputs the composite image for monitoring to the imaging device 2 and the monitor 7. As a result, the composite image for monitoring 104, in which the virtual object 6 in front of the focus position is hidden and superimposed, is displayed on the display unit 15 and the monitor 7. In this way, by hiding the area in front of the focus position, it is possible to allow the cameraman 4 to easily recognize the focus position and to allow the cameraman 4 to check only the area behind the focus position.

[0112] Furthermore, when a range other than the depth of field is determined as the display control range, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the range of the virtual object 6 other than the depth of field is hidden (indicated by a dashed line in the figure), as shown in FIG. 18C , and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 combines the input CG image 102 with the captured image 101 to generate a composite image for monitoring 104, and outputs the composite image for monitoring 104 to the image capture device 2 and the monitor 7. As a result, the composite image for monitoring 104, on which the virtual object 6 now only within the depth of field is superimposed, is displayed on the display unit 15 and the monitor 7. In this way, displaying the virtual object 6 only within the depth of field allows the cameraman 4 to easily recognize the focus position and confirm the range in focus.

[0113] [2.4.2. Control of Display Mode of Virtual Object 6 in Imaging Control Mode] When the rear side of the focus position is determined as the display control range, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the rear side of the focus position of the virtual object 6 has been changed in its display mode, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 combines the input CG image 102 with the captured image 101 to generate a composite image for monitoring 104, and outputs the composite image for monitoring to the imaging device 2 and the monitor 7. As a result, the composite image for monitoring 104, on which the virtual object 6 in which the rear side of the focus position has been changed in its display mode, is superimposed, is displayed on the display unit 15 and the monitor 7.

[0114] Furthermore, when the area in front of the focus position is determined as the display control range, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the area in front of the focus position of the virtual object 6 has been changed to a new display state, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 combines the input CG image 102 with the captured image 101 to generate a composite image for monitoring 104, and outputs the composite image for monitoring to the imaging device 2 and the monitor 7. As a result, the composite image for monitoring 104, in which the area in front of the focus position of the virtual object 6 has been changed to a new display state, is superimposed, and is displayed on the display unit 15 and the monitor 7.

[0115] Furthermore, when a range other than the depth of field is determined as the display control range, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the range other than the depth of field of the virtual object 6 has been changed in its display mode, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 combines the input CG image 102 with the captured image 101 to generate a composite image for monitoring 104, and outputs the composite image for monitoring to the imaging device 2 and the monitor 7. As a result, the composite image for monitoring 104, on which the virtual object 6 in which the range outside the depth of field has been changed in its display mode, is superimposed, is displayed on the display unit 15 and the monitor 7.

[0116] [2.4.3. Change in Transparency of Virtual Object 6 in Imaging Control Mode] When the display control range is determined to be behind the focus position, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the transparency of the virtual object 6 on the side behind the focus position decreases over time, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 generates a composite image for monitoring 104 in which the virtual object 6 having a portion whose transparency decreases over time is superimposed on the captured image 101, and outputs the composite image for monitoring 104 to the imaging device 2 and the monitor 7. As a result, the composite image for monitoring 104 in which the transparency of the side in front of the focus position decreases over time is displayed on the display unit 15 and the monitor 7.

[0117] When the display control range is determined to be the area in front of the focus position, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the transparency of the virtual object 6 on the area in front of the focus position decreases over time, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 generates a composite image for monitoring 104 in which the virtual object 6 having a portion that decreases in transparency over time is superimposed on the captured image 101, and outputs the composite image for monitoring 104 to the imaging device 2 and the monitor 7. As a result, the composite image for monitoring 104 in which the transparency of the area behind the focus position decreases over time is displayed on the display unit 15 and the monitor 7.

[0118] 19 is a diagram showing changes in the transparency of a virtual object 6 when a range other than the depth of field is determined as the display control range in the imaging control mode. When a range other than the depth of field is determined as the display control range, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the transparency of the range of the virtual object 6 other than the depth of field decreases over time, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 generates a composite image for monitoring 104 in which the virtual object 6 having a portion whose transparency decreases over time is superimposed on the captured image 101, and outputs the composite image for monitoring 104 to the imaging device 2 and the monitor 7. As a result, as shown in FIGS. 19A to 19D in order, the composite image for monitoring 104 in which the transparency of the range other than the depth of field decreases over time is displayed on the display unit 15 and the monitor 7. In this way, by displaying the virtual object 6 only within the depth of field, the cameraman 4 can easily recognize the focus position and can also confirm the range in focus.

[0119] [2.5. Virtual Object Mode] In the virtual object mode, the entire range or a partial range is determined as the display control range of the virtual object 6. In the virtual object mode, the display control range is determined according to the set control conditions.

[0120] FIG. 20 is a diagram showing an example of control conditions in the virtual object mode. The control conditions include the virtual object 6 entering a control switching area 62 as shown in FIG. 20 , the virtual object 6 entering the angle of view of the imaging device 2, and the display control conditions assigned to the virtual object 6 being satisfied. The control switching area 62 can be set in advance by the cameraman 4 in a coordinate system of real space (or a coordinate system of virtual space). When the control condition is the virtual object 6 entering the control switching area 62 as shown in FIG. 20 , the control switching area 62 is determined as the display control range. When the control condition is the virtual object 6 entering the angle of view of the imaging device 2, the entire area is determined as the display control range. When the control condition is the display control conditions assigned to the virtual object 6 being satisfied, the display control range is determined for each virtual object 6.

[0121] [2.5.1. When the control condition is set to be that the virtual object 6 has entered the control switching area 62] When the control condition is set to be that the virtual object 6 has entered the control switching area 62, the condition determination unit 54 determines whether the control condition is met based on the position of the virtual object 6 included in the CG model and the position of the control switching area 62. The condition determination unit 54 determines that the control condition is met when the virtual object 6 has entered the control switching area 62. Furthermore, the condition determination unit 54 determines that the control condition is not met when the virtual object 6 has not entered the control switching area 62.

[0122] FIG. 21 illustrates display control of a virtual object 6 when a control condition is set that a virtual object 6 has entered the control switching area 62. When the control condition is met, the condition determination unit 54 notifies the CG generation unit 51 or the image processing unit 55 that the control condition has been met. When controlling the display or non-display of the virtual object 6, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the portion of the virtual object 6 within the control switching area 62 is hidden, and outputs the CG image 102 to the second composition unit 56. As shown in FIG. 21A , the second composition unit 56 generates a monitor composite image 104 in which the portion of the virtual object 6 within the control switching area 62 is hidden, and outputs the monitor composite image 104 to the imaging device 2 and the monitor 7. As a result, the monitor composite image 104 on which the virtual object 6 with the portion within the control switching area 62 hidden is superimposed is displayed on the display unit 15 and the monitor 7. Therefore, when a virtual object 6 enters a predetermined control switching area 62, the control switching area 62 can be easily hidden. It is also possible for the cameraman 4 to always check the inside of the control switching area 62. Furthermore, since the display can be controlled according to the position of the virtual object 6 simply by specifying a position in the real space or the virtual space, the controllability of the display control of the virtual object 6 can be improved.

[0123] Furthermore, when controlling the display mode of the virtual object 6, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the portion of the virtual object 6 within the control switching region 62 has a changed display mode, and outputs the CG image 102 to the second composition unit 56. As shown in FIG. 21B , the second composition unit 56 generates a monitor composite image 104 in which the portion of the virtual object 6 within the control switching region 62 has a changed display mode (here, a semi-transparent display mode), and outputs the monitor composite image 104 to the imaging device 2 and the monitor 7. As a result, the monitor composite image 104 in which the virtual object 6 in the changed display mode is superimposed on the portion within the control switching region 62 is displayed on the display unit 15 and the monitor 7. In this way, when the virtual object 6 enters a predetermined control switching region 62, the control switching region 62 can be displayed in a different display mode. Furthermore, since control can be achieved based on the position of the virtual object 6 simply by specifying a position in real space or virtual space, the controllability of the display control of the virtual object 6 can be improved.

[0124] 22 is a diagram showing a change in the transparency of the virtual object 6 when the control condition is set to be that the virtual object 6 has entered the control switching area 62. When changing the transparency of the virtual object 6, the condition determination unit 54 notifies the CG generation unit 51 or the image processing unit 55 that the control condition is met. The CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the portion of the virtual object 6 within the control switching area 62 becomes less transparent over time, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 generates a composite image for monitoring 104 in which the virtual object 6 having a portion whose transparency becomes less transparent over time is superimposed on the captured image 101.

[0125] As a result, as shown in Figures 22A to 22D, for example, a composite image 104 for monitoring is displayed on the display unit 15 and the monitor 7, in which the transparency of the portion of the virtual object 6 within the control switching area 62 decreases as time passes after the control condition is met.

[0126] In this way, by changing the transparency of the portion of the virtual object 6 within the control switching area 62 , the cameraman 4 can confirm the real object 5 and the virtual object 6 within the control switching area 62 .

[0127] [2.5.2. When the control condition is set such that the virtual object 6 enters the angle of view of the imaging device 2] When the control condition is set such that the virtual object 6 enters the angle of view of the imaging device 2, the condition determination unit 54 determines whether the control condition is met based on the position of the virtual object 6 included in the CG model and the angle of view, etc. included in the camera information 110. The condition determination unit 54 determines that the control condition is met when the virtual object 6 is included in the angle of view of the imaging device 2. Furthermore, the condition determination unit 54 determines that the control condition is not met when the virtual object 6 is not included in the angle of view of the imaging device 2.

[0128] FIG. 23 is a diagram showing a change in transparency of a virtual object 6 when the control condition is set to be that the virtual object 6 has entered the field of view of the imaging device 2. Note that FIG. 23 illustrates a case in which the virtual object 6 is moving from right to left. Also, in FIG. 23, the lower the transparency, the darker the fill. When the control condition is met, the condition determination unit 54 notifies the CG generation unit 51 or the image processing unit 55 that the control condition is met. As shown in FIG. 23, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 such that the transparency of the portion of the virtual object 6 that has entered the field of view decreases over time, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 generates a composite image for monitoring 104 in which the virtual object 6, having a portion that has entered the field of view and whose transparency decreases over time, is superimposed on the captured image 101.

[0129] As a result, as shown in Figures 23A to 23C in order, a composite image 104 for monitoring is displayed on the display unit 15 and the monitor 7, in which the longer the time that part of the virtual object 6 is in the field of view, the less transparent it becomes.

[0130] Therefore, in the image processing system 1, it is possible to confirm the time when the virtual object 6 entered the field of view based on the transparency of the virtual object 6.

[0131] Note that, when the control condition is set such that the virtual object 6 has entered the angle of view of the imaging device 2, the display and non-display of the virtual object 6 and the display mode of the virtual object 6 are not controlled. However, when the control condition is set such that the virtual object 6 has entered the angle of view of the imaging device 2, the display and non-display of the virtual object 6 and the display mode of the virtual object 6 may be controlled.

[0132] [2.5.3. Cases in which the display control conditions assigned to the virtual object 6 are satisfied as control conditions] When the display control conditions assigned to the virtual object 6 are satisfied as control conditions, the condition determination unit 54 determines whether the control conditions are satisfied based on whether the display control conditions assigned to the virtual object 6 are satisfied. The condition determination unit 54 determines that the control conditions are satisfied when the display control conditions assigned to the virtual object 6 are satisfied. Furthermore, the condition determination unit 54 determines that the control conditions are not satisfied when the display control conditions assigned to the virtual object 6 are not satisfied. When the control conditions are satisfied, the condition determination unit 54 notifies the CG generation unit 51 or the image processing unit 55 that the control conditions are satisfied.

[0133] When controlling the display and non-display of a virtual object 6, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the virtual object 6 to which the display control condition is assigned is not displayed, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 generates a composite image for monitor 104 in which the virtual object 6 to which the display control condition is assigned is not displayed, and outputs the composite image for monitor 104 to the imaging device 2 and the monitor 7. As a result, the composite image for monitor 104 in which the virtual object 6 to which the display control condition is assigned is not superimposed is displayed on the display unit 15 and the monitor 7. Therefore, the image processing system 1 makes it possible to control the display and non-display of each virtual object 6.

[0134] When controlling the display mode of the virtual object 6, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the virtual object 6 to which the display control conditions have been assigned has a changed display mode, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 generates a composite image for monitor 104 in which the virtual object 6 to which the display control conditions have been assigned has a changed display mode, and outputs the composite image for monitor 104 to the imaging device 2 and the monitor 7. As a result, the composite image for monitor 104 in which the virtual object 6 to which the display control conditions have been assigned is superimposed in different display modes is displayed on the display unit 15 and the monitor 7. Therefore, the image processing system 1 makes it possible to control the display mode for each virtual object 6.

[0135] When changing the transparency of the virtual object 6, the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the transparency of the virtual object 6 to which a display control condition has been assigned decreases over time, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 generates a composite image for monitor 104 in which the virtual object 6 having a portion whose transparency decreases over time is superimposed on the captured image 101, and outputs the composite image for monitor 104 to the image capture device 2 and the monitor 7. As a result, the composite image for monitor 104 in which the transparency of the virtual object 6 to which a display control condition has been assigned decreases over time is displayed on the display unit 15 and the monitor 7. Therefore, the image processing system 1 makes it possible to change the transparency for each virtual object 6.

[0136] [2.6. Real Object Mode] In the real object mode, a partial range is determined as the display control range of the virtual object 6. In the real object mode, the periphery of the real object 5 recognized by the image recognition unit 53 is determined as the display control range.

[0137] The control condition is set such that the distance between the real object 5 and the virtual object 6 recognized by the image recognition unit 53 is equal to or less than a predetermined value. The image recognition unit 53 recognizes a predetermined real object 5 from the captured image 101. The condition determination unit 54 calculates the coordinate distance between the real object 5 recognized by the image recognition unit 53 and the virtual object 6. The calculated distance may be a three-dimensional distance or a two-dimensional distance. The condition determination unit 54 determines whether the control condition is met based on the calculated distance. The condition determination unit 54 determines that the control condition is met if the calculated distance is equal to or less than the predetermined value. Furthermore, the condition determination unit 54 determines that the control condition is not met if the calculated distance is not equal to or less than the predetermined value. If the control condition is met, the condition determination unit 54 notifies the CG generation unit 51 or the image processing unit 55 that the control condition is met.

[0138] [2.6.1. Control of Display and Non-Display of Virtual Object 6 in Real Object Mode] FIG. 24 is a diagram illustrating display control of the virtual object 6 in the real object mode.

[0139] 24A , the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which a portion of the virtual object 6 whose distance from the real object 5 is equal to or less than a predetermined value is hidden, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 combines the input CG image 102 with the captured image 101 to generate a composite image for monitoring 104, and outputs the composite image for monitoring to the imaging device 2 and the monitor 7. As a result, the composite image for monitoring 104 in which the virtual object 6 is not superimposed on the real object 5 is displayed on the display unit 15 and the monitor 7. In this way, by displaying the virtual object 6 so that it is not superimposed on the real object 5, the real object 5 and the virtual object 6 can be easily recognized by the cameraman 4.

[0140] 24B , the CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the portion of the virtual object 6 whose distance from the real object 5 is equal to or less than a predetermined value is displayed in a changed display mode (here, a semi-transparent display mode), and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 combines the input CG image 102 with the captured image 101 to generate a composite image for monitoring 104, which is output to the imaging device 2 and the monitor 7. As a result, the composite image for monitoring 104 in which the portion of the virtual object 6 that overlaps the real object 5 is displayed in a different display mode is displayed on the display unit 15 and the monitor 7. In this way, by changing the display mode of the portion of the virtual object 6 that overlaps the real object 5, the real object 5 and the virtual object 6 can be easily recognized by the cameraman 4.

[0141] [2.6.3. Change in Transparency of Virtual Object 6 in Imaging Control Mode] Figure 25 is a diagram illustrating a change in transparency of the virtual object 6 in the real object mode. The CG generation unit 51 or the image processing unit 55 generates a CG image 102 in which the transparency of portions of the virtual object 6 whose distance from the real object 5 is equal to or less than a predetermined value decreases over time, and outputs the CG image 102 to the second composition unit 56. The second composition unit 56 generates a composite image for monitoring 104 in which the virtual object 6 having portions whose transparency decreases over time is superimposed on the captured image 101, and outputs the composite image for monitoring 104 to the imaging device 2 and the monitor 7. As a result, as shown in Figures 25A to 25D in order, the composite image for monitoring 104 in which the transparency of portions of the virtual object 6 that overlap with the real object 5 decreases over time is displayed on the display unit 15 and the monitor 7. In this way, by decreasing the transparency of portions of the virtual object 6 that overlap with the real object 5 over time, the real object 5 and the virtual object 6 can be easily recognized by the cameraman 4.

[0142] 3. Icons 105 displayed on the composite image 104 for monitoring Fig. 26 is a diagram showing an example of an icon 105 indicating a mode. Fig. 27 is a diagram showing an example of an icon 105 indicating a display mode. Fig. 28 is a diagram showing an example of an icon 105 indicating the passage of time in a change in transparency. Fig. 29 is a diagram showing an icon 105 during special processing.

[0143] Fig. 30 is a diagram showing an example of a composite image for monitoring 104 on which icons 105 are displayed. As shown in Fig. 30, the composite image for monitoring 104 may display icons 105 indicating the currently set mode, display mode, elapsed time, etc.

[0144] As shown in Fig. 26, different shapes of the icons 105 are associated with the respective set modes. Also, as shown in Fig. 27, different contours or patterns of the icons 105 are associated with the respective set display modes. Note that Fig. 27 shows the display modes for the virtual object modes.

[0145] Therefore, in the example of Figure 30, since the mode is the virtual object mode and the display mode is the semi-transparent display mode, an icon 105 associated with the virtual object mode and the semi-transparent display mode is displayed on the composite image 104 for monitor.

[0146] Furthermore, as shown in FIG. 28, when the display mode of the virtual object 6 is changed over time, the icon 105 may be configured to display the remaining time until the final change in display mode, such as "3," "2," or "1."

[0147] Furthermore, as shown in FIG. 29A, if, for example, virtual object 6 is assigned an attribute that prohibits display control, icon 105 may be displayed in a color indicating that display control is prohibited (a color different from the color of the icons shown in FIGS. 26 and 27).

[0148] Furthermore, as shown in FIG. 29B, if, for example, virtual object 6 is assigned an attribute that prohibits display control, icon 105 may display a cross mark indicating that display control is prohibited.

[0149] 31 is a flowchart showing the processing flow for display control of the virtual object 6. Fig. 32 is a flowchart showing the processing flow for display control for each set mode.

[0150] 31, in step S1, the CPU 31 determines whether the manual mode is set. If the manual mode is set, the process proceeds to step S6. If the manual mode is not set, the process proceeds to step S2.

[0151] In step S2, the CPU 31 determines whether the camera position and orientation mode is set. If the camera position and orientation mode is set, the process proceeds to step S6. If the camera position and orientation mode is not set, the process proceeds to step S3.

[0152] In step S3, the CPU 31 determines whether the imaging control mode is set. If the imaging control mode is set, the process proceeds to step S6. If the imaging control mode is not set, the process proceeds to step S4.

[0153] In step S4, the CPU 31 determines whether the virtual object mode is set. If the virtual object mode is set, the process proceeds to step S6. If the virtual object mode is not set, the process proceeds to step S5.

[0154] In step S5, the CPU 31 determines whether the real object mode is set. If the real object mode is set, the process proceeds to step S6. If the real object mode is not set, the process proceeds to step S1.

[0155] In step S6, the CPU 31 executes a display control process corresponding to the selected mode. The display control process for each mode will be described with reference to FIG.

[0156] In this way, when any one of the manual mode, camera position and orientation mode, imaging control mode, virtual object mode, and subject mode is set, a display control process corresponding to one mode is executed, with priority given in this order: manual mode, camera position and orientation mode, imaging control mode, virtual object mode, and subject mode. Note that the order of steps S1 to S5 may be other than the order shown in Fig. 31. Furthermore, display control processes corresponding to a plurality of modes (e.g., manual mode and another mode) including manual mode, camera position and orientation mode, imaging control mode, virtual object mode, and subject mode may be executed simultaneously in parallel.

[0157] 32 , when executing the display control process for each mode, the CPU 31 determines one of the control conditions of the set mode in step S11. Then, in step S12, the CPU 31 determines a display control range corresponding to the set mode or control condition. Then, in step S13, the CPU 31 determines one of the types of display control (control of display and non-display of the virtual object 6, control of the display mode of the virtual object 6, or change in transparency of the virtual object 6).

[0158] In step S14, it is determined whether the control conditions are met. If the control conditions are not met, in step S15, the CPU 31 generates a monitor composite image 104 so that the virtual object 6 is displayed in the normal display mode, and outputs the generated monitor composite image 104 to the image capture device 2 and the monitor 7. On the other hand, if the control conditions are met, in step S16, the CPU 31 generates a monitor composite image 104 so that the virtual object 6 is displayed based on the determined display control range, display mode, etc., and outputs the generated monitor composite image 104 to the image capture device 2 and the monitor 7.

[0159] 5. Modifications Note that the embodiment is not limited to the specific example described above, and various modifications may be adopted. For example, the CPU 31 or the GPU 34 of the image processing device 3 may function as a CG generation unit 51, a first composition unit 52, an image recognition unit 53, a condition determination unit 54, an image processing unit 55, and a second composition unit 56.

[0160] However, as shown in FIG. 33, the CPU 31 or GPU 34 of the image processing device 3 may function as a CG generation unit 51, a first synthesis unit 52, an image recognition unit 53, a condition determination unit 54, and an image processing unit 55, and the imaging device 2 (camera control unit 16) may function as a second synthesis unit 56.

[0161] Also, as shown in Figure 34, the imaging device 2 (camera control unit 16) may function as a CG generation unit 51, a first synthesis unit 52, an image recognition unit 53, a condition determination unit 54, an image processing unit 55, and a second synthesis unit 56.

[0162] Furthermore, as shown in FIG. 35, the CPU 31 or GPU 34 of the image processing device 3 may function as a CG generation unit 51 and a first synthesis unit 52, and the imaging device 2 (camera control unit 16) may function as an image recognition unit 53, a condition determination unit 54, an image processing unit 55, and a second synthesis unit 56.

[0163] In the above embodiment, once a mode is determined, the control conditions corresponding to the determined mode are set. However, it is also possible to set the control conditions regardless of the mode without providing a mode.

[0164] 6. Summary of the embodiment As described above, the image processing method of the embodiment controls, based on predetermined control conditions, the display of the virtual object 6 to be composited into the captured image 101 captured by the imaging device 2. This makes it possible to, for example, control whether to display or hide the virtual object 6, or control the display mode of the virtual object 6, based on the predetermined control conditions. This makes it possible to improve the convenience of the composite image 104 for monitoring for the user (cameraman 4) during imaging.

[0165] Whether or not to display the virtual object 6 is controlled according to the control conditions. This makes it possible to change whether or not the virtual object 6 is displayed depending on whether or not the control conditions are met. Therefore, for example, when moving the imaging device 2, it is possible to reduce the possibility that the virtual object 6 is displayed on the display unit 15 and the monitor 7, making it difficult to see the real space.

[0166] The display mode of the virtual object 6 is controlled according to the control conditions. This allows the display mode of the virtual object 6 to be changed between a normal display mode and another display mode depending on whether the control conditions are met. Therefore, for example, when the real space is difficult to see, the display mode can be changed to one that makes the virtual object 6 easier to see without completely displaying it.

[0167] Of a plurality of modes with different control conditions, the display of the virtual object 6 to be composited into the captured image 101 is controlled based on the control condition corresponding to the set mode. This allows the cameraman 4 to control the display of the virtual object 6 in a mode desired by the cameraman 4, and enables the display of a more easily viewable composite image for monitoring 104.

[0168] The multiple modes include a camera position and orientation mode in which control conditions are set based on the position and orientation of the imaging device 2, an imaging control mode in which control conditions are set based on imaging information (camera information 110) of the imaging device 2, a virtual object mode in which control conditions are set based on a virtual object 6, and a real object mode in which control conditions are set based on a real object. This makes it possible to automatically (without operation by the cameraman 4) change the display of the virtual object 6 based on the control conditions corresponding to the set mode.

[0169] The control conditions include at least one of the following: a change in the position or orientation of the imaging device 2; a distance between the imaging device 2 and the virtual object 6 being within a predetermined threshold; and the imaging device 2 entering a predetermined range (control switching area 61). This makes it possible to automatically change the display of the virtual object 6 based on the position and orientation of the imaging device 2. Furthermore, it is possible to reduce deterioration of visibility caused by the virtual object 6 being displayed in the normal display mode on the display unit 15 and the monitor 7 when the imaging device 2 is close to the virtual object 6.

[0170] When the distance between the image capture device 2 and the virtual object 6 is within a predetermined threshold Th, the portion of the virtual object 6 that is within the threshold Th is hidden or its display mode is changed. This makes it possible to reduce the deterioration of visibility that occurs when the virtual object 6 is displayed in the normal display mode on the display unit 15 and the monitor 7 when the image capture device 2 is close to the virtual object 6.

[0171] When the imaging device 2 is within a predetermined range (control switching area 61), the virtual object 6 is hidden or the display mode is changed. This makes it possible to prevent the virtual object 6 from being displayed in the normal display mode when the imaging device 2 enters the control switching area 61.

[0172] The control conditions include a change in the focus position of the imaging device 2. This allows the cameraman 4 to check the focus position when the focus position of the imaging device 2 is being moved.

[0173] When the focus position of the imaging device is changed, the portion of the virtual object 6 that corresponds to the depth of field is displayed in the normal display mode, and the remaining portion is hidden or its display mode is changed. This allows the cameraman 4 to easily confirm the portion of the virtual object 6 that is in focus, and also makes it easier for the cameraman 4 to confirm the real space.

[0174] When the focus position is changed in the imaging device, the portion of the virtual object 6 in front of the focus position is displayed in a normal display mode, and the remaining portion is hidden or its display mode is changed. This allows the cameraman 4 to easily check the focus position and also makes it easier to check the real space.

[0175] When the focus position is changed in the imaging device, the part of the virtual object 6 behind the focus position is displayed in a normal display mode, and the rest of the virtual object 6 is hidden or its display mode is changed. This allows the cameraman 4 to easily check the focus position and also makes it easier to check the real space.

[0176] The control conditions include at least one of the following: the virtual object 6 has entered a predetermined range (control switching area 62), the virtual object 6 has entered the angle of view of the imaging device 2, and a condition assigned to the virtual object 6 has been met. This allows the display of the virtual object 6 to be controlled based on the position of the virtual object 6, thereby improving controllability.

[0177] When the virtual object 6 enters a predetermined range (control switching area 62), the part of the virtual object 6 within the range (control switching area 62) is hidden or the display mode is changed. This makes it possible to hide or change the display of the virtual object 6 when the virtual object 6 enters the predetermined control switching area 62, thereby improving the controllability of the display control of the virtual object 6.

[0178] When a virtual object 6 enters the angle of view of the imaging device 2, the transparency of the part of the virtual object 6 that enters the angle of view is changed. This makes it possible to confirm the time when the virtual object 6 entered the angle of view based on the transparency.

[0179] The control conditions include that the distance between the real object 5 and the virtual object 6 is within a predetermined distance. This reduces the possibility that the real object 5 will overlap the virtual object 6 and become difficult to identify when the real object 5 and the virtual object 6 come close to each other.

[0180] When the distance between the real object 5 and the virtual object 6 is within a predetermined distance, the display mode of the virtual object 6 within the predetermined distance is changed. This reduces the possibility that the real object 5 overlaps the virtual object 6, making it difficult to identify the real object 5.

[0181] When the control condition is met, the transparency of the virtual object 6 is changed over time. As a result, the display form of the virtual object 6 changes over time, allowing the user to easily confirm the virtual object 6 and the real space.

[0182] The image processing device 3 also includes a display control unit (CG generation unit 51, image processing unit 55, second composition unit 56) that controls, based on predetermined control conditions, the display of a virtual object 6 to be composited into a captured image 101 captured by the imaging device 2. The image processing system 1 also includes the imaging device 2 and the image processing device 3, wherein the imaging device 2 includes an imaging unit 12 that obtains a captured image 101 by capturing an image of real space, and the image processing device 3 includes a display control unit (CG generation unit 51, image processing unit 55, second composition unit 56) that controls, based on control conditions, the display of a virtual object 6 to be composited into the captured image 101 captured by the imaging device 2. Such an image processing device and image processing system can also achieve the same functions and effects as the image processing method of the above-described embodiment.

[0183] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0184] <8. Present Technology> The present technology may also have the following configurations. (1) An image processing method for controlling, based on a control condition, display of a virtual object to be composited into a captured image captured by an imaging device. (2) The image processing method according to (1), for controlling whether or not to display the virtual object according to the control condition. (3) The image processing method according to (1), for controlling a display mode of the virtual object according to the control condition. (4) The image processing method according to (1) or (2), for controlling display of a virtual object to be composited into the captured image based on a control condition corresponding to a set mode among a plurality of modes having different control conditions. (5) The image processing method according to any of (1) to (3), wherein the plurality of modes include any of a camera position and orientation mode in which a control condition based on a position and orientation of an imaging device is set, an imaging control mode in which a control condition based on imaging information of the imaging device is set, a virtual object mode in which a control condition based on a virtual object is set, and a real object mode in which a control condition based on a real object is set. (6) The image processing method according to (5), wherein the control condition includes at least one of a change in the position or orientation of the imaging device, a distance between the imaging device and the virtual object being within a predetermined threshold, and the imaging device entering a predetermined range. (7) The image processing method according to (6), wherein, when the distance between the imaging device and the virtual object is within a predetermined threshold, a portion of the virtual object within the threshold is hidden or its display mode is changed. (8) The image processing method according to (6) or (7), wherein, when the imaging device is within a predetermined range, the virtual object is hidden or its display mode is changed. (9) The image processing method according to any of (6) to (8), wherein the control condition is a change in the focus position of the imaging device. (10) The image processing method according to (9), wherein, when the focus position of the imaging device is changed, a portion of the virtual object corresponding to the depth of field is displayed in a normal display mode, and portions other than the portion are hidden or their display mode is changed.(11) The image processing method according to (9) or (10), in which, when a focus position has changed in the imaging device, a portion of the virtual object in front of the focus position is displayed in a normal display mode, and the remaining portion is hidden or its display mode is changed. (12) The image processing method according to any one of (9) to (11), in which, when a focus position has changed in the imaging device, a portion of the virtual object behind the focus position is displayed in a normal display mode, and the remaining portion is hidden or its display mode is changed. (13) The image processing method according to any one of (6) to (12), in which the control condition includes at least one of the virtual object entering a predetermined range, the virtual object entering an angle of view of the imaging device, and a condition assigned to the virtual object being satisfied. (14) The image processing method according to (13), in which, when the virtual object enters a predetermined range, the portion of the virtual object within the range is hidden or its display mode is changed. (15) The image processing method according to (13) or (14), wherein, when the virtual object enters the angle of view of the imaging device, the transparency of a portion of the virtual object that enters the angle of view is changed. (16) The image processing method according to any of (6) to (15), wherein the control condition includes that the distance between a real object and the virtual object is within a predetermined distance. (17) The image processing method according to (16), wherein, when the distance between the real object and the virtual object is within the predetermined distance, the display mode of the virtual object that is within the predetermined distance is changed. (18) The image processing method according to any of (1) to (17), wherein, when the control condition is met, the transparency of the virtual object is changed over time. (19) An image processing device comprising: a display control unit that controls, based on the control condition, the display of a virtual object to be composited with a captured image captured by an imaging device. (20) An image processing system including an imaging device and an image processing device, wherein the imaging device includes an imaging unit that obtains an image by imaging a real space, and the image processing device includes a display control unit that controls, based on control conditions, the display of a virtual object to be combined with the image captured by the imaging device.

[0185] REFERENCE SIGNS LIST 1 Image processing system 2 Imaging device 3 Image processing device 7 Monitor 15 Display unit 51 CG generation unit 54 Condition determination unit 55 Image processing unit 56 Second composition unit

Claims

1. An image processing method for controlling the display of a virtual object to be synthesized into an image captured by an imaging device based on control conditions.

2. The image processing method according to claim 1, further comprising controlling whether or not to display the virtual object in accordance with the control condition.

3. The image processing method according to claim 1, further comprising controlling a display mode of the virtual object in accordance with the control condition.

4. The image processing method according to claim 1, further comprising controlling display of a virtual object to be composited with the captured image based on a control condition corresponding to a set mode among a plurality of modes having different control conditions.

5. The image processing method according to claim 4, wherein the plurality of modes include any one of a camera position and orientation mode in which control conditions are set based on the position and orientation of an imaging device, an imaging control mode in which control conditions are set based on imaging information of the imaging device, a virtual object mode in which control conditions are set based on a virtual object, and a real object mode in which control conditions are set based on a real object.

6. The image processing method according to claim 1, wherein the control conditions include at least one of the following: a position or orientation of the imaging device has changed; a distance between the imaging device and the virtual object is within a predetermined threshold; and the imaging device has entered a predetermined range.

7. The image processing method according to claim 6, further comprising the steps of: when the distance between said imaging device and said virtual object is within a predetermined threshold, making the part of said virtual object within said threshold invisible or changing the display mode thereof.

8. The image processing method according to claim 6, wherein when the imaging device is within a predetermined range, the virtual object is not displayed or its display mode is changed.

9. The image processing method according to claim 1, wherein the control condition includes a change in focus position in the imaging device.

10. The image processing method according to claim 9, wherein, when a focus position is changed in the imaging device, a portion of the virtual object corresponding to the depth of field is displayed in a normal display mode, and the portion other than the portion is hidden or the display mode is changed.

11. The image processing method according to claim 9, wherein, when a focus position is changed in the imaging device, a portion of the virtual object that is in front of the focus position is displayed in a normal display mode, and the portion other than the focus position is hidden or the display mode is changed.

12. The image processing method according to claim 9, wherein, when a focus position is changed in the imaging device, a portion of the virtual object behind the focus position is displayed in a normal display mode, and the portion other than the focus position is hidden or the display mode is changed.

13. The image processing method according to claim 1, wherein the control conditions include at least one of the following: the virtual object enters a predetermined range; the virtual object enters the angle of view of the imaging device; and a condition assigned to the virtual object is satisfied.

14. The image processing method according to claim 13, wherein, when the virtual object enters a predetermined range, the part of the virtual object within the range is hidden or its display mode is changed.

15. The image processing method according to claim 13, further comprising changing a transparency of a portion of said virtual object that is within the angle of view of said imaging device when said virtual object is within said angle of view.

16. The image processing method according to claim 1, wherein the control conditions include a condition that the distance between a real object and the virtual object is within a predetermined distance.

17. The image processing method according to claim 16, further comprising the step of changing a display mode of the virtual object within a predetermined distance when the distance between the real object and the virtual object is within a predetermined distance.

18. The image processing method according to claim 1, further comprising the step of: changing the transparency of the virtual object over time when the control condition is satisfied.

19. An image processing device comprising: a display control unit that controls display of a virtual object to be combined with an image captured by an imaging device based on a control condition.

20. An image processing system comprising an imaging device and an image processing device, wherein the imaging device comprises an imaging unit that obtains an image by imaging a real space, and the image processing device comprises a display control unit that controls the display of a virtual object to be composited into the image captured by the imaging device based on control conditions.

Images