Image processing method, image processing device, and image processing system

Abstract

In an image processing method according to the present technology: a composited image in which a virtual object is superimposed on an image-captured image obtained by an image-capturing device is displayed on a display unit; and assist display, which is a display for assisting image-capturing with respect to the virtual object in the composited image, is performed.

Description

Image processing method, image processing device, and image processing system

[0001] The present technology relates to an image processing method, an image processing device, and an image processing system, and in particular to a technology for superimposing a virtual object on a captured image.

[0002] For example, as disclosed in Patent Document 1, a system has been proposed in which an image captured by a camera is combined in real time with a CG image generated by 3DCG (Three Dimensional Computer Graphics) processing, and the combined composite image (so-called AR image) is displayed on a camera monitor. In this system, the composite image can be displayed on the camera monitor in real time, allowing the cameraman to check the composite image in real time and operate the camera while viewing the composite image.

[0003] JP 2011-35638 A

[0004] In the technology disclosed in Patent Document 1, when a cameraman adjusts the focus while checking the composite image, it is difficult to focus on a virtual object displayed in a CG image simply by checking the composite image. Therefore, there is a need to improve the convenience of the composite image for the cameraman when taking a picture.

[0005] The present technology has been made in view of the above circumstances, and aims to improve the convenience of composite images for users when capturing images.

[0006] The image processing method according to the present technology displays, on a display unit, a composite image in which a virtual object is superimposed on a captured image obtained by an imaging device, and performs an assist display, which is a display for assisting in capturing the virtual object in the composite image. According to the image processing method, for example, when focusing on the virtual object, the assist display allows the photographer to easily confirm the portion that is in focus.

[0007] FIG. 1 is a diagram showing an overview of an image processing system according to a first embodiment. FIG. 2 is a diagram explaining a captured image, a CG image, and a composite image. FIG. 3 is a diagram showing the configuration of an imaging device. FIG. 4 is a block diagram showing an example configuration of an image processing device. FIG. 5 is a diagram showing the functional configuration of an image processing system. FIG. 6 is a diagram explaining an assist display. FIG. 7 is a diagram showing an example of a composite image for a monitor in which real object peaking display, virtual object peaking display, and virtual object texture display are performed. FIG. 8 is a flowchart showing the flow of assist display processing. FIG. 9 is a diagram showing a modified example of assist display in the first embodiment. FIG. 10 is a diagram showing a modified example of assist display in the first embodiment. FIG. 11 is a diagram showing the overall configuration of an imaging system according to a second embodiment. FIG. 12 is a diagram explaining a background video, a captured image, and a CG image. FIG. 13 is a diagram explaining a composite image. FIG. 14 is a diagram showing the functional configuration of an imaging system. FIG. 15 is a diagram explaining an example of assist display on a virtual object. FIG. 16 is a diagram explaining a modified example of assist display on a virtual object.

[0008] Hereinafter, with reference to the accompanying drawings, embodiments of a sensor device according to the present technology will be described in the following order. <1. First Embodiment> <2. Second Embodiment> <3. Summary of Embodiments> <4. Present Technology> Note that, in the present technology, "imaging" not only refers to imaging accompanied by recording of image data, but also includes imaging for displaying an image on a display unit without recording image data, such as so-called through images or live view images. "Image" refers to a moving image. Furthermore, "image" does not only refer to a state in which the image is displayed on a display unit, but may also refer to image data that is not displayed on a display unit. "Real object" refers not only to an object (subject) captured by an imaging device, but also includes the image of the subject captured in the image. "Virtual object" refers not only to a virtual object generated by 3DCG processing, but also includes a virtual object captured in a CG image generated by rendering.

[0009] 1. First Embodiment [1.1. Overall Configuration of Image Processing System] Fig. 1 is a diagram showing an overview of an image processing system 1 as a first embodiment. Fig. 2 is a diagram explaining a captured image, a CG image, and a composite image. As shown in Fig. 1, the image processing system 1 as an embodiment is configured to include an image capturing device 2 and an image processing device 3.

[0010] The imaging device 2 captures an image of a real object 5 (a person in this case) as a subject in response to an operation by a cameraman 4, thereby generating a captured image (moving image) 101 as shown in FIG. 2A.

[0011] The image processing device 3 is configured by, for example, a workstation. The image processing device 3 may be located at the imaging site or at a location different from the imaging site. The image processing device 3 is connected to the imaging device 2 wirelessly or via a wire, and receives the captured image 101 from the imaging device 2 and transmits the composite image 103 to the imaging device 2.

[0012] The image processing device 3 generates a CG image 102 containing a virtual object 6 (here, a car) as shown in Fig. 2B. The image processing device 3 then composites the CG image 102 (virtual object 6) with the captured image 101 to generate a composite image 103 as shown in Fig. 2C. Note that the composite image 103 may also include frames in which the CG image 102 is not composited. The image processing device 3 then transmits the generated composite image 103 to, for example, one or both of the image capture device 2 and the monitor 7.

[0013] When the imaging device 2 receives the composite image 103 from the image processing device 3, it displays the received composite image 103 on the display unit 15. When the monitor 7 receives the composite image 103 from the image processing device 3, it displays the received composite image 103. As a result, in the image processing system 1, it becomes possible for the cameraman 4 to check the composite image 103 as a moving image in almost real time while the imaging device 2 is capturing an image, that is, an image in which the virtual object 6 is superimposed on the captured image 101.

[0014] [1.2. Overall Configuration of Imaging Device] Fig. 3 is a diagram showing the configuration of the imaging device 2. As shown in Fig. 3, the imaging device 2 includes a lens system 11, an imaging unit 12, an image processing unit 13, a recording unit 14, a display unit 15, a camera control unit 16, a memory unit 17, a driver unit 18, a sensor unit 19, a communication unit 20, and an operation unit 21. Note that, hereinafter, all of the units constituting the imaging device 2 except for the display unit 15 will be collectively referred to as a main body unit 2A.

[0015] The lens system 11 includes lenses such as a zoom lens and a focus lens, an aperture mechanism, and the like. The lens system 11 also includes a motor for operating these lenses and the aperture mechanism. The lens system 11 focuses light (incident light) from the real object 5 onto the imaging unit 12. The lens system 11 may be provided integrally with the imaging device 2, or may be configured as an interchangeable lens separate from the imaging device 2.

[0016] The imaging unit 12 includes an image sensor (imaging element), such as a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled Device) type. The imaging unit 12 performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, etc. on the electrical signal obtained by photoelectrically converting light received by the image sensor, and further performs A / D (Analog / Digital) conversion processing. The imaging unit 12 then outputs the captured image signal as digital data to the image processing unit 13.

[0017] The image processing unit 13 is configured as an image processor, for example, a DSP (Digital Signal Processor), etc. The image processing unit 13 performs various signal processing on the captured image signal from the imaging unit 12. The image processing unit 13 performs, for example, pre-processing, synchronization processing, YC generation processing, resolution conversion processing, file formation processing, etc.

[0018] In the pre-processing, the captured image signal from the imaging unit 12 is subjected to clamping processing, which clamps the R, G, and B black levels to a predetermined level, and correction processing between the R, G, and B color channels. In the synchronization processing, a color separation processing is performed so that the image data for each pixel contains all R, G, and B color components. For example, in the case of an image sensor using a Bayer array color filter, demosaic processing is performed as the color separation processing. In the YC generation processing, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data. In the resolution conversion processing, a resolution conversion process is performed on the image data that has been subjected to various signal processing.

[0019] In the file creation process, the image data that has undergone the various processes described above is subjected to compression encoding for recording or communication, formatting, and the generation and addition of metadata to generate files for recording or communication. For example, an image file is generated in the MP4 format used for recording moving images and audio that conforms to MPEG-4. It is also possible to generate an image file as raw image data.

[0020] The recording unit 14 is a memory card (such as a portable flash memory) that is a recording medium that can be attached to or detached from the imaging device 2, or a flash memory or HDD (hard disk drive) that is built into the imaging device 2. The recording unit 14 records the image data output from the image processing unit 13.

[0021] The display unit 15 executes various displays on the display screen based on instructions from the camera control unit 16. For example, the display unit 15 displays a playback image of image data read from the recording unit 14. The display unit 15 also displays a composite image 103 transmitted from the image processing device 3 in accordance with instructions from the camera control unit 16. As a result, the composite image 103 during composition confirmation, video recording, etc. is displayed on the display unit 15 as a so-called through image. The display unit 15 may also execute displays on the screen of various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the camera control unit 16.

[0022] The camera control unit 16 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit). The memory unit 17 stores information used by the camera control unit 16 for processing. The memory unit 17 collectively refers to, for example, a read-only memory (ROM), a random access memory (RAM), a flash memory, and the like. The memory unit 17 may be a memory area built into the microcomputer chip that serves as the camera control unit 16, or may be configured by a separate memory chip. The camera control unit 16 controls the entire imaging device 2 by executing a program stored in the ROM, flash memory, or the like of the memory unit 17. The camera control unit 16 includes functional units serving as an imaging control unit 16a and a self-position and orientation estimation unit 16b.

[0023] The imaging control unit 16a controls the operations of various units required for imaging, such as controlling the shutter speed of the imaging unit 12, issuing instructions for various signal processing in the image processing unit 13, imaging and recording operations in response to operations by the cameraman 4, playback of recorded image data, and control operations of the lens system 11 such as zooming, focusing, and aperture adjustment. The imaging control unit 16a also calculates the angle of view, aperture value, lens distortion, and focal length of the imaging device 2 as lens information based on the specifications of each lens and aperture mechanism constituting the lens system 11, movement of the focus lens and zoom lens, and operation information such as opening and closing of the aperture mechanism. Note that known methods can be used to calculate this lens information, and therefore a description thereof will be omitted.

[0024] The self-position and orientation estimation unit 16b estimates the position and orientation of the imaging device 2 based on detection information (in this example, triaxial acceleration information and angular velocity information) from the sensor unit 19 and images captured by the imaging unit 12. Examples of position and orientation estimation methods include SLAM (Simultaneous Localization and Mapping) and VIO (Visual Inertial Odometry). For example, VIO is assumed to estimate the position and orientation using a technology such as an INS (Inertial Navigation System) using the output of an IMU (sensor unit 19) that has a higher output rate than the image sensor (imaging unit 12). Note that methods such as VO (Visual Odometry) that do not use IMU detection information may also be employed.

[0025] The RAM in the memory unit 17 is used as a working area for temporarily storing data, programs, etc. when the CPU of the camera control unit 16 processes various data. The ROM and flash memory (non-volatile memory) in the memory unit 17 are used to store an OS (Operating System) for the CPU to control each unit, application programs for various operations, firmware, various setting information, etc. The various setting information includes exposure settings, shutter speed settings, and mode settings as setting information related to imaging operations, white balance settings, color settings, and settings related to image effects as setting information related to image processing, and custom key settings and display settings as setting information related to operability.

[0026] The driver unit 18 is provided with, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for an aperture mechanism motor, etc. These motor drivers apply drive currents to the corresponding drivers in response to instructions from the camera control unit 16, and cause the focus lens or zoom lens to move, open or close the aperture blades of the aperture mechanism, etc.

[0027] The sensor unit 19 collectively refers to various sensors mounted on the imaging device 2. The sensor unit 19 may be, for example, an IMU (Inertial Measurement Unit). The IMU has a triaxial acceleration sensor and an angular velocity sensor, and outputs triaxial acceleration information and angular velocity information.

[0028] The communication unit 20 is capable of performing wired or wireless inter-device communication and network communication, which is communication with an external device via a predetermined communication network such as the Internet. The communication unit 20 transmits the captured image 101 to the image processing device 3, for example, and receives the composite image 103.

[0029] The operation unit 21 collectively represents input devices used by the user (cameraman 4) to input various operations. Specifically, the operation unit 21 includes various operators (keys, dials, touch panels), a touch panel, etc. When the operation unit 21 detects a user operation, a signal corresponding to the input operation is sent to the camera control unit 16.

[0030] [1.3. Configuration of Image Processing Device] Fig. 4 is a block diagram showing an example configuration of the image processing device 3. As shown in Fig. 4, the image processing device 3 includes a CPU 31. The CPU 31 executes various processes in accordance with a program stored in a ROM 32 or a program loaded from a storage unit 39 to a RAM 33. The RAM 33 also stores data necessary for the CPU 31 to execute various processes as appropriate. The CPU 31, the ROM 32, and the RAM 33 are connected to one another via a bus 43.

[0031] A GPU (Graphics Processing Unit) 34 is also connected to the bus 43. The GPU 34 performs 3DCG processing mainly in real time in accordance with drawing commands from the CPU 31.

[0032] An input / output interface (I / F) 35 is also connected to the bus 43. An input unit 36 ​​including controls and operation devices is connected to the input / output interface 35. For example, the input unit 36 ​​may be various controls and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, or a remote controller. An operation is detected by the input unit 36, and a signal corresponding to the detected operation is interpreted by the CPU 31.

[0033] A display unit 37, such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 38, such as a speaker, are also connected to the input / output interface 35, either integrally or separately. The display unit 37 is used to display various types of information, and may be configured, for example, by a display device provided in the housing of the image processing device 3 or a separate display device connected to the image processing device 3. The display unit 37 may be the monitor 7.

[0034] The display unit 37 displays images for various image processes, moving images to be processed, etc. on the display screen based on instructions from the CPU 31. The display unit 37 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 31.

[0035] The input / output interface 35 is connected to a storage unit 39 and a communication unit 40, which are configured from an SSD (Solid State Drive) or an HDD (Hard Disk Drive).

[0036] The communication unit 40 is capable of performing wired or wireless inter-device communication, and network communication, which is communication with an external device via a predetermined communication network such as the Internet. In particular, in this embodiment, the communication unit 40 is configured to be able to perform data communication with the communication unit 20 of the imaging device 2.

[0037] A drive 41 is connected to the input / output interface 35 as required, and a removable recording medium 42 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon.

[0038] The drive 41 can read data files such as programs used in various processes from the removable recording medium 42. The read data files are stored in the storage unit 39, and images and sounds contained in the data files are output on the display unit 37 and the audio output unit 38. Furthermore, the computer programs and the like read from the removable recording medium 42 are installed in the storage unit 39 as needed.

[0039] [1.4. Functional Configuration of Image Processing System] Fig. 5 is a diagram showing the functional configuration of the image processing system 1. As shown in Fig. 5, the CPU 31 and GPU 34 of the image processing device 3 function as a CG generation unit 51 and a first composition unit 52. Note that these functional units may be configured to function by only one of the CPU 31 and the GPU 34. Furthermore, one functional unit may be configured to function by both the CPU 31 and the GPU 34.

[0040] In the image processing system 1, the captured image 101 captured by the imaging unit 12 of the imaging device 2 (more precisely, the captured image 101 subjected to image processing by the image processing unit 13) is transmitted to the image processing device 3 via the communication unit 20.

[0041] Furthermore, the imaging device 2 transmits camera information 109 obtained while capturing the captured image 101 to the image processing device 3 via the communication unit 20. The camera information 109 may be metadata assigned to the captured image 101, or may be information independent of the captured image 101. The camera information 109 includes the position, orientation, angle of view, aperture value, lens distortion, focal length, etc. of the imaging device 2. This information is transmitted, for example, for each frame constituting the captured image 101. The position and orientation of the imaging device 2 are estimated for each frame by the self-position and orientation estimation unit 16b. Furthermore, the angle of view, aperture value, lens distortion, and focal length of the imaging device 2 are calculated by the imaging control unit 16a as lens information, as described above.

[0042] When the image processing device 3 receives the captured image 101 and the camera information 109 , it inputs the captured image 101 to the first synthesis unit 52 and inputs the camera information 109 to the CG generation unit 51 .

[0043] The CG generation unit 51 includes a vertex shader unit 51 a, a pixel shader unit 51 b, a blur expression unit 51 c, and a first image processing unit 51 d. However, the CG generation unit 51 is not limited to this, and may have another configuration as long as it can generate the CG image 102 from a 3D model.

[0044] Upon receiving the camera information 109, the vertex shader unit 51a sets the angle of view, aperture value, lens distortion, and focal length of a virtual camera that virtually captures an image in a virtual space represented by a CG model, based on the angle of view, aperture value, lens distortion, and focal length included in the camera information 109. Here, the settings of the lens system of the virtual camera are matched to the settings of the lens system 11 of the imaging device 2.

[0045] Furthermore, the vertex shader unit 51a updates the position and orientation of the virtual camera based on the position and orientation of the image capture device 2 included in the camera information 109. Here, the CG model indicates the position (coordinate position) of the virtual object 6 in a coordinate system based on a predetermined position in the virtual space. Furthermore, the coordinate system of the real space captured by the image capture device 2 is adjusted to match the coordinate system of the virtual space indicated by the CG model. Therefore, by matching the position and orientation of the virtual camera to the position and orientation of the image capture device 2, the imaging range of the imaging device 2 and the imaging range of the virtual camera can be matched. Next, the vertex shader unit 51a performs processing such as converting the shape of the virtual object 6 when viewed from the position and orientation of the virtual camera.

[0046] The pixel shader unit 51b performs processing (shading) to determine the final color for each pixel. The shading processing includes processes such as texture mapping and lighting. In addition, various other processes are performed on a fragment-by-fragment basis.

[0047] The blur representation unit 51c calculates the focus position and depth of field of the imaging device 2 (virtual camera) based on the angle of view, aperture value, lens distortion, focal distance, etc. of the imaging device 2 included in the camera information. The blur representation unit 51c then performs processing to represent image blur in parts of the virtual object 6 that are outside the depth of field. As a result, a CG image 102 is generated in which the virtual object 6 is depicted with the blur representation applied by the blur representation unit 51c. The generated CG image 102 is output to the first synthesis unit 52 and the first image processing unit 51d.

[0048] The first composition unit 52 composes the captured image 101 transmitted from the imaging device 2 with the CG image 102 generated by the CG generation unit 51, thereby generating a composite image 103 in which the virtual object 6 is superimposed on the captured image 101. Note that, in the present embodiment, an example is described in which the virtual object 6 is located closer to the viewer than the real object 5, but the virtual object 6 may also be located further back than the real object 5. In such a case, the captured image 101 may include information on the distance to the real object 5 for each pixel, and the CG image 102 may include information on the distance to the virtual object 6 for each pixel, and the composite image 103 may be generated so that the pixels on the closer side are preferentially left when combining the images.

[0049] In this way, the image processing system 1 generates in real time a composite image 103 in which the virtual object 6 is superimposed on the captured image 101. The composite image 103 generated here is stored in the storage unit 39 as, for example, the composite image 103 to be ultimately retained.

[0050] Incidentally, if the cameraman 4 wants to focus on a virtual object 6 that does not exist in real space, he or she may display the composite image 103 on the display unit 15 or the monitor 7 and operate the focus of the imaging device 2 while checking the composite image 103.

[0051] However, it is difficult to perform a focus operation to focus on the virtual object 6 (including a specific position on the virtual object 6) while checking the composite image 103.

[0052] Therefore, in the image processing system 1, a monitor composite image 104 is displayed on the display unit 15 and the monitor 7, in which an assist display is added to the composite image 103 generated as described above.

[0053] First, a description will be given of the functional units for generating the composite image for monitor 104. However, it goes without saying that the functional units described above also function as functional units for generating the composite image for monitor 104.

[0054] The first image processing unit 51d processes the virtual object 6 that has been blurred by the blur expression unit 51c to provide an assist display, and outputs a CG image 102 that shows the virtual object 6 to the imaging device 2. Examples of assist displays include peaking display and texture display. Assist displays will be described in detail later. However, there are cases where the first image processing unit 51d does not process the CG image 102 input from the blur expression unit 51c. In such cases, the unprocessed CG image 102 is output to the imaging device 2 as is.

[0055] The camera control unit 16 of the imaging device 2 functions as a second image processing unit 53 and a second composition unit 54 in addition to the imaging control unit 16a and self-position and orientation estimation unit 16b described above.

[0056] The second image processing unit 53 receives the captured image 101 from the image processing unit 13 and recognizes the real object 5 from the received captured image 101. The second image processing unit 53 then performs processing on the recognized real object 5 to provide an assist display, and outputs the processed captured image 101 to the second composition unit 56. An example of the assist display is a peaking display. The assist display will be described in detail later. However, there are cases where the second image processing unit 53 does not perform processing on the captured image 101 received from the image processing unit 13. In such cases, the unprocessed captured image 101 is output as is to the second composition unit 56.

[0057] The second composition unit 54 generates a composite image for monitoring 104 by combining the captured image 101 input from the second image processing unit 53 and the CG image 102 input from the first image processing unit 51d. The generated composite image for monitoring 104 is output to the display unit 15 and the monitor 7. The combined composite image for monitoring 104 is then displayed on the display unit 15 and the monitor 7 of the imaging device 2.

[0058] [1.5. Explanation of Assist Display] Fig. 6 is a diagram illustrating the assist display. The assist display is a display for assisting in imaging, and includes a peaking display and a texture display. The peaking display highlights the edge portions of the subject in the image that are in focus. The peaking display can be performed on both the real object 5 and the virtual object 6 in the composite image 103. In other words, the peaking display can be performed on both the captured image 101 and the CG image 102.

[0059] When performing peaking display on the captured image 101 (real object 5), the second image processing unit 53 detects edges of the real object 5 in the captured image 101 and detects in-focus edge portions of the detected edges. The in-focus edge portions may be detected by calculating high-frequency components in the captured image 101, or may be detected based on the focus position and depth of field of the imaging device 2 and the distance to the real object 5 calculated by SLAM or the like. When the second image processing unit 53 detects in-focus edge portions, it generates a captured image 101 in which the edge portions are highlighted as real object peaking display 105, as shown in FIG. 6A . Possible methods for highlighting the edge portions include displaying the edge portions with lines of a predetermined color or increasing the brightness of the edge portions. When performing the real object peaking display 105, the second composition unit 54 displays a guide 110 on the composite image for monitoring 104, indicating that the real object peaking display 105 is being performed. The guide 110 may, for example, state "real object." This allows the cameraman 4 to easily confirm the part of the real object 5 that is in focus.

[0060] When performing peaking display on the CG image 102 (virtual object 6), the first image processing unit 51d detects edges of the virtual object 6 in the CG image 102 and detects in-focus edge portions of the detected edges. The in-focus edge portions may be detected by calculating high-frequency components in the CG image 102, or may be detected based on the focus position and depth of field of the imaging device 2 and the distance from the imaging device 2 (virtual camera) to the virtual object 6. When the first image processing unit 51d detects the in-focus edge portions, it generates a CG image 102 in which the edge portions are highlighted as a virtual object peaking display 106, as shown in FIG. 6B . Possible methods for highlighting the edge portions include displaying the edge portions with lines of a predetermined color or increasing the brightness of the edge portions. When performing the virtual object peaking display 106, the second composition unit 54 displays a guide 111 on the composite image for monitoring 104, indicating that the virtual object peaking display 106 is being performed. The guide 111 has, for example, the mark "CG: Edge" written on it, which allows the cameraman 4 to easily confirm the part of the virtual object 6 that is in focus.

[0061] Texture display refers to highlighting texture portions that are in focus among the surfaces constituting the subject in the image. Texture display can be performed on the virtual object 6 in the composite image 103. In other words, texture display can be performed on the CG image 102. When performing texture display on the CG image 102 (virtual object 6), the first image processing unit 51d detects in-focus texture portions of the texture of the virtual object 6 in the CG image 102. Upon detecting the in-focus texture portion, the first image processing unit 51d generates a CG image 102 in which the texture portion is highlighted as a virtual object texture display 107, as shown in FIG. 6C . Possible highlighting methods include displaying the texture portion in a predetermined pattern (e.g., a zebra pattern, a specific color, or hatching). When performing the virtual object texture display 107, the second composition unit 54 displays a guide 112 on the composite image for monitoring 104, indicating that the virtual object texture display 107 is being performed. The guide 112 may, for example, be labeled "CG:Tex." This allows the cameraman 4 to easily confirm the part of the virtual object 6 that is in focus.

[0062] 7 shows an example of a composite image for monitoring 104 in which a real object peaking display 105, a virtual object peaking display 106, and a virtual object texture display 107 are performed. The above-described real object peaking display 105, virtual object peaking display 106, and virtual object texture display 107 can be performed simultaneously. The second composition unit 54 combines the captured image 101 in which the real object peaking display 105 is performed with the CG image 102 in which the virtual object peaking display 106 and the virtual object texture display 107 are performed. As a result, a composite image for monitoring 104 in which the real object peaking display 105, the virtual object peaking display 106, and the virtual object texture display 107 are performed, as shown in FIG. 7, for example, is generated. The generated composite image for monitoring 104 is then displayed on the display unit 15 and the monitor 7. Guides 110, 111, and 112 are also displayed in this composite image for monitoring 104.

[0063] It is also possible to perform any combination of two of the real object peaking display 105, virtual object peaking display 106, and virtual object texture display 107 on the composite image for monitoring 104.

[0064] In this way, when the real object peaking display 105 and the virtual object peaking display 106 are displayed simultaneously, the first image processing unit 51d and the second image processing unit 53 differentiate the color or thickness (width) of the lines, as shown in Fig. 7. That is, the display modes of the real object peaking display 105 and the virtual object peaking display 106 are differentiated. This allows the cameraman 4 to easily understand whether the peaking display is for the real object 5 or the virtual object 6.

[0065] The first image processing unit 51 d and the second image processing unit 53 may alternately switch between the real object peaking display 105 and the virtual object peaking display 106 and the virtual object texture display 107. Furthermore, the first image processing unit 51 d and the second image processing unit 53 may associate the real object peaking display 105 with the virtual object peaking display 106 and the virtual object texture display 107, respectively, with different operations of the operation unit 21.

[0066] The first image processing unit 51 d and the second image processing unit 53 may alternately switch between the real object peaking display 105, the virtual object peaking display 106 and virtual object texture display 107, and the real object peaking display 105, the virtual object peaking display 106 and virtual object texture display 107. In this case, the real object peaking display 105, the virtual object peaking display 106 and virtual object texture display 107, and the real object peaking display 105, the virtual object peaking display 106 and virtual object texture display 107 may be displayed in different display modes. Furthermore, the first image processing unit 51 d and the second image processing unit 53 may associate the real object peaking display 105, the virtual object peaking display 106 and virtual object texture display 107, and the real object peaking display 105, the virtual object peaking display 106 and virtual object texture display 107 with different operations of the operation unit 21.

[0067] Here, if the real object peaking display 105, virtual object peaking display 106, and virtual object texture display 107 are displayed over a large area, they may be displayed large on the monitor composite image 104, making it difficult for the cameraman 4 to determine which area is in focus. Therefore, the second image processing unit 53 detects the edges of the real object 5 from the captured image 101 using a predetermined threshold value for determining edges, and detects in-focus edge portions among the detected edges. The second image processing unit 53 then divides the captured image 101 into small grids of multiple pixels and counts the number of small grids that include at least one in-focus edge pixel, thereby calculating the area in which the real object peaking display 105 is to be performed. If the area in which the real object peaking display 105 is to be performed is equal to or larger than a predetermined area, the second image processing unit 53 determines that the area in which the real object peaking display 105 is to be performed is large, and increases the threshold value. The second image processing unit 53 detects the edges of the real object 5 from the captured image 101 using the changed threshold value, and also detects in-focus edge portions of the detected edges. The second image processing unit 53 then performs real object peaking display 105 on the detected edge portions. In this way, if the area over which the real object peaking display 105 is performed is large, the area over which the real object peaking display 105 is performed can be reduced, making it possible to make the in-focus portions of the CG image 102 easier to understand for the cameraman 4. Note that the first image processing unit 51d, when performing the virtual object peaking display 106 and the virtual object texture display 107, changes the threshold value in the same way as the second image processing unit 53, making it possible to make it easier for the cameraman 4 to understand the in-focus portions of the CG image 102.

[0068] Furthermore, it is desirable to set different threshold values ​​for use by the second image processing unit 53 when detecting edges from the captured image 101 and the CG generation unit 51 when detecting edges from the CG image 102. Generally, the real object 5 has a lot of texture, while the virtual object 6 has little texture. Therefore, the threshold value for use by the CG generation unit 51 when detecting edges from the CG image 102 is set to a value smaller than the threshold value for use by the second image processing unit 53 when detecting edges from the captured image 101. This makes it possible to prevent the area of ​​the real object peaking display 105 from becoming too large or the area of ​​the virtual object peaking display 106 from becoming too small.

[0069] Furthermore, when the blur expression unit 51c does not express blur on the virtual object 6, the virtual object peaking display 106 and the virtual object texture display 107 may not be performed (assist display for the virtual object 6 may not be performed). When the blur expression on the virtual object 6 is not performed, the virtual object 6 is not likely to be focused on, and therefore, by not performing the assist display for the virtual object 6, confusion on the cameraman 4 can be reduced.

[0070] Furthermore, if the virtual object 6 is a non-assisted virtual object that does not require blurring, such as a panel displaying news information, a person's annotations, or a live-action video, the assist display for the virtual object 6 may not be performed. The non-assisted virtual object 6 is assigned attribute information in advance in the CG model indicating that it is a non-assisted virtual object. Even if no attribute information is assigned, the first image processing unit 51d may perform a fast Fourier transform on the virtual object 6 and determine that it is a non-assisted virtual object if it contains high-frequency components. The first image processing unit 51d does not perform the assist display for the non-assisted virtual object 6.

[0071] Furthermore, when all virtual objects 6 included in the CG model are present within the depth of field of the imaging device 2, the first image processing unit 51d may not display an assist display for the virtual objects 6. If an assist display is displayed when all virtual objects 6 included in the CG model are present within the depth of field of the imaging device 2, the assist display will be displayed for all virtual objects 6, which may make the composite image for monitoring 104 difficult to view. Therefore, by not displaying an assist display for the virtual objects 6 when all virtual objects 6 included in the CG model are present within the depth of field of the imaging device 2, the composite image for monitoring 104 can be made easier to view. However, a guide indicating that all virtual objects 6 included in the CG model are present within the depth of field of the imaging device 2 may be displayed.

[0072] The colors of the real object peaking display 105, the virtual object peaking display 106, and the virtual object texture display 107 may be changed depending on the surrounding colors. For example, if a predetermined color for the virtual object peaking display 106 is similar to the surrounding color of the virtual object peaking display 106, the first image processing unit 51d inverts the color of the virtual object peaking display 106. This makes it possible to reduce the difficulty in seeing the assist display.

[0073] Furthermore, when performing virtual object peaking display 106, if a predetermined color for virtual object peaking display 106 is similar to the color of the surrounding area of ​​virtual object peaking display 106, first image processing unit 51 d may perform virtual object texture display 107 instead of virtual object peaking display 106. Similarly, when performing virtual object texture display 107, if a predetermined color or pattern for virtual object texture display 107 is similar to the color or pattern of the surrounding area of ​​virtual object texture display 107, first image processing unit 51 d may perform virtual object peaking display 106 instead of virtual object texture display 107.

[0074] [1.5. Processing flow of assist display] Next, the processing flow of assist display will be described. Fig. 8 is a flowchart showing the processing flow of assist display. Note that Fig. 8 collectively shows the processing performed in parallel by the first image processing unit 51d and the second image processing unit 53.

[0075] 8, in step S1, the first image processing unit 51d and the second image processing unit 53 determine whether the depth of field calculated based on the camera information is within a predetermined value. Here, they determine whether the depth of field is shallow enough to cause blurring of the real object 5 or the virtual object 6.

[0076] If the depth of field is not within the predetermined value (Yes in step S1), the first image processing unit 51d and the second image processing unit 53 do not perform the assist display in step S2.

[0077] If the depth of field is not within the predetermined value (No in step S1), in step S3, the first image processing unit 51d and the second image processing unit 53 determine whether the virtual object 6 is expressed in a blurred manner.

[0078] If the virtual object 6 is not blurred (No in step S3), in step S4 the first image processing unit 51d does not provide assist displays (virtual object peaking display 106, virtual object texture display 107) for the virtual object 6. Also in step S4, the second image processing unit 53 provides assist displays (real object peaking display 105) for the real object 5.

[0079] If the virtual object 6 is expressed in a blurred manner (No in step S3), the first image processing unit 51d and the second image processing unit 53 determine in step S5 whether the virtual object 6 is a non-assisted virtual object.

[0080] If the virtual object 6 is a non-assisted virtual object (Yes in step S5), in step S6 the first image processing unit 51d performs virtual object peaking display 106 for the virtual object 6, but does not perform virtual object texture display 107. Also in step S6, the second image processing unit 53 performs assist display (real object peaking display 105) for the real object 5.

[0081] If the virtual object 6 is not a non-assisted virtual object (No in step S5), in step S7 the first image processing unit 51d performs an assist display (virtual object peaking display 106, virtual object texture display 107) for the virtual object 6. Also in step S7, the second image processing unit 53 performs an assist display (real object peaking display 105) for the real object 5.

[0082] [1.6. Modifications of the First Embodiment] The first embodiment is not limited to the specific example described above, and various modifications may be made. For example, the image capture device 2 provides assist display for the real object 5, and the image processing device 3 provides assist display for the virtual object 6. However, the image capture device 2 or the image processing device 3 may provide assist display for both the real object 5 and the virtual object 6.

[0083] In the first embodiment, the assist displays are the real object peaking display 105, the virtual object peaking display 106, and the virtual object texture display 107. However, the assist displays are not limited to these, and may be other displays that assist the cameraman 4 in the imaging operation.

[0084] For example, as shown in Fig. 9, the second image processing unit 53 may assist in displaying a brightness (Exposure) distribution 120 of the real object 5 and the virtual object 6. Fig. 9A illustrates a case where the brightness of the real object 5 and the virtual object 6 is approximately the same, Fig. 9B illustrates a case where the brightness of the virtual object 6 is low, and Fig. 9C illustrates a case where the brightness of the real object 5 is low. By displaying the brightness distribution 120 as an assist in this way, the cameraman 4 can easily adjust the aperture. Furthermore, the brightness of the virtual object 6 can be reduced.

[0085] 10 , the second image processing unit 53 may be configured to assist in displaying color histogram distributions 121 of the real object 5 and the virtual object 6. Note that FIG. 10A illustrates a case where the brightness of the real object 5 and the virtual object 6 is approximately the same, FIG. 10B illustrates a case where the real object 5 is whitish, and FIG. 10C illustrates a case where the virtual object 6 is whitish. By displaying the color histogram distributions 121 as an assistive display in this way, the cameraman 4 can easily adjust the white balance. Furthermore, the cameraman 4 can also adjust the white balance of the virtual object 6.

[0086] 11, the second image processing unit 53 may display one or both of the real object 5 and the virtual object 6 in false color as an assist display. Note that FIG. 11A shows the real object 5 and the virtual object 6 not displayed in false color, FIG. 11B shows the real object 5 only displayed in false color, FIG. 10C shows the virtual object 6 only displayed in false color, and FIG. 11D shows the real objects 5 and 6 displayed in false color. Displaying the real objects 5 and 6 in false color in this manner allows the cameraman 4 to easily adjust the brightness. Furthermore, the cameraman 4 can adjust the brightness of the virtual object 6.

[0087] 2. Second Embodiment In the second embodiment, in a studio equipped with a large display device, a background image is displayed on the display device, and performers perform in front of the image, thereby providing an assist display in a so-called virtual production in which the performers and the background can be filmed.

[0088] [2.1. Overall Configuration of the Imaging System] Fig. 12 is a diagram showing the overall configuration of an imaging system 500 according to a second embodiment. As shown in Fig. 12, the imaging system 500 is provided with a performance area 501 where a performer 510, who is an example of a real object 5, performs a performance or other performance. Large display devices are disposed at least on the rear surface, left and right sides, and top surface of this performance area 501. The type of display device is not limited, but the figure shows an example in which an LED wall 505 is used as an example of a large display device.

[0089] A single LED wall 505 is formed by arranging a plurality of LED panels 506 connected vertically and horizontally to form a large panel. The size of the LED wall 505 is not particularly limited, but it may be of a size necessary or sufficient to display the background when filming the performer 510.

[0090] A required number of lights 580 are placed at required positions, such as above or to the side of the performance area 501, to illuminate the performance area 501.

[0091] Near the performance area 501, an imaging device 2 is placed for capturing images of movies and other video content.

[0092] The imaging device 2 captures both the performer 510 in the performance area 501 and the image displayed on the LED wall 505. For example, by displaying a landscape as background image vB on the LED wall 505, it is possible to capture a moving image that is similar to the image of the performer 510 actually being present in the location of that landscape and performing.

[0093] A monitor 7 is placed near the performance area 501. The monitor 7 displays in real time the captured images 101 captured by the imaging device 2. This allows the director, staff, and cameraman 4 who are producing the video content to check the captured images 101.

[0094] In this way, the imaging system 500 that films the performance of the performer 510 in a film studio with the LED wall 505 as a backdrop has various advantages over green screen filming.

[0095] For example, when shooting on a green screen, it can be difficult for the actor to imagine the background or the situation of the scene, which can affect their performance. In contrast, displaying the background image vB makes it easier for the actor 510 to act, improving the quality of their performance. It also makes it easier for the director and other staff to determine whether the actor 510's performance matches the background and the situation of the scene.

[0096] In addition, post-production is more efficient than with green screen shooting. This is because in some cases, so-called chromakey compositing is not necessary, and color correction and reflection compositing are not required. Even if chromakey compositing is required during shooting, the fact that an additional background screen is not required also contributes to efficiency.

[0097] When shooting on a green screen, the actors' bodies, costumes, and objects tend to have a green tint, which needs to be corrected. Also, when shooting on a green screen, if there are objects that reflect the surrounding scenery, such as glass, mirrors, or snow globes, it is necessary to generate and composite images of the reflections, which is a time-consuming process.

[0098] In contrast, when capturing an image using the imaging system 500, the green hue does not increase, and therefore correction for this is not necessary. In addition, by displaying the background image vB, reflections on actual objects such as glass are captured naturally, and therefore compositing of the reflection image is not necessary.

[0099] The imaging system 500 also includes a rendering engine 520 , an asset server 530 , a sync generator 540 , an operation monitor 550 , a camera tracker 560 , an LED processor 570 , a lighting controller 581 , and a display controller 590 .

[0100] The LED processor 570 is provided for each LED panel 506 and drives the corresponding LED panel 506 to display images.

[0101] The sync generator 540 generates a synchronization signal for synchronizing the frame timing of the image displayed by the LED panel 506 with the frame timing of the image captured by the imaging device 2, and supplies the synchronization signal to each LED processor 570 and the imaging device 2. However, this does not prevent the output from the sync generator 540 from being supplied to the rendering engine 520.

[0102] The camera tracker 560 generates camera information from the imaging device 2 at each frame timing and supplies it to the rendering engine 520. However, the camera information may be generated by the imaging device 2. For example, the camera tracker 560 detects, as one piece of camera information, the position of the LED wall 505 or position information of the imaging device 2 relative to a predetermined reference position, and the shooting direction of the imaging device 2, and supplies these to the rendering engine 520. A specific detection method using the camera tracker 560 is a method in which reflectors are randomly placed on the ceiling and the position is detected from the reflected light of infrared light irradiated from the imaging device 2 side. Another detection method is to estimate the self-position of the imaging device 2 by using gyro information installed on the camera platform or the main body of the imaging device 2, or by image recognition of the captured image 101 of the imaging device 2.

[0103] Furthermore, the imaging device 2 may supply the rendering engine 520 with camera information such as the angle of view, focal length, F-number, shutter speed, and lens information.

[0104] The asset server 530 is a server that stores 3D models (3D background data) on a recording medium and can read out the 3D models as needed, i.e., functions as a database (DB) for the 3D background data.

[0105] The rendering engine 520 performs processing to generate the background video vB to be displayed on the LED wall 505. To do this, the rendering engine 520 reads out the necessary 3D background data from the asset server 530. The rendering engine 520 then generates an outer frustum image to be used in the background video vB by rendering the 3D background data as viewed from pre-specified spatial coordinates. Furthermore, as processing for each frame, the rendering engine 520 specifies the viewpoint position and the like with respect to the 3D background data using camera information supplied from the camera tracker 560 and the imaging device 2, and renders the shooting area video vBC (inner frustum).

[0106] Furthermore, the rendering engine 520 generates a background image vB as one frame of video data by combining the previously generated outer frustum with the shooting area image vBC that has been rendered for each frame. The rendering engine 520 then transmits the generated one frame of video data to the display controller 590.

[0107] The display controller 590 generates a divided video signal nD by dividing one frame of video data into video portions to be displayed on each LED panel 506, and transmits the divided video signal nD to each LED panel 506. At this time, the display controller 590 may perform calibration according to individual differences / manufacturing errors in color development and the like between display sections. Note that the display controller 590 may not be provided, and these processes may be performed by the rendering engine 520. In other words, the rendering engine 520 may generate the divided video signal nD, perform calibration, and transmit the divided video signal nD to each LED panel 506.

[0108] Each LED processor 570 drives the LED panel 506 based on the divided video signal nD received, thereby displaying the overall background image vB on the LED wall 505. The background image vB includes the shooting area image vBC that is rendered according to the position of the imaging device 2 at that time, etc.

[0109] In this way, the imaging device 2 can capture the performance of the performer 510, including the background image vB displayed on the LED wall 505. The captured image 101 obtained by imaging with the imaging device 2 is recorded on a recording medium inside the imaging device 2 or in an external recording device (not shown).

[0110] The operation monitor 550 displays an operation image vOP for controlling the rendering engine 520. The engineer 511 can perform the necessary settings and operations related to the rendering of the background image vB while viewing the operation image vOP.

[0111] The lighting controller 581 controls the light emission intensity, light emission color, irradiation direction, etc. of the light 580. The lighting controller 581 may control the light 580 asynchronously with the rendering engine 520, for example, or may control the light 580 in synchronization with the shooting information or rendering processing. Therefore, the lighting controller 581 may control the light emission in response to instructions from the rendering engine 520 or a master controller (not shown).

[0112] The photographing system 500 is also provided with an image processing device 3 (indicated as a Computer in the figure). As in the first embodiment, the image processing device 3 generates a CG image 102 in which a virtual object 6 based on a 3D model is captured. The photographing device 2 then generates a composite image 103 by combining the photographed image 101 and the CG image 102, and displays the composite image 103 on the display unit 15 and the monitor 7.

[0113] FIG. 13 is a diagram illustrating a background image, a captured image, and a CG image. FIG. 14 is a diagram illustrating a composite image. In the imaging system 500, for example, as shown in FIG. 13A, a background image vB containing a virtual object 201 is projected onto an LED wall 505. When the imaging device 2 captures an image of the performer 510 and the LED wall 505 in this state, a captured image 101 is obtained in which the performer 510 (real object 5) appears in front of the virtual object 201 displayed in the background image vB, as shown in FIG. 13B. Furthermore, the image processing device 3 generates a CG image 102 containing a virtual object 6, as shown in FIG. 13C. Then, the captured image 101 shown in FIG. 13B and the CG image 102 shown in FIG. 13C are composited to generate a composite image 103 shown in FIG. 14.

[0114] Therefore, in the second embodiment, the background image vB projected on the LED wall 505 is also the virtual object 201. Therefore, it is useful to provide an assist display when focusing the imaging device 2 on the virtual object 201.

[0115] Therefore, in the photographing system 500, in addition to the real object 5 appearing in the captured image 101 and the virtual object 6 appearing in the CG image 102, an assist display is performed for the virtual object 201 based on the background image vB appearing in the captured image 101.

[0116] 15 is a diagram showing the functional configuration of the imaging system 500. Here, the configuration mainly relates to the assist display for the virtual object 201. Furthermore, a description of the same configuration as in the first embodiment will be omitted.

[0117] 15 , the functional configurations of the imaging device 2 and the image processing device 3 are the same as those in the first embodiment. The imaging device 2 is provided with a main body 2A, a display 15, a second image processing unit 53, and a second composition unit 54. The image processing device 3 is provided with a CG generation unit 51 and a first composition unit 52.

[0118] Camera information is input to a rendering engine 520 in the imaging system 500. A synchronization signal output from a sync generator 540 is input to the imaging device 2, image processing device 3, and cameraman 4 for synchronization.

[0119] [2.3. Explanation of Assist Display] FIG. 16 is a diagram illustrating an example of an assist display on a virtual object 201. When peaking display is performed on the virtual object 201, the rendering engine 520 detects a texture portion of the texture of the virtual object 201 in the background image vB that is in focus. The focus is calculated based on camera information. When the rendering engine 520 detects a texture portion that is in focus, it generates a background image vB in which the texture portion is highlighted as a virtual object texture display 210, as shown in FIG. 16A . Possible methods for highlighting the texture portion include displaying the texture portion in a predetermined pattern (e.g., a zebra pattern, a specific color, or hatching).

[0120] The generated background image vB is displayed on the LED wall 505 via the display controller 590 and the LED processor 570. Then, the background image vB with the virtual object texture display 210 applied thereto is captured by the imaging device 2, and ultimately, as shown in FIG. 16B , the monitor composite image 104 in which the virtual object texture display 210 is applied to the virtual object 201 is displayed on the display unit 15 and the monitor 7.

[0121] In this case, if the frame rate of the imaging device 2 and the LED wall 505 can be doubled, the rendering engine 520 generates, for example, a background image vB in which the virtual object texture display 210 is not displayed for even-numbered frames, and generates a background image vB in which the virtual object texture display 210 is displayed for odd-numbered frames. In other words, the rendering engine 520 alternates between displaying and not displaying the virtual object texture display 210 for each frame.

[0122] When the background video vB generated in this manner is displayed on the LED wall 505 and captured by the imaging device 2, the imaging device 2 divides the image into a captured image 101 of only even-numbered frames and a captured image 101 of only odd-numbered frames. The imaging device 2 then records the captured image 101 of only even-numbered frames as the main video. Meanwhile, the captured image 101 of only odd-numbered frames is composited with a CG image 102 to generate a composite image 104 for monitoring.

[0123] This makes it possible to generate the main video and the composite image 104 for monitoring for confirmation with a single image capture.

[0124] [2.4. Modifications of the Second Embodiment] The second embodiment is not limited to the specific example described above and may employ various modified configurations. For example, the virtual object texture display 210 is displayed on the background image vB as an assist display for the virtual object 201 appearing in the background image vB. However, the virtual object texture display 210 may also be displayed on the CG image 102. In this case, the CG generation unit 51 acquires a 3D model (3D background data) from the asset server 530 and generates the background image vB based on camera information. The CG generation unit 51 also detects a focused texture portion of the texture of the virtual object 201 in the background image vB. After that, when the CG generation unit 51 detects the focused texture portion, it generates a CG image 102 in which the virtual object texture display 210 is highlighted at a position corresponding to the focused texture portion, as shown in FIG. 17B . Note that the background image vB displayed on the LED wall 505 does not include the virtual object texture display 210, as shown in FIG. 17A .

[0125] Thereafter, a CG image 102 such as that shown in FIG. 17B is composited with an imaged image 101 obtained by capturing the background video vB using the imaging device 2, and ultimately, as shown in FIG. 17C, a composite image for monitoring 104 in which a virtual object texture display 210 is applied to the virtual object 201 is displayed on the display unit 15 and the monitor 7.

[0126] 3. Summary of the embodiment As described above, the image processing method of the embodiment displays on a display unit (display unit 15, monitor 7) a composite image (composite image for monitor 104) in which a virtual object 6 is superimposed on a captured image 101 obtained by the imaging device 2, and provides an assist display to assist in capturing an image of the virtual object 6 in the composite image 103. As a result, when focusing on the virtual object 6, for example, the assist displays (virtual object peaking display 106, virtual object texture display 107) allow the cameraman 4 to easily confirm the portion that is in focus. This makes it possible to improve the convenience of the composite image for the user when capturing an image.

[0127] An assist display is provided for the real object 5 and the virtual object 6 in the composite image 103. This allows the cameraman 4 to easily confirm the in-focus portion by the assist display (real object peaking display 105) when, for example, focusing on the real object 5. This improves the convenience of the composite image for the user when capturing an image.

[0128] As an assist display, a peaking display (virtual object peaking display 106) is performed on the portion of the virtual object 6 that is in focus with the imaging device 2. This allows the cameraman 4 to easily confirm the portion of the virtual object 6 that is in focus.

[0129] As an assist display, a texture display (virtual object texture display 107) is performed on the portion of the virtual object 6 that is in focus with the imaging device 2. This allows the cameraman 4 to easily confirm the portion of the virtual object 6 that is in focus.

[0130] When peaking display (real object peaking display 105, virtual object peaking display 106) is performed on the portions of the virtual object 6 and the real object 5 that are in focus with the imaging device 2, peaking display is performed in different display modes for the virtual object 6 and the real object 5. This makes it easy to know whether peaking display is being performed on the virtual object 6 or the real object 5.

[0131] Peaking display is performed in different display modes when assist display is performed on only the real object 5, only the virtual object 6, or both the real object 5 and the virtual object 6. This makes it easy to see whether peaking display is being performed on the virtual object 6 or the real object 5.

[0132] Assist displays for only the real object 5, only the virtual object 6, and both the real object 5 and the virtual object 6 are associated with different operations on the operation unit 21. This allows the cameraman 4 to switch between assist displays for only the real object 5, only the virtual object 6, and both the real object 5 and the virtual object 6 by operating the operation unit 21.

[0133] If the depth of field of the imaging device 2 is not equal to or less than a predetermined value, the assist display is not performed. This reduces the possibility that the depth of field is deep and all or most of the virtual objects 6 and real objects 5 are in focus, making them difficult to see if the assist display is performed.

[0134] If blurring effect is not enabled for the virtual object 6, no assist display is provided for the virtual object 6. If blurring effect is not provided for the virtual object 6, the virtual object 6 will not be brought into focus, and therefore, by not providing an assist display for the virtual object 6, confusion of the cameraman 4 can be reduced.

[0135] If the entire virtual object 6 is within the depth of field of the imaging device 2, assist display is not provided for the virtual object 6. If assist display is provided for all virtual objects 6, the composite image for monitoring 104 becomes difficult to view. Therefore, by not providing assist display for the virtual object 6, the composite image for monitoring 104 can be made easier to view.

[0136] When the area where the peaking display is performed is equal to or larger than a predetermined threshold, the sensitivity of the peaking display is reduced and the peaking display is performed. When the area where the peaking display is performed is large, the sensitivity is lowered to reduce the area where the peaking display is performed, thereby making it easier for the cameraman 4 to see the in-focus part.

[0137] As an assist display, a brightness distribution 120 of the real object 5 and the virtual object 6 is displayed. By displaying the brightness distribution 120 as an assist display in this way, the cameraman 4 can easily adjust the aperture. In addition, the brightness of the virtual object 6 can be reduced.

[0138] As an assist display, color histogram distributions 121 of the real object 5 and the virtual object 6 are displayed. By displaying the color histogram distributions 121 as an assist display in this manner, the cameraman 4 can easily adjust the white balance. In addition, the cameraman 4 can also adjust the white balance of the virtual object 6.

[0139] As an assist display, the real object 5 and the virtual object 6 are displayed in false color. By displaying them in false color in this way, the cameraman 4 can easily adjust the brightness. In addition, the cameraman 4 can adjust the brightness of the virtual object 6.

[0140] The display device (LED wall 505) on which the virtual object 201 is displayed is imaged by the imaging device 2, so that the virtual object 201 appears in the captured image 101, and the assist display is performed by the image (background image vB) displayed on the display device. In this way, when performing assist display in so-called virtual production, the assist display can be performed on the background image vB displayed on the LED wall 505.

[0141] The assist display is performed every other frame among consecutive frames displayed on the display device (LED wall 505), making it possible to generate the main video and the composite image 104 for monitoring for confirmation with a single image capture.

[0142] The virtual object 201 is displayed on a display device (LED wall 505), and the assist display is displayed on an image (CG image 102) different from the display device. As a result, the assist display is not displayed on the LED wall 505, and it is possible to prevent the assist display from being reflected in the captured image 101 obtained by capturing an image with the imaging device 2.

[0143] The image processing device 3 includes a control unit (CPU 31) that displays a composite image 103 (104) in which a virtual object 6 is superimposed on a captured image 101 obtained by the imaging device 2 on a display unit (display unit 15, monitor 7), and performs assist displays (virtual object peaking display 106, virtual object texture display 107) that are displays to assist in capturing an image of the virtual object 6 in the composite image 103. Also, an image processing system 1 includes the imaging device 2 and the image processing device 3, in which the imaging device 2 includes an imaging unit 12 that obtains the captured image 101 by capturing an image of real space, and the image processing device 3 includes a control unit (CPU 31) that displays a composite image 103 (104) in which a virtual object 6 is superimposed on the captured image 101 obtained by the imaging device 2 on a display unit (display unit 15, monitor 7), and performs assist displays (virtual object peaking display 106, virtual object texture display 107) that are displays to assist in capturing an image of the virtual object 6 in the composite image 103. Such an image processing device and image processing system can also provide the same functions and effects as the image processing method of the above-described embodiment.

[0144] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0145] <4. Present Technology> The present technology may also have the following configurations. (1) An image processing method that displays, on a display unit, a composite image in which a virtual object is superimposed on a captured image obtained by an imaging device, and performs assist display, which is a display for assisting in capturing an image of the virtual object in the composite image. (2) The image processing method described in (1), in which assist display is performed on a real object and the virtual object in the composite image. (3) The image processing method described in (1) or (2), in which, as the assist display, peaking display is performed on a portion of the virtual object that is in focus with the imaging device. (4) The image processing method described in any of (1) to (3), in which, as the assist display, texture display is performed on a portion of the virtual object that is in focus with the imaging device. (5) The image processing method described in (2), in which, when peaking display is performed on portions of the virtual object and the real object that are in focus with the imaging device, peaking display is performed in different display modes for the virtual object and the real object. (6) The image processing method according to (2) or (5), in which peaking display is performed in different display modes when an assist display is performed for only the real object, only the virtual object, or both the real object and the virtual object. (7) The image processing method according to (2), (5), or (6), in which an assist display for only the real object, only the virtual object, or both the real object and the virtual object is associated with different operations on an operation unit. (8) The image processing method according to any of (1) to (7), in which the assist display is not performed when the depth of field of the image capture device is not equal to or less than a predetermined value. (9) The image processing method according to any of (1) to (8), in which the assist display is not performed for the virtual object when blurring effect is not enabled for the virtual object. (10) The image processing method according to any of (1) to (9), in which the assist display is not performed for the virtual object when the virtual object is entirely within the depth of field of the image capture device. (11) The image processing method according to (3), wherein, when the area of ​​the peaking display is equal to or larger than a predetermined threshold, the peaking display is performed with a reduced sensitivity.(12) The image processing method according to (2), in which the assist display includes displaying brightness distributions of the real object and the virtual object. (13) The image processing method according to (2) or (12), in which the assist display includes displaying color histogram distributions of the real object and the virtual object. (14) The image processing method according to (2), (12), or (13), in which the assist display includes displaying the real object and the virtual object in false color. (15) The image processing method according to any one of (1) to (14), in which the display device on which the virtual object is displayed is imaged by the imaging device, so that the virtual object appears in the captured image, and the assist display is performed by an image displayed on the display device. (16) The image processing method according to (15), in which the assist display is performed every other frame among consecutive frames displayed on the display device. (17) The image processing method according to any one of (1) to (16), in which the virtual object is displayed on a display device, and the assist display is performed on an image different from that of the display device. (18) An image processing device comprising: a control unit that displays, on a display unit, a composite image in which a virtual object is superimposed on a captured image obtained by an imaging device, and performs an assist display that is a display to assist in capturing an image of the virtual object in the composite image. (19) An image processing system comprising: an imaging device and an image processing device, wherein the imaging device comprises an imaging unit that obtains a captured image by capturing an image of a real space, and the image processing device comprises a control unit that displays, on a display unit, a composite image in which a virtual object is superimposed on a captured image obtained by the imaging device, and performs an assist display that is a display to assist in capturing an image of the virtual object in the composite image.

[0146] REFERENCE SIGNS LIST 1 Image processing system 2 Imaging device 3 Image processing device 7 Monitor 15 Display unit 51 CG generation unit 53 Second image processing unit 54 Second synthesis unit

Claims

1. An image processing method comprising: displaying on a display unit a composite image in which a virtual object is superimposed on an image captured by an imaging device; and providing an assist display, which is a display for assisting in capturing an image of the virtual object in the composite image.

2. The image processing method according to claim 1, further comprising the step of: providing an assist display for the real object and the virtual object in the composite image.

3. The image processing method according to claim 1, wherein, as the assist display, a peaking display is performed on a portion of the virtual object that is in focus with the imaging device.

4. The image processing method according to claim 1, wherein a texture is displayed as the assist display on a portion of the virtual object on which the imaging device is focused.

5. The image processing method according to claim 2, wherein when peaking display is performed for portions of the virtual object and the real object that are in focus with the imaging device, the peaking display is performed in different display modes for the virtual object and the real object.

6. The image processing method according to claim 2, wherein assist display is performed in different display modes when assist display is performed only on the real object, only on the virtual object, or on both the real object and the virtual object.

7. The image processing method according to claim 2, wherein the assist display for only the real object, only the virtual object, and both the real object and the virtual object are associated with different operations on an operation unit.

8. The image processing method according to claim 1, wherein the assist display is not performed if the depth of field of the imaging device is not equal to or less than a predetermined value.

9. The image processing method according to claim 1, wherein if blurring is not enabled for the virtual object, assist display is not performed for the virtual object.

10. The image processing method according to claim 1, wherein if the virtual object is entirely within the depth of field of the imaging device, no assist display is performed for the virtual object.

11. The image processing method according to claim 3, further comprising the steps of: when an area in which the peaking display is performed is equal to or larger than a predetermined threshold value, lowering the sensitivity of the peaking display before performing the peaking display.

12. The image processing method according to claim 2, wherein the assist display includes displaying a brightness distribution of the real object and the virtual object.

13. The image processing method according to claim 2, wherein color histogram distributions of the real object and the virtual object are displayed as the assist display.

14. The image processing method according to claim 2, wherein the real object and the virtual object are displayed in false color as the assist display.

15. The image processing method according to claim 1, wherein the display device on which the virtual object is displayed is imaged by the imaging device so that the virtual object appears in the captured image, and the assist display is performed by the image displayed on the display device.

16. The image processing method according to claim 15, wherein the assist display is performed every other frame among consecutive frames displayed on the display device.

17. The image processing method according to claim 1, wherein the virtual object is displayed on a display device, and the assist display is performed on an image different from that of the display device.

18. An image processing device comprising: a control unit that displays on a display unit a composite image in which a virtual object is superimposed on an image captured by an imaging device, and that performs assist display, which is a display to assist in capturing an image of the virtual object in the composite image.

19. An image processing system comprising an imaging device and an image processing device, wherein the imaging device comprises an imaging unit that obtains an image by imaging a real space, and the image processing device comprises a control unit that displays a composite image on a display unit in which a virtual object is superimposed on the image obtained by the imaging device, and performs an assist display, which is a display to assist in imaging the virtual object in the composite image.

Images