Image processing device, image processing method, and program

Abstract

This image processing device generates a virtual space including a display area for displaying a photographed image taken by a real space photographing unit for photographing a real space, and controls photographing settings of the real space photographing unit in accordance with an operation instruction from a user through an avatar in the virtual space.

Description

Image processing device, image processing method, and program

[0001] The present invention relates to an image processing device, an image processing method, and a program.

[0002] Conventionally, virtual space communication systems, such as the Metaverse, allow users to set up avatars (characters that represent the user's identity) in a virtual space constructed using three-dimensional (3D) graphics. Through these avatars, users can engage in various activities, such as communicating with other avatars (other users) in the virtual space. In this type of virtual space communication system, various technologies have been developed to link the virtual space with the real world and enable interaction, allowing users to experience the experiences gained by operating the avatar as their own actions. One such technology allows users to operate an avatar in the virtual space to take photographs in the virtual space with a sense of realism similar to that of taking photographs in the real world. When taking photographs in the virtual space, a process is performed to crop a portion of the virtual space as an image (see Patent Document 1). Examples of methods for cropping an image include capturing and saving the graphics image from the avatar's viewpoint displayed on the monitor screen of a user's device, or capturing and saving a portion of the viewpoint video.

[0003] On the other hand, a system is known that combines a video of the real world (hereinafter referred to as a live-action video) with a virtual object generated by a CG computer system and displays the combined video in real time (Patent Document 2). Patent Document 2 discloses a technique for combining CG with a live-action video, but there is also a technique for combining a live-action video with an image of a virtual space. For example, a technique is known in which an image of a "window" area in a room existing in a virtual space is replaced with a video of the "sky" in the real world (live-action video) captured by a user's camera.

[0004] JP 2009-176025 A JP 2011-035638 A

[0005] However, in a virtual space that includes a real-life image area, the real-life image may be too dark (or too bright) compared to the image in the virtual space, or the white balance may be inappropriate. In such cases, a user may wish to change the exposure of the real camera to change the brightness of only the real-life image area. However, when operating a real camera capturing real-life image during virtual space communication, the user must repeat the steps of removing the HMD, changing the exposure settings of the real camera, and putting the HMD back on to check the brightness of the virtual space. As such, operating a real camera during virtual space communication has required cumbersome operations. According to one aspect of the present invention, a technology is provided that allows a user to easily set the shooting settings of a real camera that captures images of the real space displayed in the virtual space during virtual space communication.

[0006] An image processing device according to one aspect of the present invention comprises the following configuration: a virtual space generation means for generating a virtual space including a display area for a real-space image captured by a real-space imaging means for capturing an image of a real space, and an imaging control means for controlling imaging settings of the real-space imaging means in response to an operation instruction from a user via an avatar in the virtual space.

[0007] According to the present invention, it becomes possible for a user to easily set up the shooting settings of a real camera that captures images of the real space that are displayed in the virtual space while the user is communicating in the virtual space.

[0008] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0009] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments of the present invention, and are used, together with the description, to explain the principles of the present invention. A block diagram showing an example of the configuration of an image processing system. A diagram showing an overhead view of a virtual space. A diagram explaining camera images from a real camera and a virtual camera. A diagram explaining camera images from a real camera and a virtual camera. A diagram explaining camera images from a real camera and a virtual camera. A flowchart showing processing in a first embodiment. A diagram explaining a camera image and a viewed image after an instruction to change the brightness of a virtual camera is given. A diagram explaining a camera image and a viewed image after an instruction to change the brightness of a virtual camera is given. A diagram explaining a camera image and a viewed image after an instruction to change the brightness of a virtual camera is given. A flowchart showing processing according to a second embodiment. A diagram showing an example of a UI for a virtual camera according to a third embodiment. A diagram showing an example of a UI for a virtual camera according to the third embodiment. A diagram showing an example of a UI for a virtual camera according to the third embodiment. A flowchart showing processing for displaying linked selection buttons for a real camera. A flowchart showing brightness change processing for a virtual camera according to the third embodiment. A diagram showing an example of a UI display for a virtual camera according to the fourth embodiment. A diagram showing an example of a UI display for a virtual camera according to the fourth embodiment. 10 is a flowchart showing exposure change processing of a virtual camera according to a fourth embodiment;FIG. 11 is a diagram showing an example of reference information displayed on the virtual camera in the fourth embodiment;

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] First Embodiment FIG. 1 is a block diagram showing an example of the configuration of an image processing system according to a first embodiment. The image processing system of the first embodiment includes a real camera system unit 100, a CG system unit 101, a user operation instruction system unit 102, and a virtual space display system unit 103. Each of these system units may be configured as a single device or multiple devices. Furthermore, some of these system units may be implemented as a single device. The real camera system unit 100 captures images of the real world. The CG system unit 101 is an image processing device that generates virtual space images using three-dimensional (3D) CG. CG stands for Computer Graphics. The user operation instruction system unit 102 enables a user to issue operational instructions to objects in a virtual space or the real camera system unit 100 during virtual space communication. For example, a user can perform various operations in the virtual space via an avatar. The virtual space display system unit 103 displays an image (viewing image) for the user to monitor the virtual space on a display device such as a head mounted display (HMD). The real camera system unit 100, the CG system unit 101, the user operation instruction system unit 102, and the virtual space display system unit 103 are connected by a communication interface capable of high-speed processing in order to communicate image information in real time.

[0012] In this disclosure, the term "image" is used as a general term to refer to video, video, and still images. Images captured by the real space imaging unit 106 and the virtual space imaging unit 107 are typically video, but may also be still images. The following description will be given using an example in which the real space imaging unit 106 acquires video and the virtual space imaging unit 107 acquires still images. Regarding images, there are no particular limitations on the type of analog or digital, the type of data signal, the specification standards such as compressed / uncompressed, the recording device, recording medium, recording time, etc. Furthermore, there are no particular limitations on the camera device itself serving as the real space imaging unit 106 or its imaging method, etc.

[0013] (Real Camera System Unit 100) In the real camera system unit 100, a real-space imaging unit 106 can capture an image of a subject 105. The image captured by the real-space imaging unit 106 is sent to a virtual-space image generation unit 108 in the CG system unit 101 (described later). The real-space imaging unit 106 includes, for example, a digital camera or a digital video camera (hereinafter collectively referred to as a real camera) and an interface that communicatively connects the real camera to the CG system unit 101. The real camera includes a lens unit capable of focus and zoom control, an imaging sensor such as a CMOS sensor, an ND filter for adjusting exposure, and an aperture. The real camera allows for shooting settings (shutter speed, aperture value, gain (ISO sensitivity)) to adjust exposure (brightness) during shooting. The real camera also allows for shooting settings (selection of image processing to adjust WB (White Balance), hue, saturation, and gradation) to adjust the color tone of the image. The real space imaging unit 106 can change at least one of the above-mentioned imaging settings in accordance with control data from the imaging control unit 109. However, the system of this embodiment is not limited to the above-mentioned device configuration, and the configuration can be changed or devices can be added as needed.

[0014] Note that the term "user" here refers to a person who operates the real-space imaging unit 106 to capture real-space images and who operates an avatar, a character in the virtual space that serves as the user's alter ego. After determining the exposure and angle of view (image range) of the real camera in advance, the user puts on an HMD or the like, operates the avatar and virtual camera, and performs various activities in the virtual space, including capturing images with the virtual camera.

[0015] (CG System Unit 101) The CG system unit 101 includes a computer equipped with a 3DCG program and a processing processor that runs the program. The processing processor executes the 3DCG program to create CG images in real time. The CG system unit 101 may, for example, provide a virtual space communication system that builds a virtual space (virtual world) on the Internet and provides a place for users to engage in activities and communicate with each other. The virtual space capture unit 107, virtual space image generation unit 108, and capture control unit 109 included in the CG system unit 101 are functional blocks whose functions are realized by the computer. However, some or all of the virtual space capture unit 107, virtual space image generation unit 108, and capture control unit 109 may be realized by hardware such as a dedicated IC.

[0016] The virtual space image generation unit 108 generates a virtual space that does not actually exist using background data, such as the walls of a room, and generates a virtual space image. The virtual space image generation unit 108 also performs a synthesis process in which a real space image (in this example, a captured image of the subject 105) sent from the real space capture unit 106 is combined with the virtual space in a predetermined area within the virtual space to generate a new virtual space image. The virtual space image generation unit 108 can perform this synthesis process in real time. The virtual space image generation unit 108 also has the function of outputting a viewing image, which is an image for an HMD used to view the virtual space, and a virtual camera image (still image) to be recorded by a virtual camera (described later). The virtual camera image is a cropped image obtained by converting a portion of the virtual space from 3D virtual space data into two-dimensional (2D) image data and cropping it.

[0017] The virtual space capture unit 107 receives a virtual camera image from the virtual space image generation unit 108 as one piece of control data. The virtual space capture unit 107 has an image quality adjustment function that applies digital gain to the received virtual camera image to change the brightness and color (e.g., WB). The virtual space capture unit 107 then stores the image quality-adjusted virtual camera image in a storage device within the computer. As described above, the virtual space capture unit 107 has the function of a virtual camera. Furthermore, the virtual space capture unit 107 acquires, as control data, coordinate information regarding an insertion area of ​​a real space image included in a cut-out image (virtual camera image) acquired from the virtual space image generation unit 108. This allows the virtual space capture unit 107 to perform processing using the aforementioned digital gain on areas other than the insertion area of ​​the real space image.

[0018] The image capture control unit 109 controls the real space image capture unit 106 including a real camera or the virtual space image capture unit 107 including a virtual camera in response to a user's operation instruction in the user operation instruction system unit 102 (described later). The image capture control unit 109 can also simultaneously control two image capture units, the real space image capture unit 106 and the virtual space image capture unit 107. The image capture control unit 109 instructs various image capture settings of the real camera, such as brightness settings (e.g., exposure settings) for the image captured by the real camera, image processing settings related to color, gradation, and image range, and zoom (angle of view) settings. The image capture control unit 109 also instructs various image capture settings of the virtual camera, such as brightness settings, image processing settings related to color, gradation, and image range, and zoom (cropping range) settings. The image capture control unit 109 can also acquire various processing information of the virtual space image generation unit 108, such as the synthesis process of the real space image into the virtual space performed by the virtual space image generation unit 108, and the size and coordinate information of the insertion area of ​​the real space image in the synthesized image.

[0019] (User Operation Instruction System Unit 102) The operation instruction unit 110 is a known input device such as a remote control or keyboard that allows the user to operate an avatar or virtual camera in the virtual space, or a real camera that captures images in the real space. The operation instruction unit 110 is a known technology, so a detailed description thereof will be omitted.

[0020] (Virtual space display system unit 103) The virtual space display unit 111 is a display that displays 3D images, and in this embodiment, an HMD is used. By wearing the HMD on the user's head, the user can view an image (viewed image) of the virtual space that includes the real space image generated by the CG system unit 101. The virtual space display system unit 103 is a known technology, so a detailed description will be omitted. While checking the image in the virtual space displayed on the virtual space display system unit 103, the user uses the user operation instruction system unit 102 to perform operations such as moving the avatar and changing the angle of view (image range) and brightness of the virtual camera.

[0021] Next, processing in the system of this embodiment will be described. In this embodiment, when the shooting settings of the virtual camera are changed by a user operation, the shooting settings of the real camera are changed in conjunction with this. For example, when the user changes the shooting brightness setting of the virtual camera, the exposure of the real camera is changed in accordance with the change in brightness of the virtual camera. The following description will be given using an example in which when the user darkens the brightness of the virtual camera by one step, the exposure of the real camera is also darkened by one step. The change in brightness of the virtual camera and the change in exposure of the real camera are examples of shooting settings that can be made for the virtual camera and the real camera. A scene captured by the virtual camera and the real camera will be described using FIG. 2, which is an overhead view of the virtual space, and FIGS. 3A to 3C, which show images captured by the real camera and the virtual camera.

[0022] The scene captured by the virtual camera shown in the overhead view of Fig. 2 is a scene in which an avatar (photographer 201) operated by a user 104 uses a virtual camera 205 to capture an image of a subject 202 against the background of a wall 203 in a room with a window 204. The window 204 is an area in the virtual space into which an image captured by a real camera can be inserted. An example of the scene captured by the real camera is the sky as shown in real-space image 301 captured by the real camera in Fig. 3A. That is, in this example, the subject 105 in Fig. 1 is the sky.

[0023] FIG. 3B shows an image captured by a virtual camera (virtual camera image) without inserting a real-space image obtained by the real camera into the virtual space. Window 302 in FIG. 3B corresponds to window 204 in a wall in the virtual space, and in this example, is set as the insertion area of ​​the real-space image. When the real-space image is inserted into the position (area) of window 302, the virtual camera image becomes as shown in FIG. 3C. As shown in area 304 in FIG. 3C, real-space image 301 (a sky image) in FIG. 3A is displayed in the area of ​​window 302 (the insertion area of ​​the real-space image). Images 303 and 305 in FIGS. 3B and 3C each show a display of subject 202 in the virtual space.

[0024] Fig. 4 is a flowchart showing an example of a processing procedure in the image processing system of this embodiment. Also, Figs. 5A to 5C are diagrams showing images captured by each camera after an instruction to change the brightness of the virtual camera is given for the captured scenes of Figs. 3A to 3C, and an image viewed in the virtual space displayed on the HMD. Below, we will explain the capture processing flow when an instruction to change the brightness setting of the virtual camera is given, with reference to Figs. 3A to 5C.

[0025] First, in S401, the user 104 operates the virtual camera using the operation instruction unit 110. For example, the shooting control unit 109 determines the shooting angle of view in the virtual space by operating the virtual camera. Here, it is assumed that the angle is determined as shown in FIG. 3C . Next, in S402, the shooting control unit 109 determines whether or not the user 104 has issued a change instruction to the virtual camera via the operation instruction unit 110 to change the brightness of the captured image. If a change instruction has been issued (YES in S402), the process proceeds to S403; if no change instruction has been issued (NO in S402), the process skips S403 and proceeds to S404. In this embodiment, it is assumed that the user's operation on the virtual camera has issued an instruction to darken the image captured by the virtual camera by one level. Therefore, the process proceeds to S403.

[0026] In S403, the shooting control unit 109 instructs the real-space shooting unit 106 to darken the exposure by one step in response to the instruction to darken the brightness of the virtual camera by one step. For example, if the exposure of the real camera is at aperture F8, the shooting control unit 109 instructs the real-space shooting unit 106 to change the aperture of the real camera in accordance with the instruction value. As a result, the real-space image captured by the real camera becomes an image of the sky that is one step darker, such as real-space image 501 in FIG. 5A, compared to real-space image 301 in FIG. 3A. Note that the exposure control parameter for changing the exposure is not limited to aperture, and can be, for example, shutter speed, ISO sensitivity, or a combination thereof, and is not particularly limited.

[0027] Next, in S404, the virtual space image generation unit 108 composites the real space image with the exposure changed into the virtual space. Next, in S405, the virtual space image generation unit 108 generates a viewing image to be displayed on the HMD. Although 3D images are typically used in HMDs, in this embodiment, for ease of explanation, the image displayed on the HMD will be described as a 2D image shown in FIG. 5B. Also, at this time, the real space image of the region 502 corresponding to the window is an image of a dark sky. Next, in S406, the virtual space capture unit 107 generates a virtual camera image by cutting out a 2D image of a predetermined region from the virtual space region as a captured image for the virtual camera.

[0028] Next, in S407, the virtual space capture unit 107 determines whether or not there is an instruction to change the brightness of the virtual camera. If there is an instruction to change the brightness, the process proceeds to S408; if there is not, the process proceeds to S409. As described above, in this example, an instruction to change the brightness of the virtual camera by one level is issued, and the process proceeds to S408. In S408, the capture control unit 109 issues an instruction to change the brightness to the virtual space capture unit 107 in response to the brightness change instruction to the virtual camera. The virtual space capture unit 107 changes the brightness of the virtual camera image by applying digital gain to areas other than the insertion area of ​​the real space image. As a result, as shown in FIG. 5C , an image is obtained in which both the real space image area and the virtual space image area are one level darker than in FIG. 3C .

[0029] In S409, the virtual space imaging unit 107 determines whether the user 104 has instructed the virtual camera to take a picture. If a shooting instruction has been given (YES in S409), the process proceeds to S410. In S410, the virtual space imaging unit 107 saves the brightness-adjusted cropped image in a recording device within the CG system unit 101. On the other hand, if a shooting instruction has not been given to the virtual camera (NO in S409), the process ends.

[0030] As described above, in the first embodiment, an instruction to change the brightness of the virtual camera is used to instruct a change in the exposure of the real camera, eliminating the need for exposure adjustment operations on the real camera.

[0031] Second Embodiment In the first embodiment, a configuration was described in which exposure control of a real camera is linked to a brightness instruction of a virtual camera. However, when the operation exemplified in the first embodiment (an operation to lower the brightness of the virtual camera by one step) is performed, the exposure control of the real camera is linked, and as shown in FIG. 5B , only the area 502 displaying the real-space image in the image (viewed image) displayed on the HMD becomes darker. In this way, when a portion of the viewed image (the area of ​​the real-space image) becomes darker due to an operation on the virtual camera, the user viewing the viewed image may feel uncomfortable. In the second embodiment, to solve this problem, fluctuations in the brightness of the real-space image in the viewed image displayed on the HMD are reduced. This allows the brightness of the real-space image in the viewed image to be maintained or brought close to the state it was in before the virtual camera operation, thereby eliminating the aforementioned uncomfortable feeling.

[0032] In the second embodiment, the basic configuration of the image processing system is the same as in the first embodiment (see FIG. 1). The scenes captured by the real camera and virtual camera are also the same as in the first embodiment, and an example is a scene in which a one-level darker image is captured by the virtual camera. The following mainly describes the differences from the first embodiment.

[0033] The shooting process of the virtual camera according to the second embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the shooting process according to the second embodiment, and steps S601 to S610 in Fig. 6 correspond to steps S401 to S410 in the first embodiment (Fig. 4), respectively. That is, the only differences between Fig. 4 and Fig. 6 are steps S6041 and S6042, and the other steps are the same as those in the first embodiment, so their description will be omitted.

[0034] In S6041, the virtual space image generation unit 108 determines whether or not a brightness change instruction has been issued to the virtual camera. If a brightness change instruction has been issued to the virtual camera (YES in S6041), the process proceeds to S6042. If a brightness change instruction has not been issued to the virtual camera (NO in S6041), the process skips S6042 and proceeds to S605. In S6042, the shooting control unit 109 brightens the display area of ​​the real space image in the image for viewing displayed on the HMD by +1 step using image processing such as digital gain and gamma in response to the brightness change instruction from the virtual camera. This cancels the exposure change (-1 step) of the real camera performed in S603, and the image for viewing for the HMD generated in the next step, S605, can maintain the state it had before the exposure change instruction from the virtual camera. That is, the image in area 502 shown in FIG. 5B becomes +1 step brighter, resulting in a viewing image such as that shown in FIG. 3C.

[0035] As described above, according to the second embodiment, even if the shooting settings of the real camera are changed, the image changes caused by the change are reduced in the real-space image displayed in the viewing image, making it possible to provide a viewing image that feels natural or has a reduced sense of incongruity. Note that, in this embodiment, an example has been described in which the display area of ​​the real-space image in the viewing image is corrected, but this is not limited to this. For example, the virtual space itself generated by the virtual-space image generation unit 108 may be corrected. In this case, in generating the virtual space in S604, the brightness of the display area of ​​the real-space image is changed in accordance with a brightness change instruction from the virtual camera. For example, in response to a brightness change instruction to the virtual camera of -1 stop, the brightness of the display area of ​​the real-space image is changed by +1 stop, and the virtual space is generated, making the processing of S6042 unnecessary.

[0036] Third Embodiment In the first and second embodiments, a configuration was described in which a real camera is operated in conjunction with brightness control of a virtual camera. However, there are cases in which it is not necessary to link the operation of the virtual camera with the exposure control of the real camera, for example, when simply changing the appearance of the virtual camera. Therefore, in the third embodiment, a configuration will be described in which it is possible to select whether or not to link the operation of the virtual camera with the control of the real camera.

[0037] The basic configuration is the same as that of the first embodiment (FIG. 1). The assumed shooting scene is also the same as that of the first embodiment (FIG. 2, and FIGS. 3A to 3C). The shooting process according to the third embodiment will be described below with reference to FIGS. 7A to 9.

[0038] First, a display example of a user interface (hereinafter, UI) of a virtual camera according to the third embodiment will be described using FIGS. 7A to 7C. FIG. 7A shows a display example when a real camera image (real-space image) is not within the angle of view of the virtual camera, while FIGS. 7B and 7C show display examples when a real camera image (real-space image) is within the angle of view of the virtual camera. When the display area of ​​the real-space image is not within the angle of view of the virtual camera, a button (hereinafter, the link button) for instructing whether to link the real camera to a change in exposure of the virtual camera is not displayed on the display (state 701). On the other hand, when a real camera image is present, a link button 702 is displayed on the display. In this example, when the link button 702 is displayed with black text on a white background (FIG. 7B), the link button 702 is not selected (disabled), and the real camera is not linked to the operation of the virtual camera. When the link button 702 is displayed with white text on a black background (FIG. 7C), the link button 702 is selected (enabled), and the real camera is linked to the operation of the virtual camera. For example, when the user operates the avatar to touch the linked button 702, the linked button 702 alternates between a selected state and a non-selected state.

[0039] Next, referring to FIG. 8 , a display process of a link button for specifying whether to link the real camera with a change in exposure of the virtual camera will be described. When capturing an image with the virtual camera 205, the virtual space image generation unit 108 determines whether an image captured by the real camera (a real-space image) is included within the field of view of the virtual camera (S801). Specifically, the virtual space image generation unit 108 calculates the coordinates of the range included in the field of view of the virtual camera from the position and orientation of the virtual camera, and determines whether a real-space image exists within that range. If the determination results in a real-space image being included within the field of view of the virtual camera (YES in S801), the virtual space image generation unit 108 displays the link button 702 (S802). On the other hand, if it is determined that a real-space image is not included within the field of view of the virtual camera (NO in S801), the virtual space image generation unit 108 does not display the link button 702 (S803). Therefore, if a real-space image is not included within the field of view of the virtual camera, processing to change the capture settings of the real-space capture unit 106 in response to a user operation on the virtual space capture unit 107 is not performed.

[0040] It goes without saying that the display of the interlocking button 702 is not limited to the above-described display example. For example, the interlocking button 702 may be displayed in gray when the image captured by the real camera is not within the angle of view of the virtual camera, thereby indicating that the interlocking button 702 cannot be pressed (the interlocking button 702 is disabled). Furthermore, interlocking selection may be realized by a menu selection UI rather than the interlocking button 702. In this case, the menu selection may be disabled when the image captured by the real camera is not within the angle of view of the virtual camera, thereby indicating that interlocking selection is not possible. Any display may be used as long as it is possible to determine whether or not an image captured by the real camera is present.

[0041] Furthermore, even if an image captured by a real camera is within the field of view of the virtual camera, the link button 702 may be hidden if the owner of the real camera does not permit operation or if the real camera does not have the function to perform linkage. For example, the ID of the real camera capturing a real-space image within the field of view of the virtual camera may be acquired, and the real camera's operability information may be determined from the acquired ID. The link button may be displayed or hidden (enabled or disabled) depending on the operability and determination result. As a result, if the real-space capturing unit 106 does not support or permit control of the capture settings by the capture control unit 109, processing to change the capture settings of the real-space capturing unit 106 in response to a user operation on the virtual-space capturing unit 107 is not performed. Furthermore, the display state of the link button 702 allows the user to understand that the real-space capturing unit 106 does not support or permit control of the capture settings by the capture control unit 109.

[0042] Next, the brightness change process in the virtual camera in the third embodiment will be described using the flowchart in FIG. 9 . In S901, the shooting control unit 109 accepts a brightness change instruction to the virtual camera issued by the user via the avatar. In S902, the shooting control unit 109 determines whether the interlock button 702 is enabled. If the interlock button 702 is enabled (the state shown in FIG. 7C ) (YES in S902), in S903 the shooting control unit 109 instructs the real-space shooting unit 106 to change the exposure of the real camera. The amount of exposure change instructed at this time corresponds to the change accepted in S901. On the other hand, if the interlock button 702 is disabled (the state shown in FIG. 7B ) (NO in S902), the process of S903 is skipped, and the exposure of the real camera is not changed.

[0043] Next, in S904, the virtual space image generation unit 108 generates an image of the angle of view of the virtual camera from the information about the virtual space. In S905, the virtual space capture unit 107 determines the state of the interlock button 702. If the interlock button 702 is enabled (the state of FIG. 7C ) (YES in S905), the process proceeds to S906. In S906, the virtual space capture unit 107 changes the brightness of the area of ​​the virtual camera image generated from the virtual space (the area excluding the area of ​​the real space image) in accordance with the change instruction received in S901. On the other hand, if the interlock button 702 is disabled (the state of FIG. 7B ) (NO in S905), the process proceeds to S907. In S907, the virtual space capture unit 107 changes the brightness of the entire area of ​​the virtual camera image in accordance with the change instruction received in S901 (S907).

[0044] As described above, in the third embodiment, it is possible to select whether or not to perform exposure control on the real camera depending on the brightness change instruction on the virtual camera and the pressed state of the linked button. In other words, it is possible to select whether or not to change the shooting settings of the real camera in conjunction with a change in the shooting settings of the virtual camera.

[0045] Fourth Embodiment In the third embodiment, the brightness of the virtual camera image is changed by either S906 or S907 in FIG. 9 . That is, a brightness instruction to the virtual camera results in a change in the overall brightness of the virtual camera image, regardless of the state of the interlocking button 702. However, there are cases where it is desired to change the brightness of only one of the real camera and the virtual camera, such as when there is a difference in brightness between the real-space image from the real camera and the virtual camera image, or when blown-out highlights or crushed shadows occur in the real-space image from the real camera. The fourth embodiment shows an example of a configuration in which it is possible to select whether the shooting settings of the real camera or the virtual camera are changed depending on an operation on the virtual camera.

[0046] In the fourth embodiment, the basic configuration of the image processing system is the same as in the first to third embodiments. The assumed photographic scenes are also as shown in Figure 2 and Figures 3A to 3C. The following mainly describes the differences between the fourth embodiment and the third embodiment.

[0047] 10A and 10B are diagrams showing an example of a UI display of a virtual camera according to the fourth embodiment. The UI of the virtual camera according to the fourth embodiment includes a real-space button 1001 in addition to the interlocking button 702 described in FIGS. 7B and 7C. The real-space button 1001 is a button for selecting whether the area of ​​the image that changes in response to operations on the virtual camera is limited to the real-space image. For example, if the real-space button 1001 is disabled when the brightness setting of the virtual camera is changed (FIG. 10A), the overall brightness of the image acquired by the virtual camera changes. On the other hand, if the real-space button 1001 is enabled (FIG. 10B), the brightness changes only in the real-space image portion. Note that, like the interlocking button 702, the real-space button 1001 is displayed when the display area of ​​the real-space image of the real camera is within the angle of view of the virtual camera. That is, the display / hide of the real-space button 1001 is controlled in steps S802 and S803 of FIG. 8. Alternatively, similar to the linked button 702 , the display or non-display of the real space button 1001 may be controlled based on whether the real camera supports changes in the shooting settings from the shooting control unit 109 .

[0048] 11 is a flowchart showing the brightness change process of the virtual camera according to the fourth embodiment. The processes of S901 to S904 are the same as those of the third embodiment (FIG. 9). That is, when an instruction to change the brightness of the virtual camera is received, the shooting control unit 109 changes the exposure of the real camera if the link button 702 is enabled (S901 to S903). Then, the virtual space shooting unit 107 generates a virtual camera image, which is an image captured by the virtual camera (S904).

[0049] In S905, the virtual space imaging unit 107 determines the state of the interlock button 702. If the interlock button 702 is enabled (YES in S905), the process proceeds to S1101. In S1101, the virtual space imaging unit 107 determines the state of the real space button 1001. If the real space button is enabled (YES in S1101), the process ends. As a result, the brightness of only the display area of ​​the real space image in the virtual camera image is changed. In this case, the exposure of the real camera has already been changed in S903, and the brightness of the real space image in the virtual camera image has changed to the desired brightness. On the other hand, if the real space button is disabled in S1101 (NO in S1101), the process proceeds to S906 to change the brightness of the entire area of ​​the virtual camera image. If it is determined that the real space button 1001 is disabled in S1101, an instruction is issued to change the brightness of the entire area of ​​the virtual camera image while the exposure of the real camera has been changed (S903). Therefore, the virtual space photographing unit 107 changes the brightness only in the virtual space area in which the brightness has not been changed in the virtual camera image (S906).

[0050] If it is determined in S905 that the link button 702 is disabled (NO in S905), the process proceeds to S1102. In S1102, the virtual space capture unit 107 determines whether the real space button 1001 is enabled. If the real space button 1001 is enabled (YES in S1102), the process proceeds to S1103. In this case, since the exposure change of the real camera has not been performed (S903 is skipped), in S1103, the virtual space capture unit 107 changes the brightness of only the display area of ​​the real space image. This process can be used, for example, when changing the brightness balance between the real space and the virtual space is desired even when changing the exposure of the real camera is not permitted. On the other hand, if the real space button 1001 is disabled (NO in S1102), the process proceeds to S907. Since the exposure change of the real camera has not been performed as described above, the virtual space capture unit 107 changes the brightness of the entire area of ​​the virtual camera image, including the display area of ​​the real space image (S907).

[0051] Reference information that the user can refer to when operating the interlock button 702 and the real space button 1001 may be displayed, for example, on the UI of the virtual camera. An example of displaying reference information for the brightness of the virtual camera in the fourth embodiment will be described below with reference to FIG. 12 . The brightness reference information is displayed so that the user can determine whether to change the brightness of only the display area of ​​the image in real space, only the display area of ​​the image in virtual space, or the brightness of the entire area. Reference information 1201 indicates the brightness of the image in the entire area, reference information 1202 indicates the brightness of the image in virtual space (image other than area 304), and reference information 1203 indicates the brightness of the image in real space (image in area 304). Scale 1204 is a brightness scale, and in this example, one scale is 0.5 steps. Cursor 1205 indicates the brightness of each image, and appropriate position 1206 is the position when the brightness is appropriate. When cursor 1205 is at appropriate position 1206, the image has appropriate brightness. When the cursor is to the right of the appropriate position 1206, it indicates that the brightness is over, and when it is to the left, it indicates that the brightness is under.

[0052] The brightness reference information is calculated, for example, by the imaging control unit 109 acquiring brightness information of the images acquired by the real space imaging unit 106 and the virtual space imaging unit 107, and calculating the number of steps of the difference between the average brightness of each image and the brightness value of a predetermined appropriate brightness. For example, if the average brightness of the image is Y and the average brightness of the appropriate brightness is Yc, the number of steps of the difference ΔY is expressed as ΔY=log ２(Y / Yc). The shooting control unit 109 determines the position of the cursor 1205 according to the calculated value. The cursor position information determined by the shooting control unit 109 is sent to the virtual space image generation unit 108. The virtual space image generation unit 108 moves the cursor 1205 to a position according to the position information in the virtual camera UI, combines it with the image in the virtual space as brightness reference information, and displays it on the virtual space display unit 111. The example in FIG. 12 indicates that the image in the virtual space is 0.5 stops underexposed, the image in the real space is 2.5 stops overexposed, and the entire image is 1 stop overexposed. Using this information as a reference, the user can determine whether to change the exposure of the real camera or the brightness of the image in the virtual camera, and determine the states of the link button 702 and the real space button 1001.

[0053] The brightness reference information is not limited to the above example, as long as it allows comparison of the brightness of the image in each region. For example, the difference from the appropriate BV (Bright Value) value of the image in each region may be displayed numerically, or the appropriate BV value and the BV value of each image may be displayed.

[0054] Furthermore, changing the exposure of the camera in real space can improve whiteout and blackout depending on the brightness of the subject, so a histogram or other display may be used to show information about whiteout and blackout. This display may be limited to images in real space, and the user can check the extent of whiteout and blackout from the displayed histogram and determine the exposure of the real camera.

[0055] In this embodiment, an example in which only the image area in the real space is changed is shown, but a configuration in which only the image area in the virtual space can be selected to be changed, or a configuration in which both can be selected, may be used. In this case, the configuration for selection is not limited as long as it is a configuration in which an area can be specified and changed.

[0056] As described above, the image processing system of this embodiment makes it possible to change the exposure of the real camera and the brightness of each area or the entire area of ​​the virtual camera in response to the operation of the brightness of the camera in the virtual space. It is also possible to change the brightness of each area or the entire area while referring to reference information.

[0057] In the first to fourth embodiments, the shooting settings related to image brightness have been described as an example of the shooting settings performed in conjunction with the virtual camera and the real camera, but the present invention is not limited to this, and the linkage of the embodiments can be applied to various shooting settings of the real camera. For example, the shooting settings to be linked may be settings related to at least one of image processing for adjusting the color tone (WB, hue, saturation) of the shot and image processing for adjusting the gradation.

[0058] The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

[0059] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0060] This application claims priority based on Japanese Patent Application No. 2023-218433, filed December 25, 2023, the entire contents of which are incorporated herein by reference.

[0061] 100: Real camera system unit, 101: CG system unit, 102: User operation instruction unit, 103: Virtual space display unit, 104: User, 105: Subject, 106: Real space shooting unit, 107: Virtual space shooting unit, 108: Virtual space image generation unit, 109: Shooting control unit, 110: Operation instruction unit, 111: Virtual space display unit

Claims

1. An image processing device comprising: a virtual space generation means for generating a virtual space including a display area of ​​a real space image captured by a real space capturing means for capturing a real space; and a capturing control means for controlling the capturing settings of the real space capturing means in response to a user's operational instructions via an avatar in the virtual space.

2. The image processing device according to claim 1, further comprising a virtual space photographing means for capturing a portion of the virtual space as a virtual camera image, and the operational instruction is an instruction for setting photographing settings for the virtual space photographing means.

3. The image processing device according to claim 1 or 2, characterized in that the shooting settings include settings relating to image brightness.

4. An image processing device according to any one of claims 1 to 3, characterized in that the shooting settings include settings for image processing related to at least one of the color tone, gradation, and image range of the image.

5. An image processing device according to claim 2, characterized in that said photographing control means controls the photographing settings of said real space photographing means so as to correspond to the photographing settings instructed to said virtual space photographing means.

6. An image processing device as described in claim 2 or 5, further comprising a switching stage for switching between enabled and disabled the linked changing of the shooting settings of the real space shooting means and the virtual space shooting means in response to the user's operation instruction, wherein the shooting control means changes the shooting settings of the virtual space shooting means and the real space shooting means in response to the operation instruction when the enabled setting is set, and does not change the shooting settings in response to the operation instruction when the disabled setting is set.

7. An image processing device as described in claim 2 or 5, characterized in that the shooting control means is provided with a switching means which switches between controlling the shooting settings of the virtual space shooting means and the real space shooting means, controlling the shooting settings of the virtual space shooting means only, or controlling the shooting settings of the real space shooting means only, based on an instruction for shooting settings for the virtual space shooting means.

8. An image processing device as described in claim 6 or 7, further comprising a judgment means for judging whether or not the display area is included within the angle of view of the virtual space shooting means, and the shooting control means does not control the shooting settings of the real space shooting means when it is judged that the display area is not included within the angle of view.

9. The image processing apparatus according to claim 6 or 7, further comprising determination means for determining whether it is possible to control the shooting settings of the real space shooting means, wherein the shooting control means does not control the shooting settings of the real space shooting means when it is determined that it is not possible to control the shooting settings.

10. The image processing apparatus according to any one of claims 1 to 9, further comprising virtual space image generation means for generating a viewing image for viewing the virtual space, wherein the virtual space image generation means changes image processing on the display area for displaying the real space image inside the viewing image in accordance with control of the shooting settings of the real space shooting means.

11. The image processing apparatus according to claim 10, wherein the image processing on the display area processes the display area so as to reduce changes in the real space image according to control of the shooting settings for the real space shooting means.

12. The image processing apparatus according to claim 2, further comprising display means for displaying information regarding an image captured by at least one of the real space shooting means and the virtual space shooting means.

13. The image processing apparatus according to claim 12, wherein the information regarding the image includes information on the brightness of the image.

14. The image processing apparatus according to claim 12 or 13, wherein the information regarding the image includes information on overexposure or underexposure of the image.

15. An image processing method, comprising: a virtual space generation step of generating a virtual space including a display area of a real space image by real space shooting means for shooting the real space; and a shooting control step of controlling the shooting settings of the real space shooting means in accordance with an operation instruction of a user via an avatar in the virtual space.

16. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 14.

Images