Information processing system, information processing method, and computer program

Abstract

To provide an information processing system capable of reproducing a video image photographed in a virtual space in matching with a characteristic of a real space photographing section.SOLUTION: An information processing system comprising: a virtual space display unit configured to display a virtual space; a virtual space image generation unit configured to generate an image by imaging a part of the virtual space; and a real space imaging unit including an image processing unit capable of developing the image, wherein the image processing unit develops the image based on imaging information in the virtual space image generation unit and characteristic information of the real space imaging unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, a computer program, etc., relating to a virtual reality space. [Background technology]

[0002] In conventional virtual space communication systems, such as the Metaverse, users create avatars (characters that represent their own avatars) in virtual spaces constructed using 3D graphics. Through these avatars, users can engage in various activities, such as communicating with other avatars (other users) in the virtual space.

[0003] In this type of virtual space communication system, various technologies have been developed to link the real world and enable interaction so that users can experience the experiences gained by operating an avatar as their own actions. For example, one such technology allows users to take photos in a virtual space by operating a character in the virtual space, with a sense of realism similar to that of taking photos in the real world.

[0004] In Patent Document 1, when shooting in a virtual space, a graphics image from the avatar's viewpoint displayed on the monitor screen of a user terminal is captured and saved as is, or a part of the viewpoint video is captured and saved, and a part of the area monitored in the virtual space is cut out as an image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-176025 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the configuration of Patent Document 1 was unable to faithfully reproduce the appearance and image expression of an actual imaging device (hereinafter referred to as a "real camera") using an imaging device in a virtual space (hereinafter referred to as a "virtual camera").

[0007] An object of the present invention is to provide an information processing system that can reproduce an image captured in a virtual space in accordance with the characteristics of a real-space imaging unit. [Means for solving the problem]

[0008] In order to solve the above problem, an information processing system according to one aspect of the present invention comprises: a virtual space display unit that displays a virtual space; a virtual space image generation unit that captures an image of a part of the virtual space and generates an image; a real space imaging unit having an image processing unit capable of developing the image, The image processing unit develops the image based on the shooting information in the virtual space image generation unit and the characteristic information of the real space shooting unit. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an information processing system that can reproduce images captured in a virtual space in accordance with the characteristics of a real-space imaging unit. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of a basic configuration of an information processing system according to an embodiment of the present invention. [Figure 2] 2 is a functional block diagram showing an example of the configuration of a real space imaging unit 105 in a real camera system unit 100 according to an embodiment of the present invention. FIG. [Figure 3] 10A and 10B are flowcharts showing an example of a processing procedure of an information processing method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiment. In each drawing, the same members or elements are given the same reference numerals, and duplicated descriptions will be omitted or simplified.

[0012] The present embodiment relates to a method for capturing images in a virtual space in a virtual space communication system that provides a place for user activities and communication between users by constructing a virtual space (virtual world) on the Internet.

[0013] First Embodiment A first embodiment of the present invention will be described below. Fig. 1 is a diagram showing an example of the basic configuration of an information processing system according to an embodiment of the present invention.

[0014] Note that some of the functional blocks shown in FIG. 1 are realized by causing a CPU or the like serving as a computer (not shown) included in the information processing system to execute a computer program stored in a memory serving as a storage medium (not shown).

[0015] However, some or all of these functions may be implemented by hardware, such as a dedicated circuit (ASIC) or a processor (reconfigurable processor, DSP).

[0016] 1 may not be contained in the same housing, but may be configured as separate devices connected to each other via signal paths. The above explanation regarding FIG. 1 also applies to FIG. 2.

[0017] The information processing system shown in FIG. 1 includes a real camera system unit 100 in the real world and a CG system unit 101 that creates virtual space images using three-dimensional CG (Computer Graphics).

[0018] The information processing system also includes a user operation instruction system unit 102 that allows a user 104 to give instructions to operate an object in the virtual space or a real camera system, and a virtual space display system unit 103 that allows the user 104 to monitor the virtual space. The virtual space display system unit 103 is, for example, an HMD (Head Mount Display).

[0019] The real camera system unit 100, CG system unit 101, user operation instruction system unit 102, and virtual space display system unit 103 are connected by a communication interface capable of high-speed processing in order to communicate video information in real time.

[0020] In this embodiment, the type of video handled as a captured video by the real camera system unit 100 and the CG system unit 101 may be analog or digital. There are no particular limitations on the type of data, specification standards such as compressed or uncompressed, recording device, recording medium, recording time, etc., and the video can be used appropriately according to the purpose. Furthermore, there are no limitations on the camera device itself, shooting method, etc.

[0021] First, a description will be given of the real camera system unit 100 in Fig. 1. Fig. 2 is a functional block diagram showing an example of the configuration of the real space imaging unit 105 in the real camera system unit 100 in the embodiment of the present invention.

[0022] The imaging lens 201 is a group of lenses including a zoom lens, a focus lens, and a shift lens, and forms an image of a subject. The aperture 202 is used to adjust the amount of light. The ND filter 203 is a filter used for reducing light and can be inserted into the imaging optical path.

[0023] The imaging unit 204 has an imaging element configured with a CCD, CMOS element, or the like that converts an optical image into an electrical signal, and also has functions such as controlling the accumulation time using an electronic shutter, changing the analog gain, and changing the readout speed. The A / D converter 205 converts the analog signal output from the imaging unit 204 into a digital signal.

[0024] The image processing unit 206 performs processes such as predetermined pixel interpolation and resizing such as reduction, predetermined subject detection, color conversion, gamma correction, and digital gain addition on the data from the A / D converter 205. It also performs predetermined arithmetic processing using the captured image data and sends the arithmetic results to the system control unit 207. In this way, the real space imaging unit 105 has an image processing unit 206 that can develop images.

[0025] Based on the calculation results sent from the image processing unit 206, the system control unit 207 performs exposure control, distance measurement control, white balance control, camera shake correction, and the like using a TTL (Through The Lens) method.

[0026] The video output unit 208 converts the video data generated by the system control unit 207 into a predetermined video format and outputs it via a video output terminal. The communication unit 209 communicates with devices such as external control devices and transmits received communication data to the system control unit 207.

[0027] Furthermore, the communication unit 209 transmits communication data instructed by the system control unit 207. In this embodiment, the communication unit 209 receives video data from a virtual space video generation unit 106 (described later) and control data from a shooting control unit 107 (described later).

[0028] The storage unit 210 is an electrically erasable and recordable memory, such as an EEPROM. The storage unit 210 stores constants, programs, etc. for the operation of the system control unit 207. The programs referred to here are computer programs for executing the flowcharts described below.

[0029] The system control unit 207 has a built-in CPU as a computer, and controls each unit of the real space imaging unit 105 by executing a computer program stored in a storage unit 210 as a storage medium.

[0030] Next, the CG system unit 101 in Fig. 1 will be described. The CG system unit 101 includes a 3D CG program and a CPU serving as a computer equipped with a processing processor for running the program, and has the function of creating CG images in real time using the 3D CG program. Within the CG system unit 101, a virtual space image generation unit 106 and a shooting control unit 107 are functional blocks whose functions are realized by the CPU serving as the computer.

[0031] The virtual space image generating unit 106 inputs necessary information into the computer in advance, and by inputting background data such as the walls of a room, for example, it is possible to freely create virtual space images that do not actually exist.

[0032] The virtual space image generation unit 106 also has the function of outputting an HMD image for viewing the virtual space and image data to be sent to the communication unit 209. Here, outputting image data refers to the process of capturing an image of a portion of the virtual space from a specific position with a virtual camera, converting the image into 2D image data, and cropping the image to generate an image. The image capture control unit 107 controls the angle of view of the virtual camera in response to an operation instruction from the user operation instruction system unit 102, which will be described later.

[0033] The operation instruction unit 108 is an input device such as a remote controller or keyboard that allows the user to operate an avatar, a virtual camera, or a real camera in the virtual space.

[0034] The virtual space display unit 109 is a display such as an HMD for displaying 3D images as a virtual space. By wearing the HMD on the user's head, the user can view the virtual space images generated by the CG system unit 101. Note that the configuration of the virtual space display system unit 103 is a known technology, so a detailed description thereof will be omitted.

[0035] While checking the image in the virtual space displayed on the virtual space display system unit 103, the user uses the user operation instruction system unit 102 to move the avatar, change the shooting angle of the virtual camera, and so on.

[0036] 3(A) and (B) are flowcharts showing an example of the processing procedure of an information processing method in an embodiment of the present invention, where FIG. 3(A) shows an example of control in the CG system unit 101, and FIG. 3(B) shows an example of control in the real camera system unit 100.

[0037] The CPU serving as the computer of the CG system unit 101 executes a computer program stored in memory, thereby sequentially performing the operations of the steps in the flowchart of Fig. 3(A). The CPU serving as the computer of the system control unit 207 executes a computer program stored in memory, thereby sequentially performing the operations of the steps in the flowchart of Fig. 3(B).

[0038] In step S301, the CPU of the CG system unit 1017 receives information on a user operation based on an operation of the user operation instruction system unit 102.

[0039] In step S302, the CPU of the CG system unit 101 cuts out the area photographed by the virtual camera and converts it into 2D video data.

[0040] In step S303, the CPU of the CG system unit 101 transmits the virtual camera image data and shooting information related to the image data to the real camera system unit 100. Here, the virtual camera shooting information includes at least one of the F-number, focal length information, subject distance information, ISO sensitivity information, and accumulation time of the virtual camera capturing an image of a part of the virtual space. Note that the image data transmitted in step S303 is the image data before development.

[0041] 3(B), the camera function of the real camera system unit 100 is turned off. That is, the readout operation from the image sensor is turned off. In this state, the image processing unit 206 of the real camera system unit 100 develops the image from the virtual camera instead of the image signal from the image sensor.

[0042] That is, first, in step S304, the CPU of the system control unit 207 receives the video data and shooting information of the virtual camera transmitted from the CG system unit 101 via the communication unit 209.

[0043] In step S305, the CPU of the system control unit 207 acquires characteristic information of the real camera system unit 100. In step S305, if the real camera is a camera with interchangeable lenses, the optical characteristics of the attached interchangeable lens are acquired.

[0044] Here, the characteristic information of the real camera system unit 100 includes at least one of the optical characteristic information of the imaging lens 201 of the real space imaging unit 105 illustrated in FIG. 2, characteristic information of the imaging element of the real space imaging unit 105, image blur characteristics (characteristics of a gyro for image blur correction), etc.

[0045] In addition, the optical characteristic information of the lens includes at least one of peripheral light intensity, chromatic aberration of magnification, distortion aberration, etc., and the characteristic information of the image sensor includes at least one of the number of pixels of the image sensor, photoelectric conversion characteristics, noise characteristics of the image sensor depending on ISO sensitivity, etc.

[0046] In step S306, the CPU of the system control unit 207 performs development processing on the received video data using the image processing unit 206. That is, based on the shooting information of the virtual camera and the characteristic information of the real space shooting unit 105, the received video data is developed using the image processing unit 206 of the real space shooting unit 105, thereby performing development processing that reflects the characteristic information of the real space shooting unit 105.

[0047] At this time, the noise characteristics of the real space imaging unit 105 are reproduced based on the ISO sensitivity information included in the shooting information of the virtual camera and the noise characteristics of the image sensor in the real space imaging unit 105.

[0048] Similarly, image blur in the real space imaging unit 105 may be reproduced based on the accumulation time and focal length information included in the shooting information of the virtual camera and the characteristics of the gyro in the real space imaging unit 105. Note that, depending on the configuration of the real space imaging unit 105, other characteristics of the real space imaging unit 105 may also be reproduced.

[0049] In step S306, the image processing unit 206 of the real space imaging unit 105 may calculate an AF or AE evaluation value based on the video data received from the virtual camera, and may control the aperture, accumulation time, and focus lens of the virtual camera based on the evaluation value. This makes it possible to reproduce the AF or AE control characteristics of the real space imaging unit 105. AF stands for automatic focus adjustment, and AE stands for automatic exposure adjustment.

[0050] That is, the image processing unit may calculate an exposure control evaluation value based on the image received from the virtual camera, and control the aperture or accumulation time of the virtual camera based on the exposure control evaluation value. Also, the image processing unit may calculate a focus adjustment evaluation value based on the image received from the virtual camera, and control the focus lens of the virtual camera based on the focus adjustment evaluation value.

[0051] In step S307, the CPU of the system control unit 207 transmits the developed video data to the CG system unit 101 via the communication unit 209, and also stores it in the storage unit 210 of the real space shooting unit 105.

[0052] In step S308, the CPU of the CG system unit 101 receives the video data transmitted from the real camera system unit 100. The received video data is transmitted to the virtual space display system unit 103, and is displayed by the virtual space display unit 109. Furthermore, the received video data is stored in the CG system unit 101.

[0053] As described above, according to this embodiment, the appearance and visual expression of a real imaging device (hereinafter referred to as a "real camera") can be reproduced using an imaging device in a virtual space (hereinafter referred to as a "virtual camera"), thereby further enhancing the experience value.

[0054] Furthermore, according to this embodiment, the reproduction of video expressions in a virtual space can faithfully reproduce not only objects such as imaging devices, but also real places and buildings. For example, in video production, location hunting can be performed in a virtual space using a virtual camera, and then actual filming can be performed using a real camera.

[0055] Furthermore, since there are no camera shakes or optical characteristics of the lens in the virtual space, the image may differ from that of the real camera depending on the gyro sensor characteristics of the real camera and the optical characteristics of the attached imaging lens.

[0056] However, in this embodiment, even in such a case, the image of the virtual space captured by the virtual camera can be developed using the algorithm of the real camera in accordance with the characteristics of the real camera, making it possible to reproduce the image faithfully and quickly to the characteristics of the real camera.

[0057] The present invention has been described above in detail based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible based on the spirit of the present invention, and these are not excluded from the scope of the present invention.

[0058] The present invention also includes those that realize the functions of the above embodiments using, for example, at least one processor such as a CPU, memory, or circuit (for example, ASIC). Also, multiple processors may be used to perform distributed processing.

[0059] In order to realize part or all of the control in the above-described embodiments, a computer program that realizes the functions of the above-described embodiments may be supplied to an information processing system or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the information processing system or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. The present invention also includes the following combinations.

[0060] (Configuration 1) An information processing system comprising a virtual space display unit that displays a virtual space, a virtual space image generation unit that captures a portion of the virtual space and generates an image, and a real space shooting unit having an image processing unit that can develop the image, wherein the image is developed by the image processing unit based on the shooting information in the virtual space image generation unit and characteristic information of the real space shooting unit.

[0061] (Configuration 2) The information processing system according to Configuration 1, wherein the characteristic information includes at least one of optical characteristic information of the photographing lens of the real-space photographing unit, characteristic information of the image sensor of the real-space photographing unit, and image blur characteristics of the real-space photographing unit.

[0062] (Configuration 3) The information processing system described in configuration 1 or 2, characterized in that the shooting information includes at least one of the F-number, focal length information, subject distance information, ISO sensitivity information, and accumulation time of a virtual camera that captures a portion of the virtual space.

[0063] (Configuration 4) An information processing system described in any one of configurations 1 to 3, characterized in that the image processing unit calculates an evaluation value for exposure control based on the image, and controls the aperture or accumulation time of a virtual camera that captures a portion of the virtual space based on the evaluation value for exposure control.

[0064] (Configuration 5) An information processing system described in any one of configurations 1 to 4, characterized in that the image processing unit calculates a focus adjustment evaluation value based on the image, and controls a focus lens of a virtual camera that captures a portion of the virtual space based on the focus adjustment evaluation value.

[0065] (Configuration 6) The information processing system according to any one of configurations 1 to 5, wherein the image developed by the image processing unit is stored in a storage unit of the real space imaging unit.

[0066] (Configuration 7) An information processing system according to any one of configurations 1 to 6, characterized in that the image developed by the image processing unit is transmitted to the virtual space image generation unit and displayed by the virtual space display unit.

[0067] (Method) An information processing method using an information processing system having a virtual space display unit that displays a virtual space, a virtual space image generation unit that captures a portion of the virtual space and generates an image, and a real space shooting unit that has an image processing unit that can develop the image, characterized in that the image is developed by the image processing unit based on the shooting information in the virtual space image generation unit and characteristic information of the real space shooting unit.

[0068] (Program) A computer program for controlling each part of the information processing system described in any one of the configurations by a computer. [Explanation of symbols]

[0069] 100: Actual camera system section 101: CG Systems Department 102: User operation instruction system unit 103: Virtual Space Display System Department 104:User 105: Real-space photography department 106: Virtual space image generation unit 107: Shooting control unit 108: Operation instruction section 109: Virtual space display section 201: Imaging lens 202:Aperture 203:ND filter 204: Imaging unit 205: A / D converter 206: Image processing unit 207: System control unit 208: Video output section 209: Communications Department 210: Storage section

Claims

1. a virtual space display unit that displays a virtual space; a virtual space image generation unit that captures an image of a part of the virtual space and generates an image; a real space imaging unit having an image processing unit capable of developing the image, An information processing system characterized in that the image is developed by the image processing unit based on shooting information in the virtual space image generation unit and characteristic information of the real space shooting unit.

2. 2. The information processing system according to claim 1, wherein the characteristic information includes at least one of optical characteristic information of a photographing lens of the real-space photographing unit, characteristic information of an image sensor of the real-space photographing unit, and image blur characteristics of the real-space photographing unit.

3. 2. The information processing system according to claim 1, wherein the photographing information includes at least one of an F-number, focal length information, subject distance information, ISO sensitivity information, and accumulation time of a virtual camera that photographs a portion of the virtual space.

4. 2. The information processing system according to claim 1, wherein the image processing unit calculates an evaluation value for exposure control based on the image, and controls an aperture or accumulation time of a virtual camera that captures a portion of the virtual space based on the evaluation value for exposure control.

5. The information processing system according to claim 1, wherein the image processing unit calculates a focus adjustment evaluation value based on the image, and controls a focus lens of a virtual camera that captures a portion of the virtual space based on the focus adjustment evaluation value.

6. 2. The information processing system according to claim 1, wherein the image developed by the image processing unit is stored in a storage unit of the real space imaging unit.

7. 2. The information processing system according to claim 1, wherein the image developed by said image processing section is transmitted to said virtual space image generating section and displayed by said virtual space display section.

8. a virtual space display unit that displays a virtual space; a virtual space image generation unit that captures an image of a part of the virtual space and generates an image; a real space imaging unit having an image processing unit capable of developing the image, An information processing method, characterized in that the image is developed by the image processing unit based on shooting information in the virtual space image generation unit and characteristic information of the real space shooting unit.

9. A computer program for controlling each unit of the information processing system according to any one of claims 1 to 7 by a computer.

Images