Information processing device, information processing method and program
The information processing device simulates real-space photography techniques in virtual space by using model and imaging parameters, and operation information to generate virtual images, addressing the skill transfer gap between real and virtual photography.
Patent Information
- Application Number
- JP2023080782
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-16
- Publication Date
- 2024-11-28
AI Technical Summary
Existing VR photography techniques require users to learn new shooting techniques specific to VR spaces, which can be a barrier for those familiar with real-space photography, as there are significant differences between shooting in virtual and real spaces.
An information processing device and method that utilizes model parameters, imaging parameters, and operation information from a real-space imaging device to generate a photographed image in a virtual space, simulating real-space photography techniques by converting the position, orientation, and operations of the imaging device into virtual space.
Enables photography in virtual space that mimics real-space photography, allowing users to apply their existing skills effectively.
Smart Images

Figure 2024164993000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device, an information processing method, and a program for capturing images of a virtual space. [Background technology]
[0002] In recent years, research on VR (Virtual Reality) technology has progressed. As research on VR technology progresses, VR devices such as HMDs (Head Mounted Displays) have become more widespread, and communication between many users in the virtual space of VR has become more active. The form of applications aimed at communication in such virtual space is called the "metaverse."
[0003] In the virtual space of the metaverse (metaverse space), a user can communicate with other users using an avatar, which is an alter ego of the user. As an example of a communication activity, it is possible to take a photograph of the avatar in the metaverse space.
[0004] Thanks to advances in 3D rendering technology, photos taken in the Metaverse space can express optical characteristics (exposure, bokeh, etc.) comparable to those of real space. Given this background, it is expected that users who are knowledgeable about optical characteristics and have excellent photography skills (such as professional photographers) will enter the world of photography in the Metaverse space.
[0005] However, photography in the metaverse space is developing as users who are already familiar with operating VR applications acquire photography techniques specialized for VR spaces (how to operate a VR controller, how to set photography parameters, etc.) It is expected that the photography techniques of users who are unfamiliar with operating VR applications will not match with photography using a VR controller, which will be an obstacle to their entry into photography in VR spaces. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] JP 2016-81408 A [Patent Document 2] Patent Publication No. 2022-41258 Summary of the Invention [Problem to be solved by the invention]
[0007] In Patent Document 1, a device other than a dedicated VR controller can be used as the HMD controller. In Patent Document 1, a smartphone is used as the controller, but by applying this technology to a camera, it is possible to take pictures of the virtual space using a familiar device. On the other hand, Patent Document 2 describes that when a virtual camera corresponding to a terminal device takes pictures of the virtual space, the virtual camera may take pictures in response to an operation on the terminal device.
[0008] However, in the techniques of Patent Documents 1 and 2, there are many differences between shooting in a virtual space and shooting in a real space, and users cannot fully utilize the shooting techniques they have acquired through shooting in a real space. For this reason, it is not possible to realize shooting in a virtual space that is similar to shooting in a real space.
[0009] An object of the present invention is to realize photography in a virtual space similar to photography in a real space. [Means for solving the problem]
[0010] One aspect of the present invention is a method for producing a composition comprising the steps of: An information processing device that captures an image in a virtual space in which a virtual object corresponding to a first imaging device that captures an image in a real space is disposed, an acquisition means for acquiring model parameters and shooting parameters of the first imaging device; an operation information acquisition means for acquiring operation information of a user with respect to the first imaging device; a generation means for generating a captured image of the virtual space from a position and an orientation of the virtual object in the virtual space based on the model parameters, the shooting parameters, and the operation information; The information processing device is characterized by having:
[0011] One aspect of the present invention is a method for producing a composition comprising the steps of: 1. An information processing method for capturing an image in a virtual space in which a virtual object corresponding to a first imaging device that captures an image of a real space is disposed, comprising: an acquisition step of acquiring model parameters and shooting parameters of the first imaging device; an operation information acquisition step of acquiring operation information of a user with respect to the first imaging device; a generating step of generating a captured image of the virtual space from a position and an orientation of the virtual object in the virtual space based on the model parameters, the shooting parameters, and the operation information; The information processing method is characterized by having the following features. Effect of the Invention
[0012] According to the present invention, shooting in a virtual space can be realized in the same way as shooting in a real space. [Brief description of the drawings]
[0013] [Figure 1] FIG. 1 is a diagram illustrating an information processing system. [Diagram 2] FIG. 2 is a hardware configuration diagram of an information processing device and the like. [Diagram 3] FIG. 2 is a functional block diagram of an information processing device. [Figure 4] FIG. 11 is a diagram showing model parameters. [Diagram 5] FIG. 4 is a diagram showing shooting parameters. [Figure 6] FIG. 11 is a diagram showing shooting operation information. [Figure 7] 13 is a flowchart of a process for generating an image of a virtual space. [Figure 8]13 is a flowchart of displaying a virtual object. [Figure 9] FIG. 2 is a diagram illustrating shooting of a real space and shooting of a virtual space. [Figure 10] 1A and 1B are diagrams illustrating a shooting direction and a shooting angle of view. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following embodiment does not limit the present invention, and not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention. The configuration of the embodiment may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (conditions of use, environment of use, etc.). In addition, the present invention may be configured by appropriately combining parts of each embodiment described later.
[0015] <Embodiment 1> 1 is a diagram illustrating an information processing system 1 according to embodiment 1. The information processing system 1 includes an information processing device 101 and an imaging device .
[0016] The information processing device 101 is a head mounted display (HM) that can be worn by a user on the head. D). The imaging device 102 is a camera device. The imaging device 102 is, for example, a single-lens reflex camera, a mirrorless camera, a compact digital camera, an action camera (a camera that is attached to the user's body to enable hands-free shooting), a drone, or a smartphone. Note that the information processing device 101 may have a display system and a processing system separated when a small and lightweight device is required. Of these, the processing system may be disposed in a small external box computer. The user 103 is a user who uses the information processing system 1.
[0017] A front portion of an avatar 105 of another user standing against the background of the virtual space is displayed in a field of view image 104 seen by a user 103 wearing an information processing device 101. A virtual object 106 (a virtual object of the imaging device 102) corresponding to an imaging device 102 (the position, posture, and appearance of the imaging device 102) held in the hand of the user 103 in real space is also displayed in the field of view image 104. When the information processing device 101 performs body tracking, the virtual object 106 is displayed naturally together with a hand 107 of the user's avatar so that the user feels as if he or she is holding the imaging device 102 in the virtual space.
[0018] A user 103 can change the settings of a virtual object 106, which is a camera that captures images of a virtual space, by operating a real imaging device 102. In addition, when the user 103 releases the shutter of the real imaging device 102, the information processing device 101 can acquire a still image (photo) or a video of the virtual space based on the position and orientation (position and orientation) of the virtual object 106 and the settings of the virtual object 106.
[0019] FIG. 2 is a diagram showing an example of the hardware configuration of the information processing device 101 and the imaging device 102. As shown in FIG.
[0020] The information processing device 101 includes a CPU 201, a ROM 202, a RAM 203, a sensor 204, an input / output interface 205, a recording unit 206, an imaging unit 207, a display unit 208, an operation unit 209, and a bus 210. The hardware components in the information processing device 101 communicate with each other by being connected to the bus 210.
[0021] A CPU (Central Processing Unit) 201 is a system control unit that controls the entire information processing device 101. Moreover, the CPU 201 realizes information processing according to the first embodiment by executing an information processing program.
[0022] A ROM (Read Only Memory) 202 stores basic programs, initial data, etc. The ROM 202 is a read-only memory that stores programs and parameters that do not require modification.
[0023] A RAM 203 (Random Access Memory) is a memory for temporarily storing input information and calculation results (calculation results in information processing and image processing).
[0024] The sensor 204 detects the position and orientation of the information processing device 101. The sensor 204 includes sensing components such as a gyro and an IMU (inertial measurement unit).
[0025] The input / output interface 205 is capable of inputting and outputting predetermined data. In the first embodiment, the predetermined data is transmitted from the input / output interface 215 of the imaging device 102. The predetermined data is model parameters, shooting parameters, shooting operation information, or the like of the imaging device 102. The input / output connection form by the input / output interface 205 may be a form using a local connection such as USB or Bluetooth (registered trademark). The input / output connection form of the input / output interface 205 may be an internet connection using Ethernet or Wi-Fi (registered trademark).
[0026] The recording unit 206 is a device capable of writing and reading various information. Specifically, the recording unit 206 is, for example, a member (hard disk or memory card) built into or external to the information processing device 101. Alternatively, the recording unit 206 is a member (memory card, removable disk, IC card, etc.) that is detachable from the information processing device 101. The information processing program executed by the CPU 201 is recorded in the recording unit 206. The information processing program is read from the recording unit 206, expanded in the RAM 203, and executed by the CPU 201. The information processing program may be stored in the ROM 202. The recording unit 206 can also record predetermined data used by the information processing program executed by the CPU 201. Still images (photographs) and videos captured in the virtual space are also recorded in the recording unit 206.
[0027] The photographing unit 207 is an imaging device that photographs the real space. An actual image (real image) photographed in the real space is used for "tracking the joints and hand movements of the user" and "detecting a plane on which a virtual object such as a table is superimposed" by image recognition. In the first embodiment, when the accuracy of the position and orientation information of the imaging device 102 by a sensor 214 provided in the imaging device 102 described later is insufficient, the result of tracking the imaging device 102 by image recognition can be used to supplement the position and orientation information of the imaging device 102.
[0028] The display unit 208 is an electronic display device, for example a stereo display having display areas corresponding to both eyes of the user.
[0029] The operation unit 209 accepts operations for the information processing device 101. Operation members included in the operation unit 209 include a power button, a menu button, a selection button, and a decision button.
[0030] The imaging device 102 includes a CPU 211, a ROM 212, a RAM 213, a sensor 214, an input / output interface 215, a recording unit 216, an imaging unit 217, a display unit 218, an operation unit 219, and a bus 220. Each hardware component in the imaging device 102 is connected to the bus 220 and communicates with each other.
[0031] The CPU 211 is a system control unit that controls the entire imaging device 102. The CPU 211 also realizes information processing according to this embodiment by executing an information processing program.
[0032] The ROM 212 stores basic programs, initial data, etc. The ROM 212 is a read-only memory that stores programs and parameters that do not require modification.
[0033] The RAM 213 is a memory for temporarily storing input information and calculation results (calculation results of information processing and image processing).
[0034] The sensor 214 detects the position and orientation of the image capture device 102. The sensor 214 includes a sensing component such as a gyro or an IMU. The image capture device 102 can be used, for example, as a spirit level (an instrument for checking the angle of an object relative to the horizontal or vertical).
[0035] The input / output interface 215 receives input and output of predetermined data. The predetermined data is model parameters of the image capture device 102, shooting parameters, or shooting operation information. The input / output connection form of the output interface 215 may be realized by either a local connection (connection via USB or Bluetooth, etc.) or an internet connection (connection via Ethernet or Wi-Fi).
[0036] The recording unit 216 is a device capable of writing and reading various information. Specifically, the recording unit 216 is, for example, a member (hard disk or memory card) built into the imaging device 102 or attached externally. Alternatively, the recording unit 216 is a member (memory card, removable disk, IC card, etc.) that is detachable from the imaging device 102. The information processing program executed by the CPU 211 is recorded in the recording unit 216. The information processing program is read from the recording unit 216 and deployed in the RAM 213, and executed by the CPU 211. The information processing program may be stored in the ROM 212. The recording unit 216 can also record predetermined data used by the information processing program executed by the CPU 211.
[0037] The photographing unit 217 is an imaging device that photographs the real space. The photographing unit 217 records in the recording unit 206 still images and videos photographed by the imaging device 102 alone in the real space.
[0038] The display unit 218 is an electronic display device, for example, a live view display provided on the back of the imaging device 102 (camera).
[0039] The operation unit 219 receives operations for the imaging device 102. The operation members of the operation unit 219 are buttons, dials, or a touch panel. The operation members of the operation unit 219 are a focus ring (a member for controlling focus), a zoom ring (a member for controlling zoom), an aperture ring (a member for controlling aperture), or the like.
[0040] 3 is a functional block diagram showing the functional configuration of the information processing device 101. The information processing device 101 includes a model parameter acquisition unit 301, a shooting parameter acquisition unit 302, an operation information acquisition unit 303, a shot image generation unit 304, and an object display unit 305.
[0041] The model parameter acquisition unit 301 acquires model parameters held by the image capture device 102 .
[0042] 4 shows an example of model parameters. The imaging device 102 has a main body (a device including an image processing circuit, etc.) and one or more components connected to the main body. The model parameters have hardware information related to the imaging device 102 (information on the model number of the main body of the imaging device 102 and information on the model number of the lens attached to the main body of the imaging device 102). The model parameters may have information on components electrically connected to the main body of the imaging device 102 and with identifiable model numbers (such as a mount adapter, a strobe, a grip, or an extended finder). The model parameters are model-specific information that cannot be changed by the user.
[0043] The shooting parameter acquisition unit 302 acquires the shooting parameters held by the image capture device 102 .
[0044] 5 shows an example of shooting parameters. The shooting parameters are information related to shooting (such as aperture value, shutter speed, ISO sensitivity, exposure compensation value, lens focal length, white balance, and shooting mode) set in the image capture device 102. In other words, the shooting parameters are information that can be changed by the user in order to capture a picture.
[0045] The operation information acquisition unit 303 acquires shooting operation information of the image capture device 102. The shooting operation information is temporary information generated by the imaging device 102 when an operation event occurs (when an operation is performed by a user). The shooting operation information is not stored in a data store. For example, when an operation event occurs in the imaging device 102 and the operation information acquisition unit 303 receives a notification of the occurrence of the event, the operation information acquisition unit 303 acquires the shooting operation information from the imaging device 102. The operation information acquisition unit 303 may acquire the shooting operation information by polling the imaging device 102 at regular time intervals.
[0046] FIG. 6 shows shooting operation information 601 to 605 as examples of shooting operation information. Shooting operation information 601 indicates that the menu button has been changed to ON. Shooting operation information 602 indicates that the main electronic dial has been rotated by 3 (three graduations). Shooting operation information 603 indicates that the UI component "Menu 3" has been touched on the touch panel. Shooting operation information 604 indicates that the focal length of the zoom ring has been decreased by 60. Shooting operation information 605 indicates that "the position of the image capture device 102 detected by the sensor 214 is (x, y, z) = (0, 0, 160), and the attitude is (yaw, pitch, roll) = (0, 0, 0)."
[0047] The position and orientation indicated by the shooting operation information is based on an arbitrary position and orientation in real space, and is therefore used after being converted into a relative position and orientation of the imaging device 102 with respect to the information processing device 101. The operation type of the shooting operation information relates to, for example, an operation on a button, an operation on a dial, or an operation on a touch panel. The operation type of the shooting operation information relates to, for example, a focus operation (operation for controlling focus), a zoom operation (operation for controlling zoom), an aperture operation (operation for controlling aperture), or a position and orientation operation (operation for controlling position and orientation).
[0048] The captured image generating unit 304 generates a captured image (a still image or a video captured in a virtual space) of a virtual space (a virtual space experienced by a user wearing the information processing device 101). The captured image is generated using model parameters, shooting parameters, and shooting operation information.
[0049] Based on the model parameters, the captured image generating unit 304 reflects the shooting performance of the imaging device 102 (such as the zoomable magnification, the type of lens attached, or the image quality that can be captured) in the shooting performance of the virtual object 106, as shown in FIG. 1. The captured image generating unit 304 also performs adjustments so that shooting of the virtual space can be performed from the position and orientation of the virtual object 106 with the setting information indicated by the shooting parameters. Furthermore, based on the shooting operation information, the captured image generating unit 304 controls the position and orientation of the virtual object 106 and the display surface of the virtual object 106, or further adjusts the above setting information. Then, the captured image generating unit 304 generates a captured image according to the result of control using the model parameters, the shooting parameters, and the shooting operation information.
[0050] The object display unit 305 is a generation unit that generates a virtual object 106 of the imaging device 102 (an imaging device having the same appearance as the imaging device 102), and is also a display control unit that controls the display unit 208 to display the virtual object 106. Specifically, the object display unit 305 displays the virtual object 106 of the imaging device 102 in a virtual space experienced by a user wearing the information processing device 101. Model parameters and shooting operation information are used to display the virtual object 106. In addition, the object display unit 305 uses the captured image generated by the captured image generation unit 304 as a texture to be displayed on a part of the virtual object 106.
[0051] Based on the model parameters, the object display unit 305 reflects the appearance (size, shape, color, pattern, etc.) of the imaging device 102 in the appearance of the virtual object 106, as shown in FIG. 1. For example, the object display unit 305 reflects the appearance (size, shape, color, pattern, etc.) of the virtual object 106 based on the shooting operation information. The virtual object 106 is controlled in accordance with the position and orientation of the virtual object 106 or the display surface of the virtual object 106.
[0052] In addition, when a certain operation is performed on the imaging device 102 when capturing a virtual space, the same processing is performed as when the certain operation is performed on the imaging device when capturing a real space. For example, when capturing a real space, the imaging device 102 performs exposure correction for capturing the real space in accordance with the shooting operation information and the like when an exposure correction button of the imaging device 102 is pressed. On the other hand, when capturing a virtual space, the processing unit performs exposure correction for capturing the virtual space in accordance with the shooting operation information and the like when an exposure correction button of the imaging device 102 is pressed. Here, the processing unit is a functional unit that executes processing according to the operation, such as the captured image generation unit 304 and the object display unit 305.
[0053] Generation of a captured image in a virtual space will be described below with reference to the flowchart in FIG.
[0054] In step S701, the model parameter acquisition unit 301 acquires model parameters of the imaging device 102. The model parameter acquisition unit 301 stores the model parameters of the imaging device 102 as internal information of the information processing device 101. The model parameters of the imaging device 102 are acquired only once when the information processing device 101 and the imaging device 102 are connected.
[0055] In step S702, the shooting parameter acquisition unit 302 acquires shooting parameters of the imaging device 102. The shooting parameter acquisition unit 302 stores the shooting parameters of the imaging device 102 as internal information of the information processing device 101. When information on the shooting parameters previously stored is stored, the shooting parameter acquisition unit 302 overwrites and stores new shooting parameters.
[0056] In step S703, the operation information acquisition unit 303 acquires shooting operation information of the imaging device 102. The operation information acquisition unit 303 stores the shooting operation information of the imaging device 102 as internal information of the information processing device 101. The acquired shooting operation information is accumulated by queuing until processing corresponding to the shooting operation information is executed.
[0057] In step S704, the captured image generating unit 304 refers to the shooting operation information acquired in step S703 and determines whether or not a shooting instruction has been detected (a shooting instruction has been issued by the user). If it is determined that a shooting instruction has been detected, the process proceeds to step S705. If it is determined that a shooting instruction has not been detected, the process returns to step S702.
[0058] In step S705, the captured image generating unit 304 generates a captured image (an image captured in a virtual space) based on the model parameters, the shooting parameters, and the shooting operation information. That is, an image captured in a virtual space from the position and orientation of the virtual object 106 of the imaging device 102 is acquired.
[0059] As shown in Fig. 9, in capturing a real space, the capturing parameters are set in the image processing circuit of the imaging device. Then, the image processing circuit executes processing based on the capturing parameters on the input image acquired from the imaging element to generate an output image. In this way, the image processing circuit realizes capturing a real space.
[0060] On the other hand, in the case of capturing an image of a virtual space according to the first embodiment, a capturing parameter is set in the image processing logic (image processing program) executed by the information processing device 101. Therefore, the recording unit 206 of the information processing device 101 stores the image processing logic (image processing program) as software (program) for realizing the processing of the image processing circuit of the imaging device for each model (model parameters) of the imaging device. The captured image generating unit 304 stores the image processing circuit (software that reproduces the image processing circuit) in the virtual space. Therefore, in step S705, the captured image generating unit 304 uses the model parameters as a key (search key) to select the optimal image processing logic corresponding to the image processing device of the imaging device 102 from among the multiple image processing logics. By using the selected image processing logic, the captured image generating unit 304 can simulate the behavior of the image processing circuit when the shooting parameters are applied, taking into account the optical and physical environmental conditions of the virtual space. By this simulation, the captured image generating unit 304 generates an output image (captured image) from an input image (image captured with default settings in the virtual space).
[0061] 10, the position and orientation of the real image capture device 102 is reflected directly in the virtual object 106 of the image capture device 102 in the virtual space, so the shooting direction can be uniquely determined. The shooting operation information includes operation information of the position and orientation of the image capture device 102. The shooting angle of view is determined by the focal length and aperture value of the lens included in the shooting parameters. By determining the shooting direction and shooting angle of view, the range of the virtual space to be cut out as a captured image can be determined.
[0062] The captured image (image of the virtual space) generated by the captured image generating unit 304 is recorded in the recording unit 206 of the information processing device 101.
[0063] Display of a virtual object of the imaging device 102 in a virtual space will be described below with reference to the flowchart of Fig. 8. The process of the flowchart of Fig. 8 is executed in parallel with and simultaneously with the process of the flowchart of Fig. 7.
[0064] In step S801, the model parameter acquisition unit 301 acquires model parameters of the imaging device 102. The model parameter acquisition unit 301 stores the model parameters of the imaging device 102 as internal information of the information processing device 101. The model parameter acquisition unit 301 acquires the model parameters of the imaging device 102 only once when the information processing device 101 and the imaging device 102 are connected.
[0065] In step S802, the operation information acquisition unit 303 acquires shooting operation information of the imaging device 102. The operation information acquisition unit 303 stores the shooting operation information of the imaging device 102 as internal information of the information processing device 101. The acquired shooting operation information is accumulated by queuing until processing corresponding to the shooting operation information is executed.
[0066] In step S803, the object display unit 305 generates a virtual object 106 of the imaging device 102 based on the model parameters acquired in step S801 and the shooting operation information acquired in step S802. Then, the object display unit 305 displays the virtual object 106 on the display unit 208. Here, the model parameters are used to determine the appearance of the virtual object 106 (appearance of the virtual objects of the main body and lens of the imaging device 102). The correspondence between the model parameters and the virtual object is stored inside the information processing device 101. Information on the position and orientation of the imaging device 102, which is part of the shooting operation information, is used to determine the position and orientation of the virtual object 106.
[0067] There are cases where the imaging device 102 has a function for realizing a live view display (a function for displaying an image of an imaged space in real time). In this case, a captured image generated according to the flowchart of Fig. 7 may be pasted as a texture onto the display surface (live view portion) of the virtual object 106, as in the field of view image 104 shown in Fig. 1. In this case, the captured image is not generated when an instruction to capture is detected in step S704 in the flowchart of Fig. 7, but is generated periodically at a cycle for updating the image displayed on the display unit 208 of the information processing device 101.
[0068] The process of the flowchart in FIG. 8 ends when the power is turned off or an interrupt is issued to end the process.
[0069] According to the above, the model parameters, shooting parameters, and shooting operation information of the imaging device 102 are used to shoot an image of a virtual space. Therefore, shooting of a virtual space can be realized in the same way as when the imaging device 102 shoots an image of a real space in the real space.
[0070] In addition, the model parameters, shooting parameters, and shooting operation information of the imaging device 102 are used in generating the captured image, but other information may be used. For example, when the imaging device 102 is a smartphone, depending on the type of application used for shooting, the image in real space may be automatically processed (corrected). For this reason, information on the type of application used for shooting the real space (and the settings in the application) may also be used in generating the captured image. For example, the captured image generating unit 304 may acquire a captured image that has been processed (for example, by adding a frame, enlarging a person's eyes, or making the legs thinner) based on the information on the type of application.
[0071] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.
[0072] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." Therefore, unless a contradiction occurs, "equal to or greater than A" may be read as "equal to or greater than A (high; long; many)," and "equal to or less than A" may be read as "equal to or less than A (low; short; few)." And, "equal to or greater than A" may be read as "equal to or greater than A," and "equal to or less than A" may be read as "equal to or less than A."
[0073] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.
[0074] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.
[0075] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) An information processing device that captures an image in a virtual space in which a virtual object corresponding to a first imaging device that captures an image in a real space is disposed, an acquisition means for acquiring model parameters and shooting parameters of the first imaging device; an operation information acquisition means for acquiring operation information of a user with respect to the first imaging device; a generation means for generating a captured image of the virtual space from a position and an orientation of the virtual object in the virtual space based on the model parameters, the shooting parameters, and the operation information; 13. An information processing device comprising: (Configuration 2) Further comprising a plurality of image processing programs; each of the plurality of image processing programs is a program that reproduces, by software, an image processing circuit used by one imaging device to capture the real space; The generating means 1) selects an image processing program corresponding to the first imaging device from among the plurality of image processing programs in accordance with the model parameters, and 2) generates the captured image based on the selected image processing program. 2. The information processing device according to configuration 1. (Configuration 3) a processing unit configured to perform a first process for capturing an image of the virtual space when the user performs a first operation on the first imaging device, The information processing device according to configuration 1 or 2, characterized in that, when the user performs the first operation on the first imaging device to capture the real space, the first imaging device performs the first processing with respect to the capture of the real space. (Configuration 4) a second generation means for generating, as the virtual object, a second imaging device having a posture and appearance corresponding to that of the first imaging device at a position corresponding to that of the first imaging device in the virtual space based on the model parameters and the operation information; a display control means for controlling a display means to display an image of the virtual space; 4. The information processing device according to any one of configurations 1 to 3, further comprising: (Configuration 5) The first imaging device is a single-lens reflex camera, a mirrorless camera, a digital camera, an action camera, a drone, or a smartphone. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) The model parameter is information about a model number of a main body of the first imaging device, or information about a model number of a member to be attached to the main body of the first imaging device. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The member to be attached to the body of the first imaging device is a lens, a mount adapter, a strobe, a grip, or an extension finder. 7. The information processing device according to configuration 6. (Configuration 8) The shooting parameters include at least one of an aperture value, a shutter speed, an ISO sensitivity, an exposure compensation value, a lens focal length, a white balance, and a parameter related to a shooting mode. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) The operation information is information regarding at least one of an operation on a button, an operation on a dial, and an operation on a touch panel. 9. The information processing device according to any one of configurations 1 to 8. (Configuration 10) The operation information is information on an operation related to at least one of focus, zoom, aperture, and position and attitude control. 10. The information processing device according to any one of configurations 1 to 9. (Configuration 11) The captured image is a still image or a video image. 11. The information processing device according to any one of configurations 1 to 10. (method) 1. An information processing method for capturing an image in a virtual space in which a virtual object corresponding to a first imaging device that captures an image of a real space is disposed, comprising: an acquisition step of acquiring model parameters and shooting parameters of the first imaging device; an operation information acquisition step of acquiring operation information of a user with respect to the first imaging device; a generating step of generating a captured image of the virtual space from a position and an orientation of the virtual object in the virtual space based on the model parameters, the shooting parameters, and the operation information; 13. An information processing method comprising: (program) 12. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]
[0076] 101: information processing device, 102: imaging device, 301: Model parameter acquisition unit, 302: Shooting parameter acquisition unit, 303: Operation information acquisition unit, 304: Captured image generation unit
Claims
1. An information processing device that captures an image in a virtual space in which a virtual object corresponding to a first imaging device that captures an image in a real space is disposed, an acquisition means for acquiring model parameters and shooting parameters of the first imaging device; an operation information acquisition means for acquiring operation information of a user with respect to the first imaging device; a generation means for generating a captured image of the virtual space from a position and an orientation of the virtual object in the virtual space based on the model parameters, the shooting parameters, and the operation information; 13. An information processing device comprising:
2. Further comprising a plurality of image processing programs; each of the plurality of image processing programs is a program that reproduces, by software, an image processing circuit used by one imaging device to capture the real space; The generating means 1) selects an image processing program corresponding to the first imaging device from among the plurality of image processing programs in accordance with the model parameters, and 2) generates the captured image based on the selected image processing program.
2. The information processing apparatus according to claim 1,
3. a processing unit configured to perform a first process for capturing an image of the virtual space when the user performs a first operation on the first imaging device, 3. The information processing device according to claim 1, wherein when the user performs the first operation on the first imaging device to capture the real space, the first imaging device performs the first processing with respect to the capture of the real space.
4. a second generation means for generating, as the virtual object, a second imaging device having a posture and appearance corresponding to that of the first imaging device at a position corresponding to that of the first imaging device in the virtual space based on the model parameters and the operation information; a display control means for controlling a display means to display an image of the virtual space; 3. The information processing apparatus according to claim 1, further comprising:
5. The first imaging device is a single-lens reflex camera, a mirrorless camera, a digital camera, an action camera, a drone, or a smartphone.
3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.
6. The model parameter is information about a model number of a main body of the first imaging device, or information about a model number of a member to be attached to the main body of the first imaging device.
3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.
7. The member to be attached to the body of the first imaging device is a lens, a mount adapter, a strobe, a grip, or an extension finder.
7. The information processing apparatus according to claim 6,
8. The shooting parameters include at least one of an aperture value, a shutter speed, an ISO sensitivity, an exposure compensation value, a lens focal length, a white balance, and a parameter related to a shooting mode.
3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.
9. The operation information is information regarding at least one of an operation on a button, an operation on a dial, and an operation on a touch panel.
3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.
10. The operation information is information on an operation related to at least one of focus, zoom, aperture, and position and attitude control.
3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.
11. The captured image is a still image or a video image.
3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.
12. 1. An information processing method for capturing an image in a virtual space in which a virtual object corresponding to a first imaging device that captures an image of a real space is disposed, comprising: an acquisition step of acquiring model parameters and shooting parameters of the first imaging device; an operation information acquisition step of acquiring operation information of a user with respect to the first imaging device; a generating step of generating a captured image of the virtual space from a position and an orientation of the virtual object in the virtual space based on the model parameters, the shooting parameters, and the operation information; 13. An information processing method comprising:
13. A program for causing a computer to function as each of the means of the information processing device according to claim 1 or 2.
Citation Information
Patent Citations
Screen operation system by means of head-mounted display and controller cooperating with each other, program, and method
JP2016081408A
Terminal device, method, and program
JP2022041258A
