Virtual space display device, virtual space display method, program, and recording medium

Abstract

An object of the present application is to provide an apparatus, a method, a program, and a recording medium that can display a virtual space closer to a real space. In one embodiment of the present application, a processor performs processing of acquiring first information related to an actually existing camera, selecting second information related to the camera corresponding to the first information from a database in which object information related to a camera object that can be displayed in a virtual space and second information related to the camera corresponding to the object information are associated with each other, and displaying, on a display, a virtual space in which the camera object corresponding to the first information is depicted based on the object information associated with the second information.

Description

Technical Field

[0001] This invention relates to a virtual space display device, a virtual space display method, a program, and a recording medium for displaying a virtual space of a depicted object on a display. Background Technology

[0002] In technologies that provide users with images of virtual spaces such as VR (Virtual Reality) spaces or metaverse spaces, objects that move in conjunction with the user's movements, such as virtual avatars or characters, are sometimes displayed within the virtual space. In this case, the user can obtain various experiences within the virtual space through these objects, such as being able to communicate with other users within the virtual space (see, for example, Patent Document 1).

[0003] Patent Document 1 describes a technique for making a character in a virtual space act in accordance with the user's actual actions. According to this technique, two or more users can make a character displayed in the virtual space act according to facial or eye movements, thereby reproducing near-realistic communication in the virtual space.

[0004] Previous technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2017-55851 Summary of the Invention

[0007] The technical problem to be solved by the invention

[0008] When providing a virtual space to a user using the technology described in Patent Document 1, it is expected that the user will feel a sense of closeness to the virtual space if the virtual space is made closer to the real space. On the other hand, in the prior art, including the technology described in Patent Document 1, although it is possible to reproduce near-realistic interactions in the virtual space, further improvements are needed to make the virtual space even closer to the real space to the point that the user feels a sense of closeness.

[0009] One embodiment of the present invention was made in view of the above circumstances, and its object is to provide an apparatus, method, program and recording medium capable of displaying a virtual space that is closer to real space.

[0010] means for solving technical problems

[0011] The above objective is achieved by any of the following [1] to

[16] virtual space display devices.

[0012] [1] A virtual space display device includes a processor and displays a virtual space on a display screen. In the virtual space display device, the processor performs the following processing: acquiring first information related to an actual camera device; selecting second information corresponding to the first information from a database that stores object information related to a camera device object that can be displayed in the virtual space and second information related to the camera device corresponding to the object information; and displaying a virtual space on the display screen in which a camera device object corresponding to the first information and object information associated with the second information is depicted.

[0013] [2] According to the virtual space display device described in [1], the processor obtains the first information based on the camera device information obtained in the real space where the camera device is actually located.

[0014] [3] The virtual space display device according to [1] or [2], wherein the first information and the second information include information related to the image.

[0015] [4] A virtual space display device according to any one of [1] to [3], wherein the processor obtains first information based on camera device information obtained in a real space where a camera device is actually present, the camera device information being an image of a camera device actually present, the first information including feature quantities of the image of the camera device actually present, and the second information including feature quantities of the image of the camera device corresponding to the object information.

[0016] [5] The virtual space display device according to [1] or [2], wherein the first information and the second information include text information.

[0017] [6] According to the virtual space display device of [1] or [2] or [5], wherein the processor obtains first information based on camera device information obtained in a real space where a real camera device is configured, the camera device information being information output from a user terminal based on user input operations performed on the real camera device, the first information including reference information based on the output information from the user terminal, and the second information including reference information for registering the camera device corresponding to the object information.

[0018] [7] The virtual space display device according to any one of [1] to [6], wherein, in the process of selecting the second information corresponding to the first information, the processor selects the second information from the database that satisfies the selection condition in terms of similarity to the first information.

[0019] [8] The virtual space display device according to any one of [1] to [7], wherein the second information includes information on the attributes of the camera device corresponding to the object information, the function of the camera device corresponding to the attribute is associated with the camera device object, and when an action corresponding to the function is commanded to the camera device object, the processor performs function-based visual presentation in the virtual space.

[0020] [9] According to the virtual space display device described in [8], a virtual image of a user is depicted in the virtual space, which performs actions according to the user's actions. When the virtual image performs an action corresponding to a function on a camera device object, the processor performs a function-based visual presentation in the virtual space.

[0021]

[10] According to the virtual space display device described in [8] or [9], wherein the first information is information specific to the actual camera device and the second information is information specific to the camera device corresponding to the object information.

[0022]

[11] According to the virtual space display device described in

[10] , the number and type of functions that establish an association with the camera device object are determined according to the type of camera device corresponding to the object information of the camera device object.

[0023]

[12] According to the virtual space display device described in [9], the function of generating a printed object of a captured image is associated with the camera device object. When the virtual image performs a printing action on the camera device object, the processor implements the visual presentation of the printed object generated by the camera device object in the virtual space, which is equivalent to the printed object.

[0024]

[13] According to the virtual space display device described in

[12] , when the virtual image performs a printing action on the camera device object, the processor performs the following processing: it implements the visual presentation of the printable object generated by the camera device object, which includes an extracted image of the range corresponding to the camera device object in the virtual space, so that the actual existing camera device with the function of generating printables generates a printable with an image based on the extracted image.

[0025]

[14] According to the virtual space display device of

[13] , the processor causes a physically existing camera device with the function of generating prints to output a print having a composite image of an extracted image in the print object overlaid with other images.

[0026]

[15] According to the virtual space display device described in [9], the function related to the camera performed by the camera device corresponding to the object information is associated with the camera device object, and when the virtual image performs a camera-taking action on the camera device object, the processor implements a visual presentation based on the camera-related function in the virtual space.

[0027]

[16] According to the virtual space display device described in

[15] , when the virtual image performs a camera action on the camera device object, the processor performs the following processing: acquiring an extracted image of the range corresponding to the camera device object in the virtual space, and storing the extracted image in the real space in an area that can be stored by the actual existing camera device.

[0028] Furthermore, the objective of one embodiment of the present invention described above is achieved by the virtual space display method described in

[17] to

[19] below.

[0029]

[17] A virtual space display method for displaying a virtual space on a display screen, the virtual space display method comprising the following steps: a processor acquiring first information related to an actually existing camera device; the processor selecting second information corresponding to the first information from a database that stores object information related to a camera device object that can be displayed in the virtual space and second information related to the camera device corresponding to the object information; and the processor displaying on the display a virtual space depicted by a camera device object based on object information corresponding to the first information and associated with the second information.

[0030]

[18] According to the virtual space display method described in

[17] , in the step of selecting the second information corresponding to the first information, the processor selects the second information from the database whose similarity to the first information meets the selection condition.

[0031]

[19] According to the virtual space display method described in

[17] or

[18] , the second information includes information on the attributes of the camera device corresponding to the object information, which is equivalent to establishing an association between the function of the camera device and the camera device object. The virtual space display method further includes the following steps: when an action corresponding to the function is commanded to the camera device object, the processor performs a function-based visual presentation in the virtual space.

[0032] Furthermore, one embodiment of the present invention relates to a program for causing a computer to perform the steps included in the virtual space display method described in any one of

[17] to

[19] above.

[0033] Furthermore, in one embodiment of the present invention, the recording medium is a computer-readable recording medium, wherein a program is recorded for causing a computer to perform each step included in the virtual space display method described in any one of

[17] to

[19] .

[0034] Invention Effects

[0035] According to one embodiment of the present invention, by depicting an object in a virtual space that is equivalent to a camera device identified in a real space, it is possible to combine the real space and the virtual space, and provide the user with a virtual space that is closer to the real space. Attached Figure Description

[0036] Figure 1A This is a diagram representing a user using a virtual space displayed by a virtual space display device according to an embodiment of the present invention.

[0037] Figure 1B This is a diagram representing a virtual space displayed by a virtual space display device according to an embodiment of the present invention.

[0038] Figure 2A This diagram illustrates a user and an actual camera device using a virtual space display device according to an embodiment of the present invention to depict a virtual space of a camera device object.

[0039] Figure 2B This is a diagram illustrating a virtual space and a camera device object depicted by a virtual space display device according to an embodiment of the present invention.

[0040] Figure 3A It is a diagram that represents the state of a user's actions based on camera functions when commanding a camera device in real space.

[0041] Figure 3B It is a diagram that represents the state of a virtual image performing actions on a camera device in virtual space in accordance with the camera function.

[0042] Figure 4A It is a diagram representing a virtual space that generates a visual representation of printed objects.

[0043] Figure 4B It is a diagram representing the state of a printed object generated and output by an actual camera device in real space.

[0044] Figure 5 It is a diagram that represents the state of a virtual image that transmits printed objects to other users in virtual space.

[0045] Figure 6This is a diagram illustrating an information processing system that includes a virtual space display device according to one embodiment of the present invention.

[0046] Figure 7 This is a diagram representing an example of object information.

[0047] Figure 8 This is a diagram representing an example of the second piece of information.

[0048] Figure 9 This is a diagram illustrating the virtual space display process according to one embodiment of the present invention.

[0049] Figure 10 This is a diagram showing a virtual space displayed by a virtual space display device according to a variation of the present invention.

[0050] Figure 11 This is an explanatory diagram relating to a visual presentation implemented in a virtual space displayed by a virtual space display device according to a variation of the present invention.

[0051] Figure 12 This diagram illustrates a state in which a portion of a virtual space, displayed by a virtual space display device according to a variation of the present invention, is displayed on a monitor of a camera device that actually exists in real space. Detailed Implementation

[0052] The following describes specific embodiments of the present invention. In the following description, for ease of explanation, the description is sometimes given from the viewpoint of a GUI (Graphical User Interface).

[0053] Furthermore, the fundamental data processing technologies (communication / transmission technology, data acquisition technology, data recording technology, data processing / analysis technology, machine learning technology, image processing technology, image display technology, and visualization technology, etc.) used to implement this invention are well-known technologies, and therefore related descriptions are omitted.

[0054] Furthermore, regarding the equipment and apparatus used for applying the aforementioned known technologies, appropriate equipment and apparatus that can be utilized at the time of implementation of this invention may be selected.

[0055] Furthermore, in this specification, the concept of "device" includes a single device that performs a specific function, as well as a combination of multiple devices that exist separately and independently but are linked (jointly) to perform a specific function.

[0056] Furthermore, in this specification, the concept of "system" includes a system consisting of multiple devices connected in a state where they are able to communicate with each other, and a system consisting of a single device.

[0057] Furthermore, in this invention, "user" refers to a person who uses the virtual space display device of the present invention with the aim of enjoying the benefits of implementing the present invention. Specifically, a person who provides a virtual space that can be displayed through the virtual space display device of the present invention, and who utilizes the space association services described later within that virtual space, is equivalent to a user of the present invention.

[0058] Furthermore, in this specification, "person" refers to the subject performing a specific action, including individuals, groups, legal persons and organizations such as enterprises, and may also include computers and devices that constitute artificial intelligence (AI). Artificial intelligence (AI) uses hardware and software resources to realize intelligent functions such as reasoning, prediction, and judgment.

[0059] Furthermore, in this specification, the machine learning algorithms may include neural networks, convolutional neural networks, recurrent neural networks, attention, transformers, variational autoencoders, generative adversarial networks, deep learning neural networks, Boltzmann machines, matrix factorization, factorization machines, EM-based factorization machines, field-aware factorization machines, field-aware neural factorization machines, support vector machines, Bayesian networks, decision trees and random forests, and other machine learning algorithms.

[0060] Furthermore, in this specification, "space" can be three-dimensional space or two-dimensional space, but in the following description, unless otherwise stated, it will be referred to as three-dimensional space.

[0061] <<One embodiment of the present invention>>

[0062] In one embodiment of the present invention (hereinafter, the first embodiment), a virtual space is displayed on a display used by a user, and the user is able to utilize the services provided by the present invention within the virtual space.

[0063] A virtual space (hereinafter, virtual space V) is, for example, a space constructed and reproduced through a computer, specifically including Virtual Reality (VR) space, Augmented Reality (AR) space, and Mixed Reality (MR) space. Furthermore, virtual space V can include commercial virtual spaces such as the metaverse. Moreover, more than one object is depicted within virtual space V. Objects are essential components of virtual space V, such as people (specifically, virtual avatars), characters, objects, creatures, vehicles, plants, buildings, and backgrounds that can be configured within virtual space V.

[0064] like Figure 1AAs shown, the display is, for example, a head-mounted display (hereinafter, HMD30) that a user can wear on their head. By wearing the head-mounted HMD30, the user can easily immerse themselves in the virtual space V displayed on the HMD30. However, the display is not limited to the HMD30; it can also be a fixed display or a large monitor installed in a shop, etc. Furthermore, the display can also be a display with a touch panel mounted on a tablet or smartphone, or a display of a wearable device such as smart glasses.

[0065] And, as Figure 1B As shown, a humanoid object, or virtual avatar Ab, corresponding to the user is depicted in virtual space V. The virtual avatar Ab moves according to the user's actions within virtual space V. That is, the user's actions are detected by a detection device in the real space R, and the virtual avatar Ab, which moves according to the detected actions, is depicted in virtual space V. As the detection device, such as... Figure 1A As shown, wearable device 40 is capable of detecting the user's body movements.

[0066] Wearable devices 40 may include, for example, devices equipped with gyroscope sensors, devices equipped with angular velocity sensors, and devices equipped with accelerometer sensors; specifically, they may include smartwatches and motion capture devices. Furthermore, wearable devices 40 may include glove-type devices equipped with haptic sensors, such as so-called haptic feedback gloves.

[0067] In addition, the detection device is not limited to wearable device 40, but can also be a controller operated by the user or a camera that captures the user's movements.

[0068] Furthermore, within the virtual space V, the virtual avatar Ab is made to move in tandem with the user's actions in the real space R. For example, the virtual avatar Ab can perform the same hand movements as the user in the real space R. These movements include holding an object (strictly speaking, an object existing within the virtual space V), operating buttons and switches located on that object, and so on.

[0069] Furthermore, the area displayed on the HMD30 within the virtual space V—that is, the area the user visually perceives through the HMD30—is set according to the gaze of the virtual avatar Ab, and the gaze of the virtual avatar Ab can change according to the direction of the user's face. Therefore, the user can observe any area within the virtual space V through the HMD30.

[0070] Furthermore, in the first embodiment, multiple users can simultaneously utilize the same virtual space V. That is, in the first embodiment, multiple users can simultaneously observe images within the same virtual space V by specifying it. Moreover, each user can move their corresponding virtual avatar Ab (i.e., each user's virtual avatar Ab) within the same virtual space V. Thus, multiple users can communicate within the virtual space V through their respective virtual avatars Ab. Specifically, each user can move their own virtual avatar Ab and interact with the virtual avatars Ab of other users.

[0071] Here, one of the multiple users is designated as User 1, and the remaining users are designated as Users 2. Furthermore, hereinafter, User 1's virtual avatar Ab is referred to as the 1st virtual avatar, and User 2's virtual avatar Ab is referred to as the 2nd virtual avatar. User 1 can, for example, move the 1st virtual avatar and transfer an object (strictly speaking, the object itself) to the 2nd virtual avatar within the virtual space V. User 2 can utilize the transferred object within the virtual space V; specifically, User 2 can have the 2nd virtual avatar hold the object, and User 2 can observe the area within the virtual space V containing the object via HMD30.

[0072] As described above, the first user and the second user can recreate near-realistic communication in the virtual space V by moving their respective virtual avatars Ab.

[0073] Furthermore, in the first embodiment, it is possible to depict a device that actually exists in the real space R within the virtual space V; specifically, it can depict a camera device object that simulates a camera or similar imaging device. To explain in more detail, for example, as... Figure 2A As shown, the user identifies a camera present in the user's vicinity in the real space R and acquires the image information of that camera as information related to that camera (strictly speaking, camera device information, as described later). The image information of the camera can be acquired, for example, through an image sensor 32, i.e., a CCD (Charged-Coupled Device) camera, built into the HMD 30.

[0074] Then, if the camera type determined based on the acquired image information is pre-registered, such as... Figure 2B As shown, a camera device object representing a camera of the same type as the camera that acquired the image information is depicted within the virtual space V used by the user. The camera device object is one of the objects that can be displayed within the virtual space V.

[0075] That is, regarding an actual camera captured by image sensor 32, if its information (specifically, second information) is registered in database 12 (described later), an imaging device object corresponding to that actual camera is depicted in virtual space V. The virtual space V depicting the imaging device object is displayed on HMD 30, therefore... Figure 2B As shown, users can visually identify camera devices within the virtual space V.

[0076] Furthermore, users can use the virtual avatar Ab to move the aforementioned camera device as if it were a real camera device. Specifically, for example... Figure 3A and 3B As shown, it is possible to photograph objects using a camera device within a virtual space V. This effect allows the virtual space V to be combined with the real space R, making the virtual space V more closely resemble the real space R. As a result, a more intimate virtual space V can be provided to the user.

[0077] The following is an example of providing services to users by uniting virtual space V with real space R (hereinafter referred to as spatial union service). Furthermore, the camera device actually existing in real space R will be referred to as the actual camera Gc, and the camera device object depicted in virtual space V will be referred to as the virtual camera Oc.

[0078] like Figure 2B and 3B As shown, the virtual camera Oc has the same appearance as the corresponding physical camera Gc. The virtual camera Oc includes operating parts such as switches and buttons, arranged in the same configuration as the physical camera Gc. Furthermore, the virtual camera Oc is configured with the attributes of the corresponding physical camera Gc. The attributes of the physical camera Gc refer to the properties of the imaging device, specifically including the type, model and type, form, specifications, structure, manufacturer (manufacturing company), and other attributes related to the function of the physical camera Gc. Furthermore, examples of types of physical camera Gc include, for example, digital cameras, instant cameras, and film cameras.

[0079] Furthermore, the functions of the actual camera Gc, which are equivalent to the attributes set on the virtual camera Oc, are associated with the virtual camera Oc. The association between the functions of the actual camera Gc and the virtual camera Oc means that the necessary information required for the scene reproduced by the virtual camera Oc to perform the functions of the actual camera Gc, specifically, information related to the visual presentation corresponding to that function (which will be discussed later), is stored in association with the virtual camera Oc.

[0080] Here, the equivalent attribute "actual camera Gc" refers to an actual camera Gc belonging to a category or type of camera that is one of its attributes. Furthermore, the function of the actual camera Gc is related to image capture and image output. Examples of image capture-related functions include the ability to perform image capture and the ability to adjust image capture conditions. Examples of adjusting image capture conditions include zoom, autofocus, and telephoto functions. As for image output-related functions, in the case of an instant camera, for example, the ability to generate a printout of the captured image.

[0081] Then, the user can assign commands to the virtual camera Oc and the corresponding actions to the functions associated with that virtual camera Oc. Specifically, such as... Figure 3A and 3B As shown, the user moves the virtual avatar Ab, causing Ab to perform actions corresponding to the functions of the virtual camera Oc. Specifically, the user causes the virtual avatar Ab to press the button on the virtual camera Oc that corresponds to that function. Thus, the user can command the virtual camera Oc to perform the actions corresponding to the aforementioned functions.

[0082] Then, when the virtual avatar Ab performs an action corresponding to the function of the virtual camera Oc, a visual presentation based on that function is implemented in the virtual space V. Function-based visual presentation refers to reproducing the scene in the virtual camera Oc where the actual camera Gc performs that function; specifically, it involves displaying (playing) an image of the same scene within the virtual space V.

[0083] To illustrate the visual presentation in more detail, consider the following scenario: an actual camera Gc, capable of printing captured images, exists in the real space R, while a virtual camera Oc, simulating the actual camera Gc, is depicted in the virtual space V.

[0084] In this case, the actual camera Gc is an instant camera and has a printout generation function. That is, the actual camera Gc stores one or more sheets of photosensitive film inside the camera body, captures an image within the field of view by pressing the shutter button, and prints the captured image onto the photosensitive film. Then, the photosensitive film with the captured image is output as a printout, specifically, ejected from the camera body (see reference). Figure 4B ).

[0085] On the other hand, such as Figure 4A As shown, a user utilizing the virtual space V can move the virtual avatar Ab, causing the virtual avatar Ab to hold a virtual camera Oc that simulates the actual camera Gc as a real-time camera. Furthermore, the user can move the virtual avatar Ab and perform actions on the virtual camera Oc corresponding to the aforementioned printout generation function. In other words, as... Figure 4AAs shown, the virtual avatar Ab performs actions on the virtual camera Oc corresponding to the print output function according to the user's commands, such as pressing the part of the virtual camera Oc that corresponds to the shooting button.

[0086] Having performed the aforementioned actions, this serves as a trigger to implement a visual presentation based on the printed object generation function within the virtual space V. Specifically, as follows: Figure 4A As shown, a portion of a virtual space V is captured by a virtual camera Oc, and the image of a printed object simulating the captured image is displayed (played) within the virtual space V, output from the virtual camera Oc. Hereinafter, the printed object will be referred to as the printed object Po.

[0087] Here, capturing an image based on a virtual camera Oc refers to extracting an image from a virtual space V that corresponds to the orientation of the virtual camera Oc. In other words, capturing an image based on a virtual camera Oc means extracting (capturing) a range from an image in virtual space V that corresponds to the orientation of the virtual camera Oc; ​​specifically, extracting (capturing) a range equivalent to the field of view of the virtual camera Oc and acquiring this extracted image.

[0088] And, as Figure 4A As shown, by directing the virtual camera Oc towards the virtual avatar Ab to take a picture, an image (strictly speaking, an extracted image) can be obtained with the virtual avatar Ab as the subject. If multiple virtual avatars Ab exist within the virtual space V, and two or more virtual avatars Ab are within the field of view of the virtual camera Oc, then a printed object Po displaying the images of these two or more virtual avatars Ab is shown within the virtual space V. Thus, it is possible to recreate within the virtual space V a scene that appears to be a group photo of multiple users' virtual avatars Ab and output that printed object.

[0089] Furthermore, users can move their virtual avatar Ab while holding a printed object Po, thus carrying the printed object Po within the virtual space V. Moreover, users can move their own virtual avatar Ab within the virtual space V, placing the printed object Po near other users' virtual avatars Ab, thereby displaying it to other users. And, as... Figure 5 As shown, within the virtual space V, a user can directly transfer a printed object Po to another user's virtual avatar Ab using their own virtual avatar Ab. Thus, a user can transfer a printed object Po to another user within the virtual space V.

[0090] Furthermore, when the aforementioned visual presentation is implemented in the virtual space V, that is, when the printed object Po is output from the virtual camera Oc, such as Figure 4BAs shown, in conjunction with this, a printed object P is also generated on the real space R side and output from the actual camera Gc. More specifically, in the actual camera Gc, which acts as an instantaneous camera, an image based on the printed object Po in the virtual space V side (strictly speaking, an extracted image) is actually printed onto the photosensitive film within the actual camera Gc. The image in the printed object Po based on the printed image can be the same image as the printed image, or, as... Figure 4B As shown, it can also be a composite image in which other images are superimposed on the printed image. Other images superimposed on the printed image can include decorative images representing patterns or designs, images of characters, stamp images for messages, and images that apply effects to background images in the printed image.

[0091] Then, as Figure 4B As shown, the printed object P bearing the aforementioned image is output from the actual camera Gc. Thus, the user can obtain the printed object P in the real space R, which is equivalent to the printed object Po in the virtual space V. Consequently, the virtual space V can be more effectively integrated with the real space R, providing the user with a more intimate virtual space V.

[0092] Furthermore, in the real space R, if a printed object P is generated and output through an actual camera Gc, the photosensitive film within the actual camera Gc is consumed, and the remaining amount of photosensitive film decreases accordingly. Consequently, in the virtual space V, the remaining number of times a printed object Po can be generated through a virtual camera Oc also decreases; in other words, the remaining number of times a visual representation related to the generation of the printed object Po can be performed decreases.

[0093] Furthermore, in the real space R, by filling the actual camera Gc with new photosensitive film, the remaining quantity of photosensitive film can be increased. In this case, on the virtual space V side, the remaining number of times that visual representation related to the generation of the printed object Po can be performed also increases.

[0094] <<Structure Example of a Virtual Space Display System>>

[0095] refer to Figure 6 The information processing system (hereinafter referred to as virtual space display system 100) including the virtual space display device according to the first embodiment will be described.

[0096] The virtual space display system 100 is a system built to provide users with a virtual space V and related spatial services that can be experienced within the virtual space V. For example... Figure 6 As shown, the virtual space display system 100 consists of a virtual space display device 10 and a user unit 20. Additionally, in Figure 6In this system, the number of user units 20 is 2, but it is not limited to this. In fact, there are a number of user units 20 corresponding to the number of users who utilize the virtual space V and the space joint service.

[0097] User unit 20 consists of equipment used by the user, such as Figure 6 As shown, it includes a user terminal 22, an HMD 30, an image sensor 32, and a wearable device 40.

[0098] When a user utilizes virtual space V and related spatial services, HMD 30 and wearable device 40 are worn on the user's body. During the user's use of virtual space V, HMD 30 displays virtual space V, and wearable device 40 detects the user's movements. Image sensor 32, for example, is mounted within HMD 30 to capture images of the user's face in the real space R, corresponding to the user's position and orientation, during the user's use of virtual space V, and acquires the captured images.

[0099] User terminal 22 is a physical terminal that operates when a user utilizes the virtual space V and the space-linked services. That is, user terminal 22 accepts a series of input operations performed by the user in the real space R in order to utilize the virtual space V and the space-linked services. The input operations accepted by user terminal 22 include operations performed directly on user terminal 22 and operations performed through wearable devices 40, etc.

[0100] User terminal 22 can be composed of a personal computer, smartphone, mobile phone, tablet terminal, or wearable terminal, etc. Furthermore, user terminal 22 can also be a computer housed within the casing of HMD30 and integrated with HMD30. For example... Figure 6 As shown, the user terminal 22 includes a processor 22a, a memory 22b, and a communication interface 22c. The structure of these devices is based on the processor 10a, memory 10b, and communication interface 10c included in the virtual space display device 10. Furthermore, as... Figure 6 As shown, the user terminal 22 also includes an input device 22d consisting of a touch panel, mouse, and keyboard, and an output device 22e consisting of a display and speakers. The input device 22d may include a microphone as a sound collector and a camera as an image reader.

[0101] Furthermore, the user terminal 22 is equipped with an OS (Operating System) program and a specified application program (hereinafter referred to as App). By launching the App through the user terminal 22, the user can use the user unit 20 to utilize the virtual space V and space association services. Specifically, by launching the App, the user logs into the virtual space V, whereby the user's virtual avatar Ab appears within the virtual space V.

[0102] Furthermore, the user terminal 22 can communicate with the HMD 30, the image sensor 32, and the wearable device 40, and send and receive data with these devices. Specifically, the user terminal 22 sends display data of the virtual space V to the HMD 30. If the HMD 30 receives the display data, it displays the virtual space V in the HMD 30 according to the display data. Additionally, image data (image data) captured by the image sensor 32 is sent from the image sensor 32 to the user terminal 22. Furthermore, motion data (motion data) detected by the wearable device 40 related to the user's actions is sent from the wearable device 40 to the user terminal 22.

[0103] Furthermore, the user terminal 22 can communicate with the actual camera Gc and control the actual camera Gc to enable it to function. Specifically, if the user terminal 22 receives a control command from the virtual space display device 10, it controls the actual camera Gc according to the control command.

[0104] Furthermore, the communication method between each of the HMD30, image sensor 32, wearable device 40, and actual camera Gc and the user terminal 22 can be either wired or wireless. In the wireless communication method, Bluetooth, Wi-Fi, infrared communication, and other wireless communication methods can be used.

[0105] The virtual space display device 10 is composed of a computer, specifically a personal computer, workstation, or server. When the virtual space display device 10 is composed of a server, this server can be a cloud service server. Cloud service servers include ASP (Application Service Provider), SaaS (Software as a Service), PaaS (Platform as a Service), or IaaS (Infrastructure as a Service) servers. In this case, if the required information is input into the client terminal, the server performs various processing and calculations based on the input information, and the calculation results are output on the client terminal side. That is, the functions provided by the server can be utilized on the client terminal side.

[0106] In addition, the virtual space display device 10 can be composed of a single computer or multiple computers distributed in parallel.

[0107] If the structure of the virtual space display device 10 is described, then as follows: Figure 6As shown, the computer constituting the virtual space display device 10 has a processor 10a, a memory 10b, a communication interface 10c, and a storage device 10d as hardware devices.

[0108] The processor 10a may be composed of a CPU (Central Processing Unit), MPU (Micro-Processing Unit), MCU (Micro Controller Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), TPU (Tensor Processing Unit), or ASIC (Application Specific Integrated Circuit).

[0109] The memory 10b can be composed of semiconductor memories such as ROM (Read Only Memory) and RAM (Random Access Memory).

[0110] The communication interface 10c can be composed of, for example, a network interface card or a communication interface board. There are no particular limitations on the standard for data communication based on the communication interface 10c. Examples include communication via Wi-Fi (registered trademark) based wireless LAN, communication based on 3G to 5G or later generations of mobile communication systems, or communication based on LTE (Long Term Evolution).

[0111] The virtual space display device 10 can communicate with each user's terminal 22 via the communication interface 10c. Specifically, the virtual space display device 10 sends information (hereinafter referred to as virtual space information) to the user terminal 22 for displaying the virtual space V through each user's HMD 30. The user terminal 22 generates display data for the virtual space V based on the virtual space information received from the virtual space display device 10, and thus, the virtual space V based on the generated display data is displayed on the HMD 30.

[0112] Furthermore, the virtual space display device 10 can acquire information from images captured by the image sensor 32 and information related to the user's actions detected by the wearable device 40 through communication with the user terminal 22. The virtual space display device 10 then edits virtual space information based on the acquired information and sends the edited virtual space information to the user terminal 22. The user terminal 22 then generates display data for the virtual space V based on the edited virtual space information. Thus, the virtual space V, depicting a virtual avatar Ab that moves in conjunction with the user's actions and objects such as camera devices, is displayed in the HMD 30.

[0113] Furthermore, when the image sensor 32 captures an image of the actual camera Gc in the real space R, the virtual space display device 10 analyzes the image captured by the actual camera Gc and determines the feature quantities associated with the captured image. Thus, the virtual space display device 10 is able to acquire information (hereinafter referred to as first information) containing the feature quantities of the image captured by the actual camera Gc.

[0114] Image features are obtained by quantifying the features of an image (e.g., the type, location, size of the subject in the image, the scene being photographed, and the image quality). More specifically, features are represented by numerical values, vectors, or tensors.

[0115] In addition, as a technique for determining the feature quantities of an image, known image analysis techniques, such as those used for subject recognition, can be utilized.

[0116] The first information is an attribute of the actual camera Gc; more specifically, it is model-specific information used for comparison in the selection process described later. Furthermore, in the first embodiment, the first information includes image-related information, specifically, as described above, it includes feature quantities of the captured image obtained by the image sensor 32 from the actual camera Gc. Here, the captured image of the actual camera Gc corresponds to the "camera device information" of the present invention, which is information that can be obtained in the real space R where the actual camera Gc is located.

[0117] Furthermore, the first information is not limited to information containing the feature quantity of the captured image; it can also include the captured image itself, or more specifically, information containing the grayscale values ​​of the individual pixels that constitute the captured image.

[0118] Furthermore, the virtual space display device 10 can control the actual camera Gc via communication with the user terminal 22, thereby enabling the actual camera Gc to perform its functions. Specifically, the virtual space display device 10 generates control command data and sends it to the user terminal 22. If the user terminal 22 receives the control command data, it determines the content of the control command based on the data and controls the actual camera Gc according to the determined content of the control command. Thus, the virtual space display device 10 can, for example, control the actual camera Gc to save a specified image in an area that can be stored by the actual camera Gc, specifically, in the memory within the actual camera Gc or in a recording medium such as an SD memory card. Furthermore, when the actual camera Gc is a real-time camera with the function of generating a printed object P, the virtual space display device 10 can control the actual camera Gc to generate a printed object P with a specified image printed on it.

[0119] The storage device 10d can be composed of flash memory, HDD (Hard Disc Drive), SSD (Solid State Drive), FD (Flexible Disc), MO (Magneto-Optical Disc), CD (Compact Disc), DVD (Digital Versatile Disc), SD card (Secure Digital Card), or USB memory (Universal Serial Bus memory), etc. The storage device 10d can be built into the computer constituting the virtual space display device 10, or it can be externally installed on the computer body, or it can be a NAS (Network Attached Storage), etc. Furthermore, the storage device 10d can also be composed of a third computer, such as a database server or online storage, that is communicatively connected to the virtual space display device 10.

[0120] The storage device 10d stores and accumulates various information required for providing space federation services, thereby constructing... Figure 6 Database 12 is shown. In database 12, object information related to the virtual camera Oc, which is the object of the imaging device, and second information related to the actual camera Gc corresponding to the object information are stored in association with each other.

[0121] To elaborate further, database 12 stores object information for each of the various virtual cameras Oc that can be displayed in virtual space V, categorized by virtual camera. For example... Figure 7As shown, each object information includes the identification ID (object ID) of the virtual camera Oc, the appearance image of the virtual camera Oc, its attributes, and the functions associated with the virtual camera Oc. The attributes in the object information are the attributes of the actual camera Gc corresponding to the object information of the virtual camera Oc; ​​specifically, they include the type and model of the actual camera Gc. The functions associated with the virtual camera Oc are the functions possessed by the actual camera Gc corresponding to the object information of the virtual camera Oc; ​​in other words, they are the functions of the actual camera Gc equivalent to the aforementioned attributes.

[0122] Here, the actual camera Gc corresponding to the object information of the virtual camera Oc is the actual camera Gc that has the attributes contained in the object information of the virtual camera Oc, such as the actual camera Gc of a model that has the same appearance as the virtual camera Oc.

[0123] Furthermore, the number and types of functions associated with the virtual camera Oc are determined based on the model of the actual camera Gc corresponding to the object information of the virtual camera Oc. That is, in a virtual camera Oc that simulates an actual camera Gc with functions f1, f2, ..., fn (where n is a natural number), functions f1, f2, ..., fn are associated. For example, when the object information of the actual camera Gc, which is a digital camera, corresponds to the object information of the virtual camera Oc, functions such as recording, displaying captured images, and saving captured images are associated with the virtual camera Oc. Furthermore, when the object information of the actual camera Gc, which is an instant camera, corresponds to the object information of the virtual camera Oc, functions such as recording and generating printed copies of captured images are associated with the virtual camera Oc.

[0124] And, as Figure 7 As shown, the object information regarding the visual presentation based on the function associated with the virtual camera Oc also includes its presentation content and the conditions for starting the presentation. The presentation content is the specific content of the scene reproduced within the virtual space V, which is the image information of that scene. The conditions for starting the presentation are information representing the user's action detected in the real space R that triggers the execution of the visual presentation.

[0125] In addition, the object information may also include information other than those mentioned above.

[0126] The second information is information about the actual camera (actual camera Gc) corresponding to each of the object information stored in database 12, and it is registered in database 12 along with the object information. Therefore, the database 12 stores the same amount of the second information as the object information. Furthermore, the second information is the attribute of the actual camera Gc corresponding to the object information, specifically model-specific information, which is used as reference information in the selection process described later.

[0127] like Figure 8 As shown, each piece of second information includes the feature values ​​of the image from the actual camera Gc, the attributes of the actual camera Gc, and the object ID (identification ID of the virtual camera Oc) contained in the object information corresponding to the actual camera Gc. Thus, since the second information contains the object ID, the second information and the object information are associated with each other through the object ID.

[0128] The image features included in the second information are information related to the image of the actual camera Gc, determined by analyzing the image representing the appearance of the actual camera Gc. Furthermore, as with the first information, known image analysis techniques, such as those used for subject recognition, can be utilized to determine the image features.

[0129] The second piece of information contains attributes related to the actual camera GC corresponding to the object information, specifically information related to the type and model of the actual camera GC. This attribute information can be obtained from the actual camera GC's catalog or instruction manual, or from information provided by the actual camera GC's manufacturer.

[0130] In addition, the second information may also include information other than that mentioned above. Furthermore, the information in the second information related to the image of the actual camera Gc is not limited to the image's feature values; it may also be the image itself, or more specifically, information about the grayscale values ​​of the individual pixels that constitute the image.

[0131] The amount of object information and second information accumulated in database 12 increases by storing new object information and second information. For example, if a new type of camera is developed in real space R and put on the market, second information is generated for that camera, and object information is generated for a virtual camera Oc that imitates that camera. Then, the generated object information and second information are associated with each other and re-stored in database 12.

[0132] Returning to the description of the structure of the virtual space display device 10, an operating system (OS) program and a virtual space display information processing program are installed in the computer constituting the virtual space display device 10. These programs are equivalent to the "programs" of this invention, and are programs that control the computer constituting the virtual space display device 10. That is, by executing the above-mentioned programs by the processor 10a, the computer performs the function of the virtual space display device 10. Specifically, the processor 10a performs a series of processes described later according to the above-mentioned programs for the purpose of providing virtual space V and space association services.

[0133] In addition, the aforementioned virtual space display information processing program can be obtained by reading from a computer-readable recording medium (media), or it can be obtained (downloaded) via a network such as the Internet or intranet.

[0134] <<Operational Examples of Virtual Space Display Devices According to the First Embodiment>>

[0135] In the virtual space display device 10 according to the first embodiment, the processor 10a of the computer constituting the virtual space display device 10 executes, with the purpose of providing virtual space V and space association services to users. Figure 9 The series of processes shown. Hereinafter, the process executed based on processor 10a will be referred to as the virtual space display process, see reference. Figure 9 The process of displaying virtual space is explained.

[0136] The virtual space display method of this invention is used in the virtual space display process. That is, Figure 9 The steps (processes) in the virtual space display process shown correspond to the constituent elements of the virtual space display method of the present invention. However, Figure 9 The flowchart shown in the figure is only an example. Without departing from the spirit of the present invention, some steps in the flowchart may be deleted, new steps may be added to the flowchart, or the execution order of two steps in the flowchart may be changed.

[0137] Before implementing the virtual space display process, the database 12 is pre-built. That is, when implementing the virtual space display process, the database 12 stores the object information of the virtual camera Oc and the second information of the actual camera Gc corresponding to the object information in an associated state.

[0138] In each step of the virtual space display process, the processor 10a mainly reads the information processing program for virtual space display and executes the data processing corresponding to each step. Furthermore, during the processing of each step, the processor 10a appropriately communicates with the user terminal 22 of the user who wants to use the virtual space V (hereinafter also referred to as the user), sending to or receiving from the user terminal 22 the information required to utilize the virtual space V.

[0139] Specifically, if a user launches an App on user terminal 22, processor 10a is triggered to begin display processing (S001). During display processing, processor 10a communicates with user terminal 22 to display the virtual space V on the HMD 30 worn by the user. The virtual avatar Ad of the user is depicted in the virtual space V.

[0140] Furthermore, if an action is displayed in the virtual space V on the user's HMD30, the action is detected using the wearable device 40 worn by the user. In this case, the processor 10a obtains information about the detected user's action through communication with the user terminal 22 and controls the display of the virtual space V so that the virtual avatar Ab performs actions in conjunction with the movement.

[0141] Furthermore, assuming that during the period when the virtual space V is displayed in the HMD30, the image sensor 32 captures an image of an actual camera Gc positioned around the user in the real space R (S002). In this case, the captured image is sent from the user terminal 22 to the virtual space display device 10 as camera device information. If the captured image is received (S003), the processor 10a executes an acquisition process (S004). The acquisition process is performed based on the captured image from the actual camera Gc; specifically, in this acquisition process, the captured image is analyzed to acquire first information containing the feature quantities of the captured image.

[0142] After acquiring the first information, processor 10a executes a selection process (S005). In the selection process, second information corresponding to the first information acquired in the previous step S004 is selected from database 12. That is, in the selection process, the first information and the second information stored in database 12 are compared respectively, and the second information of a camera of the same or similar type as the actual camera Gc captured by image sensor 32 is searched.

[0143] Specifically, in the selected process, processor 10a calculates the similarity between the feature quantities of the image contained in the first information and the feature quantities of the images contained in each of the second information stored in database 12. At this time, known techniques such as those for calculating the similarity of image feature quantities can be used, including those for calculating Mahalanobis distance, Euclidean distance, or cosine similarity as a measure of similarity. Furthermore, artificial intelligence (AI) can be used as a method for calculating the similarity of image feature quantities.

[0144] Then, processor 10a selects from the second information stored in database 12 the second information that satisfies the selection criteria of the first information in terms of similarity. The selection criteria, for example, is that the similarity to the first information is the greatest. However, it is not limited to this; the similarity to the first information can also be the range from the maximum to the k-th digit (k is a natural number). Alternatively, the similarity to the first information can be a value above a set benchmark.

[0145] After performing the selection process, processor 10a performs a drawing process (S006). In the drawing process, object information stored in database 12 that is associated with the second information selected in the previous step S005 is determined. Here, the second information selected in step S005 is selected based on the first information obtained in step S004, so the object information associated with the second information can be said to be the object information corresponding to the first information.

[0146] Then, in the rendering process, the processor 10a renders a virtual camera Oc within the virtual space V based on the object information determined by the above-described method, and displays the virtual space V rendering the virtual camera Oc on the HMD 30 for the user. Thus, the user observing the virtual space V displayed on the HMD 30 can identify the virtual camera Oc within the virtual space V, which simulates the actual camera Gc positioned near the user in the real space R. As a result, the virtual space V can be combined with the real space R, providing a virtual space V that creates a greater sense of intimacy for the user.

[0147] Observing the use of virtual space V, the user can enable the virtual avatar Ab to hold a virtual camera Oc depicted within virtual space V (see reference). Figure 3B and Figure 4A Furthermore, the user can use the virtual avatar Ab to command the virtual camera Oc and perform actions corresponding to the functions associated with that virtual camera Oc. To explain this in detail, let's assume that the function of the printed object P that generates the captured image, i.e., the function of the real-time camera, is associated with the virtual camera Oc depicted in the virtual space V.

[0148] After the rendering process is executed, the virtual avatar Ab is configured to interact with the user's actions in the real space R, performing an action on the virtual camera Oc corresponding to the function of generating printed materials, i.e., a printing action (S007). Specifically, the virtual avatar Ab is configured to perform the action of the user pressing the part of the virtual camera Oc that corresponds to the shooting button. The processor 10a is thus triggered to execute the rendering process (S008).

[0149] In the performance processing, processor 10a performs a visual presentation based on the aforementioned functions within a virtual space V. Specifically, it performs a visual presentation within the virtual space V of a printed object Po, generated by a virtual camera Oc, which is equivalent to a printed object. In this visual presentation, the printed object Po output from the virtual camera Oc includes an extracted image of the range corresponding to the orientation of the virtual camera Oc at the moment when the aforementioned printing action was performed in the virtual space V. That is, in the visual presentation, an image of a printed object is displayed (played) within the virtual space V as if the range corresponding to the field of view of the virtual camera Oc was captured by the virtual camera Oc, and the captured image is printed to generate the printed object.

[0150] As described above, in the first embodiment, by implementing the aforementioned visual presentation, it is possible to reproduce the state in which the virtual camera Oc, simulating the actual camera Gc, functions within the virtual space V. As a result, the virtual space V can be more effectively integrated with the real space R, creating a virtual space V that provides a greater sense of intimacy for the user.

[0151] Furthermore, it can be observed that users of virtual space V can have their virtual avatar Ab hold a printed object Po depicted within virtual space V and move it to any location within virtual space V. Moreover, by moving their own virtual avatar Ab, users can transfer the printed object Po held by their virtual avatar Ab to other users' virtual avatars Ab, thereby transferring the printed object Po to other users.

[0152] Furthermore, after implementing the aforementioned visual presentation, the processor 10a further executes control processing (S009). In this control processing, the processor 10a, in conjunction with the user terminal 22, controls the actual camera Gc in the real space R, enabling the actual camera Gc to perform its function (specifically, the function of generating printed matter). Through this control processing, the actual printed matter P is generated by the actual camera Gc in the real space R and output from the actual camera Gc (see reference). Figure 4B ).

[0153] An image is printed on the printout P output from the actual camera Gc, based on a printed image (i.e., an extracted image that extracts a portion of the virtual space V) from a printout object Po depicted within a virtual space V. This image can be the same as the printed image in the printout object Po, or it can be a composite image overlaid with other images. In the case of printing a composite image, the other images overlaid on the printed image can be determined based on the printed image, specifically, based on the expression of the virtual avatar Ab contained in the printed image, the atmosphere created by the printed image, or the scene of the printed image, etc.

[0154] As described above, in the first embodiment, in conjunction with the output of a printed object Po from a virtual camera Oc in virtual space V, a printed object P is output from a physical camera Gc in real space R. This allows the visual presentation implemented in virtual space V to be reproduced in real space R. As a result, virtual space V and real space R can be more effectively combined, providing a virtual space V that creates a greater sense of intimacy for the user.

[0155] In the above series of steps, steps S007 to S009 are repeated until the user finishes using the virtual space V (S010). Then, when the user finishes using the virtual space V, the virtual space display process ends. Furthermore, the virtual space display process is implemented whenever the user launches the App on the user terminal 22 and logs in to the virtual space V as the user.

[0156] <<Other Implementation Methods>>

[0157] The specific embodiments of the present invention have been described above, but these embodiments are merely examples for ease of understanding and do not limit the present invention. That is, the present invention can be modified or improved from the embodiments described below without departing from its spirit. Furthermore, the present invention includes equivalents. In addition, embodiments of the present invention may include combinations of the above embodiments with one or more of the following variations.

[0158] (Variations on information 1 and information 2)

[0159] In the first embodiment, the first and second information used as comparison information in the selected process are assumed to be image-related information, specifically, information containing image feature quantities. However, this is not a limitation; for example, the first and second information may include information other than image feature quantities, and this information may be used as comparison information. This is the second embodiment of the present invention, and the second embodiment will be described below.

[0160] In the second embodiment, during the acquisition process in the virtual space display flow, the processor 10a of the virtual space display device 10 acquires camera device information that can be acquired in the real space R for the actual camera Gc. Here, the camera device information is information output from the user terminal 22 based on user input operations performed on the actual camera Gc. Specifically, the user performs an input operation by inputting the model, device name, or serial number of the actual camera Gc through the input device 22d of the user terminal 22. In this case, the text information input by the user is output from the user terminal 22 as camera device information.

[0161] If the processor 10a acquires camera device information (i.e., output information from the user terminal 22), it acquires first information based on that information. The first information includes text information, specifically text information input by the user. As described above, this text information represents the model, device name, or serial number of the actual camera Gc, and is used as reference information in the selected process. That is, in the second embodiment, the first information includes the aforementioned text information as reference information based on the output information from the user terminal 22. This text information is model-specific information of the actual camera Gc.

[0162] On the other hand, database 12 stores object information for the virtual camera Oc and second information used for comparison, registered for the actual camera (actual camera Gc) corresponding to the object information. In the second embodiment, the second information is text information, specifically, information indicating the model, device name, or serial number of the actual camera Gc. This text information is model-specific to the actual camera Gc (i.e., the actual camera corresponding to the object information).

[0163] Furthermore, in the second embodiment, during the selection process, the processor 10a compares the acquired first information with the second information stored in the database 12, and selects the second information corresponding to the first information. Specifically, the processor 10a calculates the similarity between the text information contained in the first information and the text contained in each of the second information pieces, and selects the second information whose similarity to the first information satisfies the selection criteria.

[0164] In addition, known techniques can be used to calculate the similarity of textual information.

[0165] As described above, in the second embodiment, information other than image feature quantities is used as the first and second information. As a result, in the second embodiment, the user can utilize the spatial fusion service by inputting the actual camera Gc model, device name, or serial number, etc., into the user terminal 22.

[0166] Furthermore, in the second embodiment, the user inputs the model, device name, or serial number of the actual camera Gc through the input device 22d of the user terminal 22, but is not limited to this. For example, the camera built into the user terminal 22 can read identification symbols such as QR codes marked on the body of the actual camera Gc. In this case, the image read from the identification symbol is equivalent to camera device information.

[0167] Furthermore, by analyzing the image read from the identification symbol, the first information containing the information of that identification symbol can be obtained. The second information stored in database 12 can contain the identification symbol of the actual camera (the actual camera) corresponding to the object information. Thus, the first information and each piece of second information stored in database 12 can be compared, and the second information whose identification symbol is consistent with or similar to the first information can be selected as the second information corresponding to the first information.

[0168] (Examples of variations regarding presentation processing and control processing)

[0169] In the first embodiment, the function of generating printed objects is associated with the virtual camera Oc. Furthermore, in the first embodiment, if the virtual avatar Ab performs a printing action on the virtual camera Oc in the virtual space V, a visual representation of a printed object Po generated by the virtual camera Oc is implemented. Also in the first embodiment, in the real space R, the actual camera Gc is controlled in conjunction with the implementation of the visual representation, and the actual printed object P is generated by the actual camera Gc.

[0170] However, the presentation processing and control processing are not limited to the above-described content, and other examples can be considered. The third embodiment of the present invention will be described as performing presentation processing and control processing different from the above-described content. Hereinafter, reference will be made to... Figures 10-12 The third embodiment will be described.

[0171] In describing the third embodiment, it is assumed that an actual camera Gc, which is a digital camera, is positioned near the user in the real space R. In this case, object information related to the type of the digital camera and second information about the actual camera (i.e., the actual camera Gc) corresponding to that object information are stored in the database 12 in an associated manner. Therefore, if the actual camera Gc takes a picture in the real space R using the image sensor 32, the processor 10a of the computer constituting the virtual space display device 10 performs the acquisition processing, selection processing, and drawing processing in the same steps as in the first embodiment. As a result, as Figure 10 As shown, the virtual camera Oc, which simulates the actual camera Gc, is depicted in the virtual space V.

[0172] The functions related to shooting based on the actual camera Gc as a digital camera are associated with the virtual camera Oc depicted in the virtual space V. The shooting-related functions (hereinafter, video recording functions) include functions for acquiring captured images and functions for adjusting shooting conditions, such as zoom functions, autofocus functions, and telephoto functions.

[0173] In the third embodiment, assuming the user moves the virtual avatar Ab, and the virtual avatar Ab performs a camera-taking action on the virtual camera Oc, such as pressing a part equivalent to the camera's shutter button. Then, the processor 10a performs a camera-based visual presentation within the virtual space V. In this visual presentation, an image of the scene captured within the field of view of the virtual camera Oc in the virtual space V is displayed within the virtual space V. Here, capturing an image within the field of view of the virtual camera Oc means extracting (capturing) the range in the virtual space V corresponding to the orientation of the virtual camera Oc and acquiring its extracted image. Therefore, for example, if the virtual camera Oc performs a camera-taking action while facing the virtual avatar Ab, an image of the virtual avatar Ab is captured in the aforementioned visual presentation.

[0174] In addition, the images captured by the virtual camera Oc, i.e., a portion of the virtual space V, will be referred to as "captured images in the virtual space V".

[0175] Furthermore, in the aforementioned visual presentation, such as Figure 11 As shown, in the virtual camera Oc, the image captured in the virtual space V is displayed in the portion corresponding to the rear monitor (hereinafter, monitor portion Om). Strictly speaking, the image in monitor portion Om in the virtual space V is replaced with the captured image in the virtual space V. Therefore, by observing monitor portion Om of the virtual camera Oc in the virtual space V, the user can confirm the captured image in the virtual space V. Furthermore, if there are other users' virtual avatars Ab (hereinafter, other virtual avatars) in the virtual space V, the user can move their own virtual avatar Ab to move the virtual camera Oc to the position of the other virtual avatars and make monitor portion Om of the virtual camera Oc face the other virtual avatars. This allows other users to observe the captured image in the virtual space V.

[0176] Then, in the third embodiment, as the above-described visual presentation is implemented, the processor 10a executes control processing based on the captured images in the virtual space V. Specifically, the processor 10a generates data that extracts images (capture images) from the range in the virtual space V corresponding to the orientation of the virtual camera Oc. Then, the processor 10a controls the actual camera Gc in the real space R to save the captured images in the virtual space V in an area that can be stored by the actual camera Gc. The area that can be stored by the actual camera Gc is the storage area used by the actual camera Gc, which is a digital camera, to save image data. Specifically, it is the internal memory mounted on the camera body or the SD card inserted into the slot of the camera body. The storage destination (save destination) of the captured images in the virtual space V can be specified in advance and can also be appropriately changed according to the user's change operation.

[0177] As described above, in the third embodiment, images captured in the virtual space V can be stored in a designated storage area in the real space R. Therefore, images captured in the virtual space V can be used in the real space R. For example, if an SD card storing images captured in the virtual space V is removed from the actual camera Gc and inserted into the slot of another terminal, the captured images in the virtual space V can be read from the SD card by that terminal and displayed on a display or similar device. Alternatively, as... Figure 12 As shown, it is also possible to read the captured images in the virtual space V from the internal memory built into the actual camera Gc and display them on the rear display Gm of the actual camera Gc.

[0178] In the third embodiment, since the images captured in the virtual space V can be saved and utilized on the real space R side, the real space R and the virtual space V can be effectively combined. Furthermore, the user can view the images captured in the virtual space V even after finishing using it.

[0179] (Example of a structural variation of a virtual space display device)

[0180] In the above embodiment, the virtual space display device 10 is composed of a computer, specifically a computer different from the user terminal 22, but the user terminal 22 may also perform some of the functions of the virtual space display device 10. In this case, the virtual space display device of the present invention is constituted by the linkage between the aforementioned computer and the user terminal 22.

[0181] (Regarding the processor's architecture)

[0182] The virtual space display device of the present invention includes at least one processor, which may include various processors. Among the various processors, for example, there is a general-purpose processor, namely a CPU, that executes software (programs) to perform various processing functions.

[0183] Furthermore, each process in the various embodiments of the present invention can be executed by any computer. Moreover, any computer can execute these processes via a processor, a program, or a combination thereof. Any computer can be a general-purpose computer, a special-purpose computer, a workstation, or other hardware components capable of executing programs.

[0184] A processor can consist of one or more hardware components, and the type of hardware is not limited. For example, a processor can be composed of programmable logic devices such as CPU (Central Processing Unit), MPU (Micro Processing Unit), FPGA (Field Programmable Gate Array), special-purpose circuits such as ASIC (Application Specific Integrated Circuit) for performing specific processing, GPU (Graphics Processing Unit), or NPU (Neural Processing Unit).

[0185] Furthermore, the processor has units or means that execute the various processes described in this invention. The hardware can be a combination of different types of hardware. When multiple pieces of hardware are configured to execute one or more processes of a processor, these multiple pieces of hardware can exist in physically separate devices or in the same device. Furthermore, in any embodiment of this invention, the order of the processor's processes is not limited to the above order and can be appropriately varied. Additionally, the hardware is composed of circuits, such as those combining semiconductor elements.

[0186] Furthermore, the various embodiments of the present invention can be implemented by hardware, software, firmware, microcode, or a combination thereof. Software, firmware, and microcode constitute programs. Furthermore, a program can be, for example, a group of program modules, each of whose functions can be implemented by a processor configured to execute those functions. A program can also be program code and multiple code segments stored in one or more non-transitory computer-readable media (e.g., storage media or other storage devices). A program can be segmented and stored in multiple non-transitory computer-readable media existing in physically separate devices. Program code or code segments can represent any combination of steps, functions, subroutines, routines, subroutines, modules, software packages, categories or commands, data structures, or program statements. Program code or code segments can be connected to other code segments or hardware circuits by sending and receiving information, data, arguments, parameters, or memory contents.

[0187] Symbol Explanation

[0188] 10-Virtual space display device, 10a-Processor, 10b-Memory, 10c-Communication interface, 10d-Storage device, 12-Database, 20-User unit, 22-User terminal, 22a-Processor, 22b-Memory, 22c-Communication interface, 22d-Input device, 22e-Output device, 30-HMD (Handset Media Display), 32-Image sensor, 40-Wearable device, 100-Virtual space display system, Ab-Virtual avatar, Gc-Actual camera (actually existing camera device), Gm-Rear monitor, Oc-Virtual camera (camera device object), Om-Monitor part, P-Printed object, Po-Printed object, R-Real space, V-Virtual space.

Claims

1. A virtual space display device, comprising a processor and displaying a virtual space on a monitor, wherein, The processor performs the following processing: Obtain first information relating to an actual existing camera device; From a database that stores object information related to camera devices that can be displayed in the virtual space and second information related to the camera devices corresponding to the object information, the second information corresponding to the first information is selected; and The display shows a virtual space in which the camera device object, corresponding to the first information, is depicted based on the object information associated with the second information.

2. The virtual space display device according to claim 1, wherein, The processor obtains the first information based on camera device information acquired in a real space where an actual camera device is configured.

3. The virtual space display device according to claim 1, wherein, The first and second information include information related to the image.

4. The virtual space display device according to claim 1, wherein, The processor obtains the first information based on camera device information acquired in a real space where an actual camera device is configured. The camera device information is an image of the actual existing camera device. The first piece of information includes feature quantities of the image from the actual existing camera device. The second information includes feature quantities of the image of the camera device corresponding to the object information.

5. The virtual space display device according to claim 1, wherein, The first and second information include text information.

6. The virtual space display device according to claim 1, wherein, The processor obtains the first information based on camera device information acquired in a real space where an actual camera device is configured. The camera device information is information output from the user terminal based on the user's input operations on the actual existing camera device. The first information includes reference information based on the output information from the user terminal. The second information includes comparison information registered for the camera device corresponding to the object information.

7. The virtual space display device according to claim 1, wherein, In the process of selecting the second information corresponding to the first information, the processor selects the second information from the database that meets the selection criteria in terms of similarity to the first information.

8. The virtual space display device according to claim 1, wherein, The second piece of information includes information about the attributes of the camera device corresponding to the object information. The camera device object is associated with a camera device function corresponding to the aforementioned attribute. When the camera device object is instructed to perform an action corresponding to the function, the processor performs a visual presentation based on the function in the virtual space.

9. The virtual space display device according to claim 8, wherein, The virtual space depicts a virtual avatar of the user that reacts to the user's actions. When the virtual avatar performs an action corresponding to the function on the camera device object, the processor implements a visual presentation based on the function within the virtual space.

10. The virtual space display device according to claim 8, wherein, The first piece of information is information specific to the model of the actual camera device. The second piece of information is information specific to the model of the camera device corresponding to the object information.

11. The virtual space display device according to claim 10, wherein, The number and type of functions associated with the camera device object are determined based on the model of the camera device corresponding to the object information of the camera device object.

12. The virtual space display device according to claim 9, wherein, The camera device is associated with the function of generating a printed image of the captured image. When the virtual image performs a printing action on the camera device object, the processor implements the visual presentation of the printed object generated by the camera device object in the virtual space, which is equivalent to the printed object.

13. The virtual space display device according to claim 12, wherein, When the virtual avatar performs the printing action on the camera device object, the processor performs the following processing: The visual presentation is implemented as follows: the printed object is generated from the camera device object, comprising an extracted image obtained by extracting a range in the virtual space corresponding to the camera device object. The existing camera device, which has the function of generating the printed object, generates the printed object having an image based on the extracted image.

14. The virtual space display device according to claim 13, wherein, The processor causes the actual existing camera device, which has the function of generating the printed object, to output the printed object having a composite image, which is an image obtained by overlaying other images on the extracted image in the printed object.

15. The virtual space display device according to claim 9, wherein, The camera device object is associated with the functions related to camera recording of the camera device corresponding to the object information. When the virtual avatar performs a camera-taking action on the camera device object, the processor implements the visual presentation based on the camera-related functions within the virtual space.

16. The virtual space display device according to claim 15, wherein, When the virtual avatar performs a camera-taking action on the camera device object. The processor performs the following processing: An extracted image is obtained by extracting the area in the virtual space corresponding to the camera device object. The extracted image is stored in the real space in an area that can be stored by the actual existing camera device.

17. A method for displaying virtual space, wherein a virtual space is displayed on a monitor. The virtual space display method includes the following steps: The processor acquires first information related to the actual existing camera device; The processor selects the second information corresponding to the first information from a database that stores object information related to a camera device object that can be displayed in the virtual space and second information related to the camera device corresponding to the object information; and The processor displays a virtual space on the display, in which the camera device object, corresponding to the first information, is depicted based on the object information associated with the second information.

18. The virtual space display method according to claim 17, wherein, In the step of selecting the second information corresponding to the first information, the processor selects the second information from the database that satisfies the selection condition of similarity to the first information.

19. The virtual space display method according to claim 17, wherein, The second piece of information includes information about the attributes of the camera device corresponding to the object information. The camera device object is associated with a camera device function corresponding to the aforementioned attribute. The virtual space display method further includes the following steps: When the camera device object is instructed to perform an action corresponding to the function, the processor performs a visual presentation based on the function in the virtual space.

20. A program for causing a computer to perform the steps included in the virtual space display method according to any one of claims 17 to 19.

21. A recording medium, which is a computer-readable recording medium, wherein, The document contains a program for causing a computer to perform the steps included in the virtual space display method according to any one of claims 17 to 19.

Images