Virtual object display program, virtual object display system, and virtual object display method

Abstract

To provide a virtual object corresponding to a real article that a user can decide even when the virtual object corresponding to the real article cannot be specified as one.SOLUTION: A virtual object display program provides a computer with an acquisition function for acquiring a captured image, a retrieval function for retrieving one or more virtual objects corresponding to an object reflected in the captured image, a display function for, when a plurality of corresponding virtual objects is retrieved, displaying two or more virtual objects among the retrieved virtual objects on a display device, and a lock release function for changing a utilization state of the virtual object selected by a user among the displayed virtual objects from an unavailable state to an available state.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] At least one of the embodiments of the present invention relates to a virtual object display program, a virtual object display system, and a virtual object display method.

Background Art

[0002] Goods such as figures related to games are on sale.

[0003] Patent Document 1 describes an article identification device including an image interface and a processor. The image interface acquires a photographed image. The processor identifies the classification of an article shown in the photographed image based on a classification identification dictionary generated by machine learning. The processor selects an article identification dictionary corresponding to the classification of the article from a plurality of article identification dictionaries respectively corresponding to a plurality of classifications. The processor identifies the article from the photographed image based on the selected article identification dictionary.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When a user has a real article such as a figure and links the real article with a virtual object in a virtual space such as a game, the user's interest is enhanced.

[0006] As a method for linking a real object and a virtual object, for example, a method of imaging a real object with a camera and identifying a virtual object based on the captured image can be considered. However, real objects are diverse, and some objects may be similar to each other. Therefore, it may be difficult to narrow down and identify the corresponding virtual object to one.

[0007] An object of at least one embodiment of the present invention is to solve the above problems and provide a virtual object display program, a virtual object display system, and a virtual object display method that enable a user to determine the corresponding virtual object even when it is not possible to identify the virtual object corresponding to a real object as one.

Means for Solving the Problems

[0008] From a non-limiting perspective, a virtual object display program according to an embodiment of the present invention causes a computer to have an acquisition function for acquiring a captured image, a search function for searching for one or more virtual objects corresponding to an object shown in the captured image, a display function for causing a display device to display two or more of the searched virtual objects when a plurality of virtual objects are searched, and an unlocking function for changing the usage state of the virtual object selected by the user among the displayed virtual objects from an unusable state to an available state.

[0009] From a non-limiting perspective, a virtual object display system according to an embodiment of the present invention is a virtual object display system including a communication network, a server, and a user terminal, and includes an acquisition means for acquiring a captured image, a search means for searching for one or more virtual objects corresponding to an object shown in the captured image, a display means for causing a display device to display two or more of the searched virtual objects when a plurality of virtual objects are searched, and an unlocking means for changing the usage state of the virtual object selected by the user among the displayed virtual objects from an unusable state to an available state.

[0010] From a non - limiting perspective, a virtual object display method according to an embodiment of the present invention is a method for displaying virtual objects by a computer, including an acquisition process of acquiring a captured image, a search process of searching for one or more virtual objects corresponding to an object reflected in the captured image, a display process of causing two or more of the searched virtual objects to be displayed on a display device when a plurality of virtual objects are searched, and an unlocking process of changing the usage state of a virtual object selected by a user from an unusable state to a usable state among the displayed virtual objects.

Advantages of the Invention

[0011] One or two or more deficiencies are solved by each embodiment of the present application.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0013] Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. Note that various components in the examples of the embodiments described below can be appropriately combined as long as there is no contradiction. In addition, the description of the content described as an example of a certain embodiment may be omitted in other embodiments. Also, operations and processes not related to the characteristic parts of each embodiment may be omitted. Furthermore, the order of various processes constituting the various flows and sequences described below can be changed as long as there is no contradiction in the processing content.

[0014] Hereinafter, a virtual object display program executed on a server, which is an example of a computer, will be illustrated and described. However, the computer may be another device such as a user terminal.

[0015] FIG. 1 is a block diagram showing an example of the configuration of a virtual object display system 100 corresponding to at least one of the embodiments of the present invention. The virtual object display system 100 includes a server 10 and a user terminal 20 used by a user of the virtual object display system 100. User terminals 20A, 20B, and 20C are each an example of the user terminal 20. The configuration of the virtual object display system 100 is not limited to this. For example, the virtual object display system 100 may be configured such that a single user terminal is used by a plurality of users. The virtual object display system 100 may include a plurality of servers.

[0016] The server 10 and the user terminal 20 are examples of computers. The server 10 and the user terminal 20 are each communicably connected to a communication network 30 such as the Internet. The connection between the communication network 30 and the server 10 and the connection between the communication network 30 and the user terminal 20 may be a wired connection or a wireless connection. For example, the user terminal 20 may be connected to the communication network 30 by performing data communication with a base station managed by a communication carrier via a wireless communication line.

[0017] The virtual object display system 100 includes a server 10 and a user terminal 20, and realizes various functions for executing various processes according to user operations.

[0018] The server 10 may be a game processing server that controls the progress of a game. In this case, the server 10 controls the progress of the game according to the operations of the players. The server 10 may provide various electronic services to the user, not limited to games.

[0019] The server 10 includes a processor 11, a memory 12, and a storage device 13. The processor 11 is a central processing unit such as a CPU (Central Processing Unit) that performs various operations and controls. Also, when the server 10 includes a GPU (Graphics Processing Unit), some of the various operations and controls may be performed by the GPU. The server 10 uses the data read into the memory 12 to execute various information processes by the processor 11, and stores the obtained processing results in the storage device 13 as necessary.

[0020] The storage device 13 has a function as a storage medium for storing various information. The configuration of the storage device 13 is not particularly limited, but from the viewpoint of reducing the processing load on the user terminal 20, it may be configured to be able to store all the various information necessary for the control performed in the virtual object display system 100. Examples of such include HDDs and SSDs. However, the storage device for storing various information only needs to have a storage area in a state accessible by the server 10, and for example, it may be configured to have a dedicated storage area outside the server 10.

[0021] The server 10 may be configured by an information processing device such as a game processing server capable of rendering game images.

[0022] The user terminal 20 is managed by the user. Examples of the user terminal 20 include, for example, a mobile phone terminal, a smartphone, a PDA (Personal Digital Assistant), a personal computer, a tablet, a stationary game device, a portable game device, and the like. The user terminal 20 may be a VR goggle, AR glasses, smart glasses, AR contact lenses, or other wearable device that can be operated by the user.

[0023] The user terminal 20 may be a communication terminal capable of playing a network-distributed game.

[0024] The user terminal 20 is connected to the communication network 30 and includes hardware and software for executing various processes by communicating with the server 10. Each of the plurality of user terminals 20 may be configured to be able to directly communicate with each other without going through the server 10.

[0025] The user terminal 20 may have a display device built therein. Alternatively, a display device may be wirelessly or wiredly connected to the user terminal 20. Since the display device has a very common configuration, its illustration is omitted here. The display device is an example of an output device. When the user plays a game using the user terminal 20, the game screen is displayed by the display device as a composite image, and the user recognizes this composite image. Also, when the user performs various information processes using the user terminal 20, the screen (information processing screen) necessary for the information process is displayed by the display device as an image, and the user recognizes this image. The game screen and the information processing screen are displayed, for example, on a display which is an example of the display device included in the user terminal, or on a display which is an example of the display device connected to the user terminal. The display device includes, for example, a hologram display device capable of hologram display, a projection device that projects an image (including the game screen and the information processing screen) onto a screen or the like, a device that performs XR display, and the like. XR includes VR (Virtual Reality), AR (Augmented Reality), MR (Mixed Reality), and SR (Substitutional Reality), and the like.

[0026] The user terminal 20 includes a processor 21, a memory 22, and a storage device 23. The processor 21 is, for example, a central processing unit such as a CPU (Central Processing Unit) that performs various operations and controls. Also, when the user terminal 20 includes a GPU (Graphics Processing Unit), a part of the various operations and controls may be performed by the GPU. The user terminal 20 executes various information processes by the processor 21 using the data read out to the memory 22, and stores the obtained processing results in the storage device 23 as necessary. The storage device 23 has a function as a storage medium for storing various information.

[0027] The user terminal 20 may have an input device built therein. Alternatively, an input device may be wirelessly or wiredly connected to the user terminal 20. The input device receives operation inputs from the user. In response to the operation inputs from the user, a processor provided in the server 10 or a processor provided in the user terminal 20 executes various control processes. Examples of the input device include a touch panel screen provided in a smartphone, a tablet, etc., a keyboard, a mouse, a game pad, a joystick, and other controllers. Further, a camera built in or connected to the user terminal 20 may also correspond to the input device. The user performs operation inputs by gestures such as moving a hand in front of the camera (gesture input). The user terminal 20 may include a microphone. The user can perform voice input by speaking words in natural language toward the microphone.

[0028] In addition, the user terminal 20 may include other output devices such as a speaker. The other output devices output voice, vibration, and other various types of information to the user.

[0029] FIG. 2 is a block diagram showing the configuration of a server corresponding to at least one of the embodiments of the present invention. The server 10 includes an acquisition unit 101, a search unit 102, a display unit 103, a lock control unit 104, and a display control unit 105. The processor included in the server 10 functionally realizes the acquisition unit 101, the search unit 102, the display unit 103, the lock control unit 104, and the display control unit 105 by referring to a virtual object display program held in a storage device and executing the program.

[0030] The acquisition unit 101 has a function of acquiring a captured image.

[0031] The search unit 102 has a function of searching for one or more virtual objects corresponding to an object reflected in the captured image.

[0032] The display unit 103 has a function of causing a display device to display two or more virtual objects out of the searched virtual objects when a plurality of virtual objects are searched. Note that the display unit 103 may AR-display a virtual object on the display device.

[0033] The lock control unit 104 has a function of changing the usage state of the virtual object selected by the user among the displayed virtual objects from an unusable state to an available state.

[0034] The display control unit 105 has a function of controlling the virtual object displayed on the display device in response to a natural language input by the user.

[0035] The captured image acquired by the acquisition unit 101 is typically an image captured by a camera provided in the user terminal 20. However, the captured image may be an image captured by a device other than the user terminal 20. The captured image may be, for example, a captured image captured by a camera installed in a store visited by the user. The user visits the store with a real object such as a figure. Or, the user purchases or rents a real object at the store. The object presented by the user is captured by a camera installed in the store, and a captured image is generated. The generated captured image is acquired by the acquisition unit 101 of the server 10 via the communication network 30.

[0036] The object reflected in the captured image is a real object such as a figure reflected in the captured image.

[0037] The virtual object is a virtual object that can be displayed in a virtual space. Typically, the virtual object is registered as 3D model information, but it may be 2D model information or the like. The virtual object is pre-registered in a storage device or the like accessible from the server 10. The virtual object is registered in the storage device in the form of a database, for example.

[0038] The search unit 102 may search for a predetermined number of virtual objects that have a high degree of similarity to the objects reflected in the captured image. The predetermined number mentioned here may be determined in advance, such as three, for example. For instance, if the specification is to present a maximum of three virtual objects to the user, the predetermined number is three. The search unit 102 performs matching between the objects reflected in the captured image and the virtual objects registered in a database or the like in advance, and searches (extracts) a predetermined number of corresponding virtual objects in descending order of the similarity between the two. Regarding the specific algorithm of the matching process, a conventional matching algorithm may be used, so detailed description is omitted.

[0039] When a plurality of virtual objects are searched, the display unit 103 causes the display device to display two or more of the searched virtual objects. For example, when five virtual objects are searched, three or five of them are caused to be displayed on the display device.

[0040] For example, assume that it is predetermined in advance that the maximum number of virtual objects presented as recommendations to the user is three. In this case, the display unit 103 causes the display device to display up to three virtual objects that are similar to the objects reflected in the captured image. As a result, the user can select the virtual object that the user truly intends among the virtual objects corresponding to the object presented by the user.

[0041] In addition, the display unit 103 may cause the display device to display a virtual object whose appearance has been changed according to the features detected from the objects reflected in the captured image.

[0042] The display unit 103 detects the features of the object reflected in the captured image. The features of the object include any one or more of the shape of the object, the pose of the object, the accessories attached to the object, the color of the object, and the pattern of the object. Based on the detected features, the display unit 103 changes the appearance of the retrieved virtual object. For example, the display unit 103 reads out the 3D model information of the virtual object and changes the shape of the virtual object, the pose of the virtual object, the accessories attached to the virtual object, the color of the virtual object, the pattern of the virtual object, etc. based on the detected features. The display unit 103 causes the display device to display the changed virtual object.

[0043] The lock control unit 104 changes the usage state of the virtual object selected by the user among the displayed virtual objects from the unusable state to the usable state. Information indicating the usage state of the virtual object is stored in a storage device accessible from the server 10, etc., in association with the virtual object, for example, in the form of a flag. The lock control unit 104 refers to the stored information indicating the usage state and changes and registers the value indicating the unusable state to the value indicating the usable state. The display control unit 105 described later may control the virtual object that has become available.

[0044] The display control unit 105 controls the virtual object displayed on the display device according to the natural language input by the user. When the user input is text input, the user operates the user terminal 20 or the like and inputs an instruction as text in the input box. The instruction is input in natural language, for example, like "Place the character on the table in front of my eyes". Note that this natural language input may be input by manually operating the user terminal 20, or may be the instruction uttered by the user collected by the microphone provided in the user terminal 20. Note that the virtual object display system 100 may convert the collected voice information into text information.

[0045] The display control unit 105 analyzes the instructions in natural language input by the user, understands the content thereof, and controls the virtual object according to the content of the instructions. As an algorithm for natural language understanding, for example, pattern matching is known. That is, matching is performed based on whether a predetermined phrase or an expression similar to the phrase is included in the text obtained by natural language input, and the matched one is determined as the instruction content. Note that, as an algorithm for natural language understanding, a method other than the above pattern matching may be used.

[0046] Based on the instructions in natural language input by the user, the display control unit 105 performs control such as the following, for example. · Change the shape of the virtual object. · Change the pose of the virtual object. · Equip the virtual object with items such as a sword, a shield, and accessories. · Remove the items that the virtual object is already equipped with. · Change the color of at least a part of the virtual object. · Change the pattern of at least a part of the virtual object. · Change the size of the virtual object. · Change the position and / or orientation of the virtual object in the virtual space. · Increase or decrease the number of virtual objects.

[0047] Note that the instructions in the form of natural language may be issued by a subject other than the user. For example, the display control unit 105 may acquire text information in the form of natural language generated by the meta AI that controls the game, and control the virtual object based on the instructions.

[0048] FIG. 3 is a flowchart showing an example of processing of a virtual object display program corresponding to at least one of the embodiments of the present invention.

[0049] For example, the camera included in the user terminal 20 captures an image of a real object (St101). The object may be, for example, a real figure, a mini car, or the like, but is not limited thereto.

[0050] The user terminal 20 transmits the captured image to the server 10. The acquisition unit 101 of the server 10 acquires the captured image (St102).

[0051] The search unit 102 searches for one or more virtual objects corresponding to the object captured in the captured image (St103).

[0052] If the number of the searched virtual objects is 0, the user is notified and the process returns to step St101.

[0053] If the number of the searched virtual objects is 1, the process proceeds to step St105. If the number of the searched virtual objects is two or more, the process proceeds to step St104.

[0054] In step St104, the display unit 103 causes the display device to display two or more of the searched virtual objects.

[0055] Here, refer to FIG. 4 together. FIG. 4 is a conceptual diagram showing a display example of displaying two or more virtual objects and making recommendations corresponding to at least one of the embodiments of the present invention.

[0056] In the present embodiment, it is assumed that the user terminal 20 is a smartphone for explanation. Two virtual objects similar to the object captured by the camera of the user terminal 20 are searched. In step St104, the display unit 103 causes the display device to display the two searched virtual objects, thereby recommending the two virtual objects to the user.

[0057] The user selects the virtual object that the user desires from the two displayed virtual objects. This selection may be made, for example, by tapping the desired virtual object on the touch panel provided in the user terminal 20.

[0058] Return to the description of FIG. 3. In step St105, the lock control unit 104 changes the usage state of the virtual object selected by the user among the displayed virtual objects from the unusable state to the usable state. When the number of virtual objects retrieved in step St103 is 1, there is no need for the user to select the desired one from among the plurality of virtual objects. Therefore, in this case, the lock control unit 104 changes the usage state of the retrieved one virtual object from the unusable state to the usable state.

[0059] The user operates the user terminal 20 to input an instruction in the form of natural language. The input natural language-form instruction information is transmitted via the communication network 30, and the display control unit 105 of the server 10 acquires the natural language-form instruction information (St106).

[0060] The display control unit 105 controls the virtual objects displayed on the display device in response to the natural language input by the user (St107).

[0061] FIG. 5 is a conceptual diagram showing an example of controlling virtual objects in response to a user's natural language input corresponding to at least one of the embodiments of the present invention. FIG. 6 is a conceptual diagram showing an example of controlling virtual objects in response to a user's natural language input corresponding to at least one of the embodiments of the present invention.

[0062] The display unit 103 causes the selected virtual object OBJ to be displayed on the display device. In the example of FIG. 5, the display is an AR display. That is, the virtual object OBJ is displayed so as to be superimposed near the base of the Tokyo Tower reflected on the touch panel screen of the user terminal 20.

[0063] The user operates the user terminal 20 to input an instruction in the form of natural language. For example, the user inputs a natural sentence such as "Move the character to the observation deck of the Tokyo Tower" into the user terminal 20. In step St107, the display control unit 105 controls the virtual object OBJ displayed on the display device according to the natural language input by the user. In this example, as shown in FIG. 6, the virtual object OBJ is moved to the location of the observation deck of the Tokyo Tower.

[0064] Note that the Tokyo Tower is reflected in the captured image being taken in real time by the camera of the user terminal 20. The user terminal 20 or the display control unit 105 detects the Tokyo Tower by general image recognition and estimates the position of the observation deck in a predetermined coordinate space. Then, the display control unit 105 performs AR display of the virtual object OBJ so as to overlap with the position of the observation deck.

[0065] As described above, one or more deficiencies are solved by each embodiment of the present application. Note that the effects according to each embodiment are non-limiting effects or examples of effects.

[0066] In each of the above-described embodiments, the user terminal 20 and the server 10 execute the above-described various processes according to various control programs (for example, virtual object display programs) stored in the storage devices provided by themselves. Also, other computers not limited to the user terminal 20 or the server 10 may execute the above-described various processes according to various control programs (for example, virtual object display programs) stored in the storage devices provided by themselves.

[0067] Furthermore, the configuration of the virtual object display system 100 is not limited to the configuration described as an example of the above-described embodiments. For example, a configuration may be adopted in which a part or all of the processes described as processes executed by the user terminal are executed by the server, or a configuration may be adopted in which a part or all of the processes described as processes executed by the server are executed by the user terminal. Also, a configuration may be adopted in which a part or all of the storage unit (storage device) provided in the server is provided in the user terminal. That is, in the virtual object display system 100, a part or all of the functions provided in either the user terminal or the server may be provided in the other.

[0068] Further, the program may be configured to implement a part or all of the functions described as examples of the above-described embodiments in a single device that does not include a communication network.

[0069] [Appendix] The description of the above-described embodiments has been made so that at least the following invention can be implemented by a person having ordinary knowledge in the field to which the invention pertains.

[0070] [1] On a computer, an acquisition function for acquiring a captured image, a search function for searching for one or more virtual objects corresponding to an object reflected in the captured image, a display function for causing a display device to display two or more of the searched virtual objects when a plurality of corresponding virtual objects are searched, an unlocking function for changing the usage state of a virtual object selected by the user from an unusable state to an available state among the displayed virtual objects, A virtual object display program for realizing the above.

[0071] According to the above virtual object display program, even when it is not possible to specify a single virtual object corresponding to a real article, the user can determine the corresponding virtual object.

[0072] [2] The search function is a function of searching for a predetermined number of virtual objects with a high similarity to the object. The virtual object display program described in [1].

[0073] According to the above virtual object display program, the user can preferentially identify the corresponding virtual object from those similar to the object presented by the user.

[0074] [3] The display function causes the display device to display the virtual object whose appearance has been changed according to the features detected from the object reflected in the captured image. The virtual object display program described in [1].

[0075] [4] The features are any one or more of the shape of the object, the pose of the object, the accessories attached to the object, the color of the object, and the pattern of the object. The virtual object display program described in [3].

[0076] According to the above virtual object display program, even when the user changes the pose, color, pattern, equipment, etc. of the object, the change can also be reflected in the virtual object.

[0077] [5] On the computer, A display control function for controlling the virtual object displayed on the display device according to the natural language input by the user is Further realized by the virtual object display program described in [1].

[0078] According to the above virtual object display program, the user can control the virtual object only by instructing in natural language.

[0079] [6] A virtual object display system including a communication network, a server, and a user terminal, wherein: An acquisition means for acquiring a captured image; A search means for searching for one or more virtual objects corresponding to an object reflected in the captured image; A display means for causing a display device to display two or more of the searched virtual objects when a plurality of corresponding virtual objects are searched; An unlocking means for changing the usage state of a virtual object selected by a user from an unusable state to a usable state among the displayed virtual objects; Including: A virtual object display system.

[0080] According to the virtual object display system described above, even when it is not possible to specify a single virtual object corresponding to a real article, the user can determine the corresponding virtual object.

[0081] [7] A method for displaying virtual objects by a computer, including: An acquisition process for acquiring a captured image; A search process for searching for one or more virtual objects corresponding to an object reflected in the captured image; A display process for causing a display device to display two or more of the searched virtual objects when a plurality of corresponding virtual objects are searched; An unlocking process for changing the usage state of a virtual object selected by a user from an unusable state to a usable state among the displayed virtual objects; Including: A method for displaying virtual objects.

[0082] According to the virtual object display method described above, even when it is not possible to specify a single virtual object corresponding to a real article, the user can determine the corresponding virtual object.

Industrial Applicability

[0083] According to one embodiment of the present invention, even when it is not possible to specify a single virtual object corresponding to a real-world article, it is useful as a virtual object display program, a virtual object display system, and a virtual object display method that allow a user to determine the corresponding virtual object.

Explanation of Signs

[0084] 10 Server 11 Processor 12 Memory 13 Storage Device 20, 20A, 20B User Terminal 21 Processor 22 Memory 23 Storage Device 30 Communication Network 100 Virtual Object Display System 101 Acquisition Unit 102 Search Unit 103 Display Unit 104 Lock Control Unit 105 Display Control Unit

Claims

1. A virtual object display program that causes a computer to have an acquisition function for acquiring a captured image, a search function for searching for one or more virtual objects corresponding to an object reflected in the captured image, a display function for causing a display device to display two or more of the searched virtual objects when a plurality of corresponding virtual objects are searched, and an unlocking function for changing the usage state of a virtual object selected by a user from an unusable state to a usable state among the displayed virtual objects.

2. The virtual object display program according to claim 1, wherein the search function realizes a function of searching for a predetermined number of virtual objects having a high degree of similarity to the object.

3. The virtual object display program according to claim 1, wherein the display function causes the display device to display the virtual object whose appearance has been changed according to a feature detected from an object reflected in the captured image.

4. The virtual object display program according to claim 3, wherein the feature is any one or more of the shape of the object, the pose of the object, a wearable attached to the object, the color of the object, and the pattern of the object.

5. The virtual object display program according to claim 1, which further causes the computer to realize a display control function for controlling the virtual object displayed on the display device according to a natural language input by a user.

6. A virtual object display system including a communication network, a server, and a user terminal, acquisition means for acquiring a captured image, search means for searching for one or more virtual objects corresponding to an object reflected in the captured image, display means for causing a display device to display two or more of the searched virtual objects when a plurality of corresponding virtual objects are searched, and unlocking means for changing the usage state of a virtual object selected by a user from an unusable state to a usable state among the displayed virtual objects.

7. A method for displaying a virtual object by a computer, the method including: an acquisition process for acquiring a captured image, a search process for searching for one or more virtual objects corresponding to an object reflected in the captured image, When a plurality of corresponding virtual objects are retrieved, a display process for causing a display device to display two or more of the retrieved virtual objects; An unlocking process for changing the usage state of the virtual object selected by the user among the displayed virtual objects from an unusable state to a usable state; A virtual object display method including the above.

Images