Augmented reality display method and augmented reality display program

Abstract

To provide an augmented reality display method capable of achieving various performances and an augmented reality display program.SOLUTION: An augmented reality display method comprises a feature point acquisition step, a feature point collation step, an augmented reality image display step, and a virtual reality image display step. The feature point acquisition step recognizes a toy image with a camera of a portable terminal and acquires feature points of the toy. The feature point collation step collates the acquired feature points with the feature points stored in a storage device. The augmented reality image display step displays an augmented reality image related to the toy whose feature points match the feature points stored in the storage device in a display part of the portable terminal. The virtual reality image display step displays the virtual reality image related to the toy in the display part when a prescribed operation is performed with the augmented reality image displayed in the display part in the augmented reality image display step.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an augmented reality display method and an augmented reality display program for reading toys such as capsule toys with a camera of a mobile terminal and displaying an augmented reality image or the like on a display unit of the mobile terminal. [Background technology]

[0002] Conventionally, as an augmented reality display method or program using a toy such as a capsule toy, there is known a terminal program controlled by a computer used in the portable terminal device in an augmented reality system having a toy including a figure installation base having a specific mark attached at a predetermined position and a figure body, and a portable terminal device having at least an imaging unit, a display unit, an audio output unit, and a memory unit, which takes an image of the toy with the imaging unit to generate an augmented reality image and displays it on the display unit, the terminal program causing the computer to execute a process of acquiring an image obtained by photographing the toy with the imaging unit, a process of image-recognizing the acquired image of the figure installation base, and a process of superimposing a video of a performance prepared in association with each of the image-recognized figure installation bases on the acquired image or on an image of the figure body stored in the memory unit and displaying it on the display unit (Patent Document 1).

[0003] However, such programs only superimpose images for presentation on the figures, and are unable to display any further presentation or images. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2014-211879 A Summary of the Invention [Problem to be solved by the invention]

[0005] In view of the above-mentioned conventional drawbacks, the present invention aims to provide an augmented reality display method and an augmented reality display program capable of producing a variety of effects. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the augmented reality display method described in claim 1 of the present invention is characterized in that it comprises a feature point acquisition process of image-recognizing a toy with a camera of a mobile terminal and acquiring feature points of the toy, a feature point matching process of comparing the feature points with feature points stored in a storage device, an augmented reality image display process of displaying an augmented reality image related to the toy whose feature points match in the feature point matching process on a display unit of the mobile terminal, and a virtual reality image display process of displaying a virtual reality image related to the toy on the display unit when a predetermined operation is performed with the augmented reality image displayed on the display unit.

[0007] The augmented reality display method according to claim 2 is characterized in that the virtual reality image display step displays an internal structure of the toy.

[0008] The virtual reality image display step of the augmented reality display method described in claim 3 is characterized in that, when a user performs a predetermined operation while the virtual reality image is being displayed, information related to the toy is further displayed.

[0009] The toy in the augmented reality display method according to claim 4 is characterized in that it is composed of a base and a toy body provided on the base.

[0010] The augmented reality display program according to claim 5 is an augmented reality display program for displaying an augmented reality image on a display unit of a mobile terminal, The method includes a process of performing image recognition of a toy using a camera of a mobile terminal, a process of acquiring feature points of the image-recognized toy, a process of comparing the acquired feature points with feature points stored in a storage device, a process of displaying an augmented reality image related to the toy with matching feature points on a display unit of the mobile terminal, and a process of determining whether a specified operation has been performed after the augmented reality image is displayed, and if the specified operation has been performed, a process of displaying a virtual reality image related to the toy on the display unit.

[0011] The virtual reality image of the augmented reality display program according to claim 6 displays an internal structure of the toy.

[0012] When a user performs a predetermined operation while the virtual reality image of the augmented reality display program described in claim 7 is being displayed, processing is performed to further display information related to the toy together with the virtual reality image.

[0013] The toy of the augmented reality display program according to claim 8 is characterized in that it is composed of a base and a toy body provided on the base. Effect of the Invention

[0014] As is clear from the above description, the present invention provides the following effects. (1) In the invention described in claims 1 to 8, by performing a predetermined operation while an augmented reality image is being displayed, a virtual reality image can also be displayed. Therefore, a variety of effects are possible. [Brief description of the drawings]

[0015] 1 to 3 are explanatory diagrams showing a first embodiment of the present invention. [Figure 1] 3 is a process diagram of the augmented reality display method according to the first embodiment. [Diagram 2] FIG. [Diagram 3] 1 is a flow diagram of an augmented reality display program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the embodiments shown in the drawings.

[0017] In a first embodiment for carrying out the present invention shown in FIGS. 1 to 3, reference numeral 1 denotes an augmented reality display method for displaying an augmented reality image on a mobile terminal.

[0018] 1, the augmented reality display method 1 includes a feature point acquisition step 5 for performing image recognition of a toy 3 with a camera 4 of a mobile terminal 2 and acquiring feature points of the toy 3, a feature point matching step 7 for matching the feature points with feature points stored in a storage device 6 of the mobile terminal 2, an augmented reality image display step 9 for displaying an augmented reality image related to the toy 3 whose feature points match in the feature point matching step 7 on a display unit 8 of the mobile terminal 2, and a virtual reality image display step 10 for displaying a virtual reality image related to the toy 3 on the display unit 8 when the augmented reality image displayed on the display unit 8 in the augmented reality image display step 9 or the toy 3 displayed on the display unit 8 via the camera 4 is operated. These steps are executed by an augmented reality display program 15 described later.

[0019] In the present invention, "image" includes not only still images but also videos, and also still images and videos that change when the person operating the mobile terminal 2 taps, etc., and also includes information such as character strings displayed on the display unit.

[0020] In this embodiment, the toy 3 is a so-called capsule toy, and this toy 3 is composed of a base 12 and a toy body 11 attached to the base 12, and this toy body 11 is formed in the shape of a vehicle such as a ship or a car.

[0021] In this embodiment, capsule toys are used as toys, but models of machines, vehicles, animals, insects, buildings, etc. are also included.

[0022] The feature point acquisition process 5 is a process in which the user operates the portable terminal 2 to perform image recognition of the toy 3 with the camera 4 of the portable terminal 2, and acquire the feature points of the toy. Specifically, in this embodiment, the external shape of the toy body 11 is measured by infrared rays, and the external shape of the base 12 is also measured by infrared rays, and the feature points of the toy body 11 and the base 12 are acquired. These feature points are acquired as point cloud data indicating the external shapes. Note that a two-dimensional code or the like containing information such as the feature points of the toy body 11 may be written on the base 12, and this information may be recognized.

[0023] In this embodiment, the toy 3 is made up of the base 12 and the toy body 11, and the feature points of the toy body 11 and the base 12 are obtained and the feature points are matched. However, if the base 12 is not required due to its shape, size, etc., the toy 3 may be made up of only the toy body 11, and only the feature points of the toy body 11 may be obtained and the feature points may be matched.

[0024] In this embodiment, the mobile terminal 2 is assumed to be an electronic device such as a smartphone or tablet equipped with a camera 4, a memory device 6, a display unit 8, a speaker 14, a computing device 16, etc., as shown in Figure 2, but is not limited to these and may be any electronic device that has a camera 4 and a display unit 8 that can display an image recognized by the camera 4.

[0025] In the feature point matching process 7, the feature points of the toy 3 acquired in the feature point acquisition process 5 are matched with the feature point database 13 stored in the storage device 6 of the mobile terminal 2, and an augmented reality image of the toy with matching feature points is extracted.

[0026] This feature point database 13 records point cloud data of the target toy 3, i.e., a group of XYZ coordinates of a large number of feature points of the toy 3, and is also linked to an augmented reality image or a virtual reality image of the toy 3.

[0027] In this embodiment, the feature point database 13, augmented reality images and virtual reality images are stored in the memory device 6 of the mobile terminal 2, but they may also be stored in an external memory device 6 connected to the mobile terminal 2 via the Internet.

[0028] In the augmented reality image display step 9, an augmented reality image related to the toy 3 whose feature points match in the feature point matching step 7 is displayed on the display unit 8 of the mobile terminal 2.

[0029] This augmented reality image is a so-called AR image. For example, if the toy 3 is a model ship, an image of the water surface around the ship is displayed on the display unit 8 of the mobile terminal 2, giving the impression that the ship is sailing.

[0030] In virtual reality image display step 10, when a predetermined operation is performed with an augmented reality image displayed on display unit 8, a virtual reality image related to the toy 3 is displayed on display unit 8. The predetermined operation refers to, for example, touching a predetermined position on display unit 8 of mobile terminal 2, touching or pressing a button, key, etc. of mobile terminal 2. In this embodiment, the virtual reality image is displayed on display unit 8 by touching the image of toy 3 displayed on display unit 8 via camera 4.

[0031] The virtual reality image is a so-called VR image, and for example, if the toy 3 is a model ship, touching the image of the ship will remove the roof of the ship and display the internal structure of the ship. Furthermore, when the roof is removed and the inside of the ship is touched, an interior VR image is displayed as a virtual reality image, and by manipulating this interior VR image, the inside of the ship can be viewed in 360 degrees, and an image is displayed in which the inside of the ship moves in the direction of the touch operation.

[0032] Furthermore, if the user performs a predetermined operation while this virtual reality image is being displayed, further information related to the toy 3 is displayed. For example, if the toy 3 is a model ship, tapping a predetermined position on the ship displays information about the ship's specifications, the designer, and the like.

[0033] Incidentally, when an augmented reality image or a virtual reality image is displayed in the augmented reality image display step 9 or the virtual reality image display step 10, further effects may be provided using dramatic CG such as appearance effects. Furthermore, when an augmented reality image or a virtual reality image is displayed or while an augmented reality image or a virtual reality image is being displayed, music, sound effects, and the like may be played from the speaker 14 of the mobile terminal 2. In this embodiment, the CG, music, and the like are stored in the storage device 6 of the mobile terminal 2, but for example, these data may be stored in an external storage device 6 connected via the Internet.

[0034] The augmented reality display program 15 used in the augmented reality display method 1 of the present invention and installed in the mobile terminal 2 includes a process of performing image recognition of the toy 3 with the camera 4 of the mobile terminal 2, a process of acquiring feature points of the image-recognized toy 3, a process of comparing the acquired feature points with feature points stored in the memory device 6, a process of displaying an augmented reality image related to the toy 3 whose feature points match on the display unit 8 of the mobile terminal 2, and a process of determining whether a specified operation has been performed after the augmented reality image is displayed, and if the specified operation has been performed, displaying a virtual reality image related to the toy 3 on the display unit 8.

[0035] As shown in Fig. 3, the augmented reality display program 15 first recognizes the image of the toy 3 via the camera 4 of the mobile terminal 2, acquires the feature points of the toy 3, and compares them with the feature points of the toy 3 stored in the feature point database 13. The feature points are acquired using a two-dimensional code and a visual positioning system. The visual positioning system scans the toy 3 in advance, refers to the data of the feature points of the toy 3 stored in the feature point database 13, and estimates the position and direction on the mobile terminal 2 based on the image recognized by the camera 4. In this embodiment, the visual positioning system is also used for displaying augmented reality images.

[0036] If a matching toy 3 is found as a result of the comparison, an appearance effect is played and then an augmented reality image is displayed on the display unit 8.

[0037] If the result of the comparison is that there is no matching toy 3, a message to that effect is displayed and the process ends, but the comparison may be performed again as appropriate.

[0038] While an augmented reality image is being displayed on the display unit 8, it is determined whether a specified operation has been performed, and if an operation has been performed, an appearance effect or the like is displayed on the display unit 8, and then a virtual reality image is displayed on the display unit 8.

[0039] Here, a specified operation may be, for example, touching the toy 3 displayed on the display unit 8, pressing or touching a button on the mobile terminal 2, touching a button (soft key) for displaying a virtual reality image or a VR mode displayed on the display unit 8, etc.

[0040] When this virtual reality image is displayed, the internal structure of the toy 3 can be displayed by the user's operation, and if necessary, information about the toy 3 can also be displayed by the user's operation.

[0041] Meanwhile, when a virtual reality image is displayed and a predetermined operation is performed, such as touching a predetermined part of the display unit 8 or pressing or touching a button on the mobile terminal 2, the display of the virtual reality image may be terminated and only the augmented reality image may be displayed.

[0042] When an augmented reality image or a virtual reality image is displayed, a function for taking a screenshot of the image displayed on the display unit 8 or a process for sharing the image on a social networking site or the like may be provided. [Industrial Applicability]

[0043] The present invention can be utilized in industries that use an augmented reality display method and an augmented reality display program. [Explanation of symbols]

[0044] 1: Augmented reality display method; 2: Mobile terminal; 3: Toys, 4: Cameras, 5: feature point acquisition process; 6: storage device; 7: feature point matching process; 8: display unit; 9: augmented reality image display process; 10: virtual reality image display process; 11: Toy body, 12: Base, 13: Feature point database, 14: Speaker, 15: Augmented reality display program; 16: Computing device.

Claims

1. An augmented reality display method comprising: a feature point acquisition step of performing image recognition on a toy with a camera of a mobile terminal and acquiring feature points of the toy; a feature point matching step of comparing the feature points with feature points stored in a storage device; an augmented reality image display step of displaying on a display unit of the mobile terminal an augmented reality image related to the toy whose feature points match in the feature point matching step; and a virtual reality image display step of displaying on the display unit a virtual reality image related to the toy when a specified operation is performed with the augmented reality image displayed on the display unit.

2. 2. The method according to claim 1, wherein the virtual reality image display step displays an internal structure of the toy.

3. The augmented reality display method according to claim 2, characterized in that in the virtual reality image display process, if a user performs a predetermined operation while the virtual reality image is being displayed, further information related to the toy is displayed.

4. 4. The augmented reality display method according to claim 1, wherein the toy is composed of a base and a toy body attached to the base.

5. An augmented reality display program for displaying an augmented reality image and a virtual reality image on a display unit of a mobile terminal, An augmented reality display program including the steps of: performing image recognition of a toy using a camera of a mobile terminal; acquiring feature points of the image-recognized toy; comparing the acquired feature points with feature points stored in a storage device; displaying, on a display unit of the mobile terminal, the augmented reality image related to the toy whose feature points match; and, after displaying the augmented reality image, determining whether a specified operation has been performed, and, if the specified operation has been performed, displaying the virtual reality image related to the toy on the display unit.

6. 6. The augmented reality display program according to claim 5, wherein the virtual reality image displays an internal structure of the toy.

7. The augmented reality display program described in claim 6, characterized in that when a user performs a predetermined operation while the virtual reality image is being displayed, processing is performed to further display information related to the toy together with the virtual reality image.

8. 8. The augmented reality display program according to claim 5, wherein the toy is composed of a base and a toy body attached to the base.

Images