Method, information processing system, and object recognition method

Abstract

A method for identifying an AR marker and generating an AR prompt effect and an AR final effect includes the steps of: detecting whether the shape of the AR marker matches a database; detecting whether the image of the AR marker matches a database; identifying the position of the AR marker located in a corresponding frame of an AR UI; and generating the AR prompt effect.

Description

Method, information processing system, and object recognition method

[0001] The present invention relates to a method, an information processing system, and an object recognition method.

[0002] In a conventional augmented reality (AR) system, a technique is known in which a plurality of image markers (AR markers) are simultaneously recognized and AR content corresponding to the markers is displayed (see Patent Document 1).

[0003] However, with this technology, it is difficult to accommodate productions that give meaning to the order and placement of markers, or interactive developments that respond to user operations.

[0004] Patent No. 7606061

[0005] The objective is to provide a highly scalable AR experience using AR markers.

[0006] [1] A method according to the present disclosure is a method for identifying an AR marker and generating an AR prompt effect and an AR final effect, comprising the steps of: detecting whether a shape of the AR marker matches a database; detecting whether an image of the AR marker matches a database; identifying a position of the AR marker located within a corresponding frame of an AR UI; and generating the AR prompt effect.

[0007] [2] In the method according to [1], the step of detecting the shape, the step of detecting the image, and the step of specifying the position are performed in this order.

[0008] [3] In the method of the present disclosure, in [1], the shape determination step, the image determination step, and the position determination step are executed for a plurality of AR markers displayed in the AR space, and the AR prompt effect generation step generates an AR prompt effect corresponding to the plurality of AR markers when the determinations in the shape determination step, the image determination step, and the position determination step are all positive.

[0009] [4] In the method according to the present disclosure, in any one of [1] to [3], if at least one of the determinations in the shape determination step, the image determination step, and the position determination step is negative, the processing is terminated.

[0010] [5] An information processing system according to the present disclosure is an information processing system for determining an AR (Augmented Reality) marker, comprising: a shape determination means for acquiring first shape information of the AR marker displayed in the AR space and determining whether it matches second shape information; an image determination means for acquiring first image information of the AR marker and determining whether it matches second image information; a position determination means for acquiring first position information of the AR marker and determining whether it matches second position information; and an effect generation step for generating an AR prompt effect when the determinations of the shape determination means, the image determination means, and the position determination means are all positive.

[0011] [6] In the information processing system according to the present disclosure, in [5], the AR prompt effect is generated when the AR marker matches a virtual card slot displayed in the AR space.

[0012] [7] The information processing system according to the present disclosure is the information processing system according to [5], wherein the shape determination means is executed with priority over the image determination means and the position determination means.

[0013] In [8] and [7], the information processing system according to the present disclosure first executes a shape determination means, then executes an image determination means (e.g., analyzing the content of an image or recognizing a pattern), and finally executes a position determination means (e.g., determining coordinates or spatial position).

[0014] [9] The object recognition method according to the present disclosure is a method for recognizing an object in an AR (Augmented Reality) space, and includes: a determination step of determining whether or not a shape, an image, and a position of at least one object in real space match one of a plurality of indicators; a termination step of terminating the determination process when it is determined that at least one of the shape, the image, and the position does not match; and a generation step of generating a determination completion AR prompt effect when it is determined that the shape, the image, and the position all match.

[0015]

[10] In the object recognition method according to the present disclosure, in [9], at least one of the plurality of indicators includes coordinate information, and the determination is performed preferentially for an indicator whose relative position between the at least one object in real space and any of the plurality of indicators is closest.

[0016] FIG. 1 is a diagram for explaining a conventional technology. FIG. 2 is a diagram for explaining an example of an AR image of the information processing system according to the present embodiment. FIG. 3 is a diagram for explaining an example of an AR image of the information processing system according to the present embodiment. FIG. 4 is a diagram for explaining an example of an application of the information processing system according to the present embodiment. FIG. 5 is a conceptual diagram for explaining a hardware configuration of a user terminal according to the present embodiment. FIG. 6 is a diagram for explaining an example of an application of the information processing system according to the present embodiment. FIG. 7 is a diagram for explaining an example of an application of the information processing system according to the present embodiment. FIG. 8 is a conceptual diagram for explaining a hardware configuration of a user terminal according to the present embodiment.

[0017] 2 is a conceptual diagram showing an image captured by the user terminal 1 of this embodiment displayed on the display unit (display) of the user terminal 1. As shown in FIG. 2( a), the processor of the user terminal 1 detects an AR marker in real space from the image captured by the imaging unit. The AR marker is, for example, a "trigger card" that triggers the display of a graphic interface described below. For example, when a user places a "trigger card" on a desk or the like and captures an image of the "trigger card" with the imaging unit of the user terminal 1, the processor detects the AR marker (trigger card) in real space from the captured image.

[0018] When the processor of the user terminal 1 detects an AR marker (trigger card) in real space, this triggers the display of a magic matrix (a graphic interface that is an AR object) on the captured image (FIG. 2(b)). An image in which graphics are displayed on the captured image in this way is called an AR image (augmented reality image or AR (Augmented Reality) space).

[0019] This magic matrix includes, for example, six boxes (two circular, two triangular, and two heart-shaped objects; AR card slots) displayed around the AR marker (trigger card). The processor may display on the display unit of the user terminal 1 a message prompting the user to place items in the boxes. Here, the items are, for example, AR cards in real space.

[0020] 3A, the user of the user terminal 1 places an AR card in real space (e.g., triangle "1" and triangle "2") within the field of view (imaging range) of the imaging unit of the user terminal 1. The processor detects the AR card that has entered the field of view.

[0021] 3B, when the AR card on the captured image is superimposed on the box corresponding to the AR card, the processor displays an AR effect (e.g., a yellow flash) on the display unit. After the AR effect is displayed, the processor may display on the display unit of the user terminal 1 a message prompting the user to place an item in another box on the AR image.

[0022] 4(a), the user of the user terminal 1 places an AR card in real space (for example, a circle "3", a circle "4", a heart-shaped "5", or a heart-shaped "6") within the field of view (imaging range) of the imaging unit of the user terminal 1. The processor detects the AR card that has entered the field of view.

[0023] 4B, when the AR card in the captured image is placed superimposed on the box corresponding to the AR card, the processor displays an AR effect on the display unit. The AR effect may be, for example, an effect in which beams are emitted from the six boxes on the AR image toward the central trigger card, along with an effect in which a predetermined object (e.g., the number (text) "231") is superimposed on the central trigger card. Here, "231" is calculated, for example, according to Equation (1), with the numbers "1" to "6" corresponding to each box as variables.

[0024] (1+2)×(3+4)×(5+6)=231 Equation (1) Figure 5 is a diagram illustrating a method for determining whether or not to generate an AR effect (AR result) in the user terminal 1 according to this embodiment.

[0025] The processor of the user terminal 1 in this embodiment determines whether to generate an AR effect based on at least (1) the relative position (marker coordinates) between the marker and the indicator (box) and (2) the shape of the marker (square, triangle, circle, heart, hexagon, etc.).

[0026] First, the processor determines whether the shape (first shape information) of the AR marker (AR card) in the captured image matches the shape (second shape information) stored in the database (step S1). If the determination is affirmative, the process proceeds to step S2. On the other hand, if the shape of the AR marker does not match the shape stored in the database (if the AR card does not match the shape of the frame (box) even when placed in the frame on the AR image), the processor stops the calculation (step S5). This allows for saving computational resources when determining whether an AR effect is generated.

[0027] Next, if the determination in step S1 is affirmative, the processor determines whether the marker image (first image information) matches an image (pattern, color, etc.) in the database (second image information) (step S2). If the determination is affirmative, the process proceeds to step S3. On the other hand, if the image of the AR marker does not match the image stored in the database, the processor stops calculation (step S5). This allows for saving computational resources in determining whether an AR effect is to be generated.

[0028] Next, if the determination in step S2 is affirmative, the processor determines whether the relative position (first position information) between the AR marker and the indicator (box) placed on the AR image is within a predetermined range (whether it matches the second position information) (step S3). If the determination is affirmative, the process proceeds to step S4. On the other hand, if the determination is negative (for example, if the AR card is placed beyond the allowable range of the frame (box) (if it protrudes too far from the frame)), the processor stops the calculation (step S5). This allows for saving computational resources when determining whether an AR effect is to be generated.

[0029] If the determination in step S3 is affirmative, the processor causes the display unit of the user terminal 1 to display the AR result (AR effect) corresponding to the AR marker (and its arrangement).

[0030] Thus, instead of generating the three step AR effects of steps S1 to S3, the processor of the user terminal 1 according to this embodiment may generate a predefined AR prompt effect, and the system then executes the later stages of the processing according to the predefined prompt effect.

[0031] The system performs marker detection, determines whether the input data qualifies as detection data according to a hierarchical decision logic that prioritizes shape recognition, then performs image recognition, and finally evaluates relative position.

[0032] The information processing system according to this embodiment can continuously generate multiple changes according to the determination process of the AR marker, and can therefore be integrated into product functions, marketing themes, special events, storylines, and can provide a variety of combinations by combining AR cards, transportation tokens, tourist souvenirs, stylish chess pieces, character figurines, etc. For example, the information processing system according to this embodiment can be used in museums, schools, exhibitions, children's picture books, escape rooms, etc.

[0033] Below is an example.

[0034] (A) Example of application to a picture book: When the camera of the information processing system detects a page of a picture book, it may generate a corresponding 3D virtual space (AR space) of a forest scene and display necessary landmarks in the forest (such as Little Red Riding Hood's house, the location of the Big Bad Wolf, and the sacred tree) on the device screen. Three landmarks (frames) need to be placed at three specific AR markers. When marker A is placed at the first landmark, the corresponding AR effect is displayed. Next, when marker B is placed at the second landmark, the wolf is activated and starts moving toward Little Red Riding Hood's house. The user needs to use marker C to stop the wolf and knock down the sacred tree to block the path to Little Red Riding Hood's house.

[0035] (B) Example of application to tourism activities: When a virtual map of a tourist spot is displayed, a user may place the tourist commemorative coins they have collected in the frame of the map to generate an AR effect corresponding to the map.

[0036] (C) Example of application to escape rooms: This technology can make the puzzle-solving process more interactive and improve the stability and smoothness of the game process.

[0037] 9 is a diagram illustrating the hardware configuration of the user terminal 1 (information processing system) according to this embodiment. The information processing system may be configured from the user terminal 1 and a server.

[0038] In the user terminal 1, the CPU 101 is a processing device that controls the overall operation of the user terminal 1. The ROM 102 is a non-volatile memory that stores control programs and various data executed by the CPU 101. The RAM 103 is a volatile memory used as a load area and work area for programs executed by the CPU 101. The storage device 104 is a storage means for storing various information (characteristic information of the AR object (shape, image (pattern, color, etc.)), etc.), and may be built into the user terminal 1 main body or may have a removable storage medium. The input device 105 is a device for a user of the user terminal 1 to input information, and is, for example, a touch panel or a microphone. The display 106 is a display device that displays various information (user interface, etc.). The image sensor 107 is a photoelectric conversion element that captures an image of a subject. The communication I / F (interface) 108 is an interface for connecting to a server. The bus 109 is a bus line that connects the above-mentioned components to each other. Note that the information stored in the storage device 104 does not necessarily have to be provided within the storage device 104; some or all of the information may be provided in another device that is communicably connected to the user terminal 1 via the network 4.

[0039] Any part or all of the functional units described in this specification may be realized by a program. The program mentioned in this specification may be non-transitoryly recorded on a computer-readable recording medium.

[0040] Such a program may be installed on a computer (a so-called native application), in which case the program may be downloaded to the computer via a communication line (including wireless communication) such as the Internet, or may be distributed in a state where it is installed on the computer.

[0041] Based on the above description, a person skilled in the art may be able to conceive additional effects and various modifications of the present invention, but the aspects of the present invention are not limited to the individual embodiments described above. For example, inventions that extract only a part of each embodiment or inventions that combine multiple embodiments are naturally envisioned. Various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and spirit of the present invention, which can be derived from the content defined in the claims and their equivalents.

[0042] For example, what is described in this specification as one device (or component, the same applies hereinafter) (including what is depicted as one device in the drawings) may be realized by multiple devices. Conversely, what is described in this specification as multiple devices (including what is depicted as multiple devices in the drawings) may be realized by one device. Alternatively, some or all of the means or functions included in one device may be included in another device. Furthermore, a "system" may be composed of one device, or two or more devices.

[0043] Furthermore, not all of the features described in this specification are essential requirements. In particular, features described in this specification but not included in the claims can be considered optional additional features.

[0044] The problem that the present invention aims to solve should be identified by considering the entire specification. For example, if the specification states that a specific effect is achieved by a specific configuration, it can also be said that the invention solves a problem that is the reverse of the specific effect. However, it is not intended that such a specific configuration is necessarily a required requirement.

[0045] 1 User terminal 101 CPU 102 ROM 103 RAM 104 Storage device 105 Input device 106 Display 107 Image sensor 108 Communication interface 109 Bus

Claims

1. A method for identifying an AR marker and generating an AR prompt effect and an AR final effect, the method comprising: detecting whether a shape of the AR marker matches a database; detecting whether an image of the AR marker matches a database; identifying a position of the AR marker located within a corresponding frame of an AR UI; and generating the AR prompt effect.

2. The method according to claim 1, wherein the steps of detecting the shape, detecting the image, and specifying the position are performed in this order.

3. The method of claim 1, wherein the shape determination step, the image determination step, and the position determination step are executed for a plurality of AR markers displayed in the AR space, and the AR prompt effect generation step generates an AR prompt effect corresponding to the plurality of AR markers if the determinations in the shape determination step, the image determination step, and the position determination step are all positive.

4. The method according to any one of claims 1 to 3, wherein the processing is terminated when a negative determination is made in at least one of the shape determination step, the image determination step, and the position determination step.

5. An information processing system for determining an AR (Augmented Reality) marker, comprising: a shape determination means for acquiring first shape information of the AR marker displayed in the AR space and determining whether it matches second shape information; an image determination means for acquiring first image information of the AR marker and determining whether it matches second image information; a position determination means for acquiring first position information of the AR marker and determining whether it matches second position information; and an effect generation step for generating an AR prompt effect when the determinations of the shape determination means, the image determination means, and the position determination means are all positive.

6. The information processing system according to claim 5, wherein the AR prompt effect is generated when the AR marker matches a virtual card slot displayed in the AR space.

7. The information processing system according to claim 5, wherein the shape determination means is executed with priority over the image determination means and the position determination means.

8. The information processing system according to claim 7, wherein the image determination means is executed with priority over the position determination means.

9. A method for recognizing an object in an AR (Augmented Reality) space, comprising: a determination step for determining whether the shape, image, and position of at least one object in real space match one of a plurality of indicators; a termination step for terminating the determination process when it is determined that at least one of the shape, image, and position does not match; and a generation step for generating a determination completion AR prompt effect when it is determined that the shape, image, and position all match.

10. The object recognition method according to claim 9, wherein at least one of the plurality of indicators includes coordinate information, and the determination is performed preferentially for an indicator whose relative position between the at least one object in real space and any of the plurality of indicators is closest.

Images