A system and method for improving player interaction using augmented reality.

Abstract

A computer-implemented method, gaming device, and computer-readable medium for an augmented reality (AR) app. The method includes maintaining an AR game environment during a game session. The method also includes receiving image data representing a real-world scene including one or more real-world objects. The method also includes detecting at least one real-world object in the image data from a set of detectable real-world objects. The method further includes generating at least one virtual character in the AR game environment upon detecting the at least one real-world object in the image data, the at least one virtual character having an association with the at least one real-world object.

Description

[Technical Field]

[0001] This application claims the interests of U.S. Provisional Patent Application No. 62 / 939,889, filed on November 25, 2019, for the invention “System and Method for Enhancing Player Interaction Using Augmented Reality,” and is incorporated herein by reference.

[0002] The present invention relates to computer games in general, and more particularly to interactive computer games that utilize augmented reality. [Background technology]

[0003] One of the main objectives of computer games is to provide a rewarding gaming experience characterized by high player engagement and interest.

[0004] Augmented reality (AR) is a technology that overlays images of computer-generated virtual objects onto images of the real world captured by a camera. For example, AR can be applied to images of real-world objects and then decorated with computer graphics. AR filters offered in applications such as Instagram® and Snapchat® are one example of this.

[0005] The proliferation of mobile devices equipped with positioning systems (GPS, Global Positioning System), digital compasses, and accelerometers has led to an expansion of augmented reality (AR) use in mobile games. In particular, the use of AR in games allows players to view the virtual world (including images of computer-generated virtual objects) and the real world (including images of real-world objects) side by side. The use of AR in mobile games has evolved to make virtual characters appear based on location information, and players interact with these characters as part of the game's strategy and plot. This has resulted in the creation of highly entertaining mobile games and has brought significant revenue to the gaming industry ecosystem. [Overview of the project] [Problems that the invention aims to solve]

[0006] However, current AR-based mobile games generally require players to move to predetermined locations that can be determined by positioning systems. This can make gameplay inconvenient, especially in confined spaces. Furthermore, game outcomes depend more on the device being in a predetermined location and having a certain orientation than on the visual content of the images. This increases predictability, which can diminish the game's appeal over time.

[0007] Therefore, the gaming industry is showing interest in providing AR-based mobile games that expand the range of experiences players can have, as a function of visual content in the surrounding environment. [Means for solving the problem]

[0008] In various embodiments, the present disclosure relates to a method performed by a processor of a computing system. The method comprises maintaining an augmented reality (AR) game environment during a game session and receiving image data representing a real-world scene including one or more real-world objects. The method also comprises detecting at least one real-world object from a discoverable set of real-world objects from the image data, and, once the at least one real-world object is detected in the image data, generating at least one virtual character in the AR game environment, the at least one virtual character having association with the at least one real-world object.

[0009] In yet another aspect, the disclosure relates to a gaming device comprising at least one processor, memory for storing instructions for execution by the processor, at least one input device configured to receive input from a user, and at least one output device configured to provide output to a user, wherein the at least one processor is configured to execute instructions in the memory to implement an interactive computer program that generates the output in response to the received input, the interactive computer program includes at least one process for maintaining an augmented reality (AR) game environment during a game session and receiving image data representing a real-world scene including one or more real-world objects. The process also includes detecting at least one real-world object from the image data from a set of discoverable real-world objects. The process further includes generating at least one virtual character in the AR game environment when the at least one real-world object is detected in the image data, the at least one virtual character having association with the at least one real-world object.

[0010] In yet another aspect, the Disclosure relates to a computer-readable storage medium which, when read and executed by at least one processor of a gaming device, causes the gaming device to execute a method included in an interactive computer program, the method comprising maintaining an augmented reality (AR) game environment during a game session and receiving image data representing a real-world scene including one or more real-world objects. The method also comprises detecting at least one real-world object from a discoverable set of real-world objects from the image data, and, when the at least one real-world object is detected in the image data, generating at least one virtual character in the AR game environment, the at least one virtual character having association with the at least one real-world object.

[0011] These and other aspects of the present disclosure will now be apparent to those skilled in the art by examining the description of the embodiments in conjunction with the accompanying drawings. [Brief explanation of the drawing]

[0012] For a more complete understanding of this disclosure, please refer here to the following description relating to the attached drawings.

[0013] [Figure 1A] Figure 1A shows an image of the first scene presented by an AR application according to a non-limiting embodiment. [Figure 1B] Figure 1B shows an image of the first scene presented by an AR application according to a non-limiting embodiment. [Figure 1C] Figure 1C shows an image of the first scene presented by an AR application according to a non-limiting embodiment. [Figure 1D] Figure 1D shows an image of the first scene presented by an AR application according to a non-limiting embodiment. [Figure 2A] Figure 2A shows an image of the second scene presented by an AR application according to a non-limiting embodiment. [Figure 2B] Figure 2B shows an image of the second scene presented by an AR application according to a non-limiting embodiment. [Figure 3A] Figure 3A shows an image of the third scene presented by an AR application according to a non-limiting embodiment. [Figure 3B] Figure 3B shows an image of the third scene presented by an AR application according to a non-limiting embodiment. [Figure 3C] Figure 3C shows an image of the third scene presented by an AR application according to a non-limiting embodiment. [Figure 3D] Figure 3D shows an image of the third scene presented by an AR application according to a non-limiting embodiment. [Figure 3E]FIG. 3E is a diagram showing content of a non-limiting example of a collection screen of virtual characters generated by an AR application. [Figure 4A] FIG. 4A is a diagram showing an image of a fourth scene presented by an AR application according to a non-limiting embodiment. [Figure 4B] FIG. 4B is a diagram showing an image of a fourth scene presented by an AR application according to a non-limiting embodiment. [Figure 4C] FIG. 4C is a diagram showing an image of a fourth scene presented by an AR application according to a non-limiting embodiment. [Figure 5] FIG. 5 is a diagram showing other non-limiting exemplary content of a collection screen of virtual characters generated by an AR application. [Figure 6] FIG. 6 is a flowchart showing steps of a method executed by an AR application according to a non-limiting embodiment. [Figure 7] FIG. 7 is a block diagram showing a structure of a mobile terminal for executing an AR application according to a non-limiting embodiment. [Figure 8] FIG. 8 is a block diagram conceptually showing components of an AR application according to a non-limiting embodiment. [Figure 9] FIG. 9 is a block diagram showing a relationship between a memory of a mobile terminal and a process shown in FIG. 6 according to a non-limiting embodiment. [Figure 10A] FIG. 10A is a block diagram conceptually showing an example of real-world object data according to a non-limiting embodiment. [Figure 10B] FIG. 10B is a block diagram conceptually showing an example of game data according to a non-limiting embodiment. [Figure 10C] FIG. 10C is a diagram conceptually showing an example of a real-world object library according to a non-limiting embodiment. [Figure 10D] FIG. 10D is a diagram conceptually showing an example of a virtual character library according to a non-limiting embodiment. [Figure 10E]Figure 10E is a conceptual diagram illustrating an example of character data according to a non-limiting embodiment. [Figure 11] Figure 11 is a table containing information about a set of detectable real-world objects, according to a non-limiting embodiment. [Figure 12] Figure 12 is a table storing information about a virtual character, according to a non-limiting embodiment. [Figure 13] Figure 13 is a table containing information about the relationship between real-world objects and virtual characters, according to a non-limiting embodiment. [Figure 14] Figure 14 is a flowchart showing the steps of a real-world object selection algorithm according to a non-restrictive embodiment. [Figure 15] Figure 15 is a flowchart showing the steps of a positioning algorithm according to a non-limiting embodiment. [Figure 16] Figure 16 is a flowchart showing the steps of a method performed by an AR app, according to a non-limiting embodiment. [Figure 17A] Figure 17A shows an image of the fifth scene presented by an AR application, according to a non-limiting embodiment. [Figure 17B] Figure 17B shows an image of the fifth scene presented by the AR application, according to a non-limiting embodiment. [Figure 17C] Figure 17C shows an image of the fifth scene presented by the AR application, according to a non-limiting embodiment. [Figure 18A] Figure 18A shows an image of the sixth scene presented by an AR application, according to a non-limiting embodiment. [Figure 18B] Figure 18B shows an image of the sixth scene presented by an AR application, according to a non-limiting embodiment. [Figure 18C] Figure 18C shows an image of the sixth scene presented by an AR application, according to a non-limiting embodiment. [Figure 19]Figure 19 is a flowchart showing the steps of a selection / matching algorithm according to a non-limiting embodiment. [Figure 20A] Figure 20A shows an image of the seventh scene presented by an AR application, according to a non-limiting embodiment. [Figure 20B] Figure 20B shows an image of the seventh scene presented by an AR application, according to a non-limiting embodiment. [Figure 20C] Figure 20C shows an image of the seventh scene presented by an AR application, according to a non-limiting embodiment. [Figure 20D] Figure 20D shows an image of the seventh scene presented by an AR application, according to a non-limiting embodiment. [Figure 20E] Figure 20E shows an image of the seventh scene presented by an AR application, according to a non-limiting embodiment. [Figure 21] Figure 21 shows a shared AR experience between a first player and a second player according to a non-limiting embodiment. [Figure 22] Figure 22 is a flowchart showing the steps of a virtual character selection algorithm according to a non-limiting embodiment. [Modes for carrying out the invention]

[0014] It should be clearly understood that this specification and the drawings are for illustrative purposes only and to aid in understanding specific embodiments. They are not intended to be, and should not be, limiting.

[0015] With respect to Figure 7, a mobile device 10 is shown. In various embodiments, the mobile device 10 may be a smartphone, tablet, game console, headset, or so-called "smart" glasses or "smart" contact lenses, by several non-limiting possibilities.

[0016] The mobile terminal 10 comprises memory 14 and at least one processor 12. The at least one processor 12 may include one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs). The mobile terminal 10 can run an operating system such as Android®, iOS®, or any other mobile operating system. A bus 18 may enable communication between the at least one processor 12 and the memory 14. The mobile terminal 10 also comprises a display 11 (e.g., a touchscreen 16), a camera 26, a microphone 24, and a speaker 20.

[0017] The mobile terminal 10 may be connected to a data network 30 via a network input / output interface 25. Depending on the embodiment, the data network 30 may be the Internet, a local area network, a wireless network, a combination of such networks, or any other form of data network.

[0018] Player 1 provides player input via at least one input device 28 (including, for example, one or more of a joystick, touchscreen 16, keyboard, controller, microphone 24, camera 26, and / or gesture sensor). The display 11 (e.g., touchscreen 16), camera 26, microphone 24, and speaker 20 may be connected to at least one processor 12 via input / output interface (I / O) 21 and bus 18.

[0019] A mobile AR application can be viewed as an interactive computer program defined by computer-readable instructions 32 stored in memory 14 and read and executed by at least one processor 12. In some embodiments, a mobile AR application may be referred to as a mobile AR game.

[0020] In a non-limiting embodiment, Figure 8 is a block diagram of a mobile AR application 1000 which can be seen as being composed of a plurality of interconnected modules. The mobile AR application 1000 may consist of an object detection module (ODM) 1010, a computer vision plug-in (CVP) 1030, and an AR application programming interface (AR API) 1020. In some embodiments, the ODM 1010, together with the CVP 1030 and the AR API 1020, is a software processing block that implements the method shown in Figure 6.

[0021] CVP1030 can be an optional software component that adds computer vision capabilities, such as computer vision machine learning, to ODM1010. Examples of computer vision capabilities include those provided by TensorFlow® Lite for the Android® operating system and CoreML® for the iOS® operating system.

[0022] AR API1020 can be any software component that provides or otherwise enables access to augmented reality features such as point cloud data to ODM1010. Examples of augmented reality features include those provided by ARCore® in the Android® operating system and ARKit® in the iOS® operating system.

[0023] Images of the scene in the AR game environment 1005 may be captured by a mobile AR application 1000. Specifically, the ODM 1010 captures images from the camera feed (i.e., video frames) of the mobile terminal 10. The scene may consist of real-world objects that appear in the images of the scene. In some embodiments, one or more real-world objects may be inanimate objects (e.g., a table, remote control, hose, etc.). In other embodiments, one or more real-world objects may be living objects (e.g., a person, animal, etc.). In general, there is no particular limit to the number of real-world objects that may be present in a given scene.

[0024] For example, Figure 1A shows an image 100A of a first scene that may be captured and presented on the display 11 of a mobile device 10 by a mobile AR application 1000. Image 100A is shown to include two real-world objects, namely a real-world table 101 and a real-world remote control 102.

[0025] Figures 1B to 1D show subsequent images 100B to 100D of a scene presented by the mobile AR application 1000 on the display 11 of the mobile device 10. As the images progress, a virtual character 104 appears to be emitted from / generated by the real-world remote control 102.

[0026] The generation of the aforementioned virtual characters and their association with real-world objects is a result of method 600, which is implemented by the mobile AR application 1000, which will be described in more detail in Figure 6.

[0027] Step 610 The mobile AR application 1000 is configured to acquire or receive images captured by the camera 26 of the mobile device 10. As previously mentioned, the ODM 1010 captures images from the camera feed (i.e., video frames) of the mobile device 10. Generally speaking, the acquired camera images (or more generally, "camera data") may or may not contain machine-recognizable real-world objects in the set of discoverable real-world objects 58 associated with the mobile AR application 1000.

[0028] The discoverable real-world object set 58 may be a subset of the real-world object set stored in the real-world object library 300 associated with the mobile AR application 1000. The real-world object library 300 may be encoded in game data 35 stored in memory 14. Each real-world object in the real-world object library 300 (including each real-world object in the discoverable real-world object set 58) may include identification information. For example, each real-world object in the real-world object library 300 may be assigned an object name 50x.

[0029] In some embodiments, the mobile AR application 1000 may be configured to classify each real-world object in the real-world object library 300 (including each real-world object in the discoverable real-world object set 58) into an object family 501x ​​("object family"). In some embodiments, the object family 501x ​​may be associated with a physical location in the real world. In some embodiments, the object family 501x ​​may be associated with an expected physical location in the real world where player 1 is likely to find such an object. For example, the object family 501x ​​may include kitchen objects, living room objects, backyard objects, and / or supermarket objects. The classification of real-world objects in the real-world object library 300 into object families 501x ​​may be encoded in game data 35 by the game designer during game design.

[0030] In the illustrated exemplary embodiment, the camera image includes a real-world table 101 and a real-world remote control 102. In this example, it is assumed that the real-world object library 300 includes real-world objects identified by the object name 50x for the "table" and real-world objects identified by the object name 50x for the "remote control", but that only the real-world objects identified by the object name 50x for the "remote control" are found in the discoverable real-world object set 58. Next is step 620.

[0031] Step 620 The mobile AR app 1000 is configured to process the camera image acquired or received in step 610. Specifically, an attempt is made to detect one or more real-world objects within the detectable real-world object set 58 in the image.

[0032] In some embodiments, attempts to detect real-world objects in a received image can be performed using a well-trained computer vision model (CVM). For example, ODM1010 sends a request to CVP1030 to process the received image using a pre-trained CVM1040. Specifically, CVM1040 attempts to identify known objects by examining the shape, color, and / or texture (pattern) in the image received from one or more viewpoints. In one implementation of this embodiment, the known objects may include real-world objects in a real-world object library 300.

[0033] Step 630 The output of CVM1040 indicates whether or not a real-world object was detected. In step 630, the output of CVM1040 is evaluated.

[0034] If the output of CVM1040 is positive, this indicates that a real-world object was detected in the captured image, and the next step is step 640. If it is negative, the mobile AR app 1000 repeats steps 610 and 620 until step 630 yields a positive result.

[0035] If a specific real-world object is detected in the scene, the CVM 1040 may be configured to assign an object label 60x to that specific real-world object. The object label 60x may also function as identification information associated with the real-world object. The object label 60x of the specific real-world object may be stored in memory 14.

[0036] In the illustrated exemplary embodiment, it is assumed that the real-world remote control 102 is ultimately detected, leading to a positive result in step 630. In this example, the mobile AR app 1000 is configured to assign the object label 60x "remote control" to the real-world remote control 102.

[0037] Each detected real-world object may have a set of real-world object data 62 stored in memory 14. The real-world object data 62 for each real-world object includes an object label 60x.

[0038] Given the positive result of step 630 (i.e., a specific real-world object was detected in the scene), the next step is step 640.

[0039] Step 640 In step 640, the AR API 1020 of the mobile AR app 1000 is configured to provide the anchor 46 for the final virtual character 104.

[0040] In the illustrated embodiment, anchor 46 may be established in relation to the position of the real-world remote control 102 in order to provide an anchor for the final virtual character 104. Thus, the spatial coordinates (XYZ) of the real-world remote control 102 RW And the final spatial coordinates (XYZ) of virtual character 104 VC They may be nearly identical, or they may be identical.

[0041] Once the anchor 46 is roughly positioned at the location of the specific real-world object, in this embodiment, the mobile AR application 1000 is configured to communicate information to the player 1 indicating the presence of the specific real-world object on the display 11. For this reason, the mobile AR application 1000 may provide feedback to the player 1 in response to the detection of the specific real-world object. For example, the feedback may be an auditory signal, a visual signal, or a tactile signal emitted by the mobile terminal 10.

[0042] In one implementation example, the graphical user interface (GUI) 48 of the mobile AR application 1000 may be configured to display a visual indicator 34 on the display 11 to inform the player 1 of the presence of a particular real-world object. The visual indicator 34 may be a graphical indicator 36 (e.g., a shape that defines the particular real-world object) and / or a text indicator 38 (e.g., a text-based message to the player 1 indicating the presence of the particular real-world object). In another embodiment, once the anchor 46 is established, the mobile AR application 1000 does not transmit information to the player 1 indicating the presence of the particular real-world object on the display 11.

[0043] In some embodiments, when a particular real-world object is detected by the CVM 1040, the mobile AR app 1000 may be configured to provide the player 1 with an opportunity to provide input related to that particular real-world object. For example, the mobile AR app 1000 may respond when the player 1 touches a portion of the display 11 / touchscreen 16 on which the particular real-world object is displayed. In other examples, the mobile AR app 1000 may respond when the player 1 touches another portion of the display 11 / touchscreen 16 or provides input via any other suitable mechanism. In a non-limiting example, the mobile AR app 1000 may be configured to provide the player 1 with an opportunity to provide an audible command transmitted via the microphone 24 of the mobile terminal 10. The next step is step 650.

[0044] Step 650 The mobile AR application 1000 establishes an association between a specific real-world object and a virtual character (hereinafter referred to as the "associated virtual character"). This association may be triggered as a result of player input (as described above) or may be triggered independently of player input. Therefore, the mobile AR application 1000 is configured to store the association between the specific real-world object and the associated virtual character in memory 14. In some embodiments, the mobile AR application 1000 is configured to simultaneously display the virtual character associated with the specific real-world object on the display 11.

[0045] In some embodiments, the mobile AR application 1000 may be configured to select a relevant virtual character from a virtual character library 400 stored in the memory 14 of the mobile terminal 10. The virtual character library 400 may include one or more virtual characters that can be selected for association with a real-world object, such as the particular real-world object. The virtual character library 400 may be encoded in game data 35 stored in the memory 14.

[0046] Each virtual character in the virtual character library 400 may include a representation 88 such as a two-dimensional or three-dimensional representation of the virtual character. Each virtual character in the virtual character library 400 may have other representations 88 such as a non-graphic representation (e.g., a numerical, geometric, or mathematical representation). The game data 35 may include character data 37 associated with each virtual character in the virtual character library 400. The character data 37 stores data related to the current representation of the virtual character, such as a graphical representation in a game image frame, or a numerical, geometric, or mathematical representation. The character data 37 includes shape data 44, color data 52, image data, and position data (e.g., the spatial coordinates (XYZ) of the virtual character). VC), attributes 90 such as material / texture data, physical state data, field of view data, lighting data (e.g., direction, position, color and / or intensity), sound data, motion data, collision data, environment data, timer data and / or other data related to the virtual character may be stored.

[0047] Here, we refer to the flowchart in Figure 22, which shows the steps in an algorithm for selecting a related virtual character from a related virtual character family 700x to a given object family 501x ​​in which real-world objects in the real-world object library 300 are classified. Here, the steps of the virtual character selection algorithm 2200 will be described in more detail with respect to Figure 22.

[0048] In step 2210, the mobile AR app 1000 is configured to identify the object label 60x that was assigned to the specific real-world object as a result of detection by the CVM 1040 in step 630 (as described above). Thus, in step 2210, the mobile AR app 1000 accesses the memory 14 and identifies the object label 60x that was assigned to the specific real-world object in step 630.

[0049] In the example provided in step 630 above, the mobile AR app 1000 assigned the object label 60x "remote control" to the real-world remote control 102. Therefore, in step 2210, the mobile AR app 1000 accesses memory 14 and identifies the object label 60x "remote control" assigned to the real-world remote control 102. Next is step 2220.

[0050] In step 2220, the mobile AR app 1000 accesses the real-world object library 300 and is configured to match the object label 60x assigned to the specific real-world object with the object name 50x of the real-world object stored in the real-world object library 300.

[0051] For example, in step 2220, the mobile AR app 1000 is configured to match the object label 60x assigned to the real-world remote control 102, i.e., "remote control," with the object name 50x of a real-world object stored in the real-world object library 300.

[0052] As described above, it is assumed that the real-world object library 300 contains a real-world object identified by the object name 50x "remote control". Therefore, the output of step 2220 matches the object label 60x "remote control" assigned to the real-world remote control 102 with the object name 50x "remote control" of the real-world object stored in the real-world object library 300.

[0053] In step 2230, the mobile AR app 1000 is configured to determine the object family 501x ​​to which the real-world object stored in the real-world object library 300 belongs, the object name 50x of the real-world object matching the object label 60x assigned to the particular real-world object in step 2220. In this way, the mobile AR app 1000 is configured to determine the relationship between the particular real-world object and the object family 501x.

[0054] In this example, the real-world object in the real-world object library 300 associated with object name 50x "remote control" is classified under the "living room object" object family 501x. Therefore, in step 2230, the mobile AR app 1000 determines that the real-world remote control 102 is associated with the "living room object" object family 501x ​​in this example.

[0055] In this embodiment, each object family 501x ​​may have an associated virtual character family 700x ("virtual character family"). The associated family of a virtual character 700x may be a subset of virtual characters from the virtual character library 400. In the non-limiting embodiment of Figure 5, the mobile AR application 1000 is configured to include at least two unique virtual characters associated with each virtual character / object family 700x, 501x. For example, the virtual character collection screen 500 shows a collection of virtual characters 900 consisting of object families 5011, 5012, 5013, and 5014 and families of virtual characters 7001, 7002, 7003, 7004, and 7005 and a plurality of unique virtual characters associated with each virtual character / object family 700x, 501x ​​(e.g., 711, 712, 713, 714 associated with virtual character family 7011 / object family 5011).

[0056] In step 2240, the mobile AR app 1000 is configured to select a relevant virtual character by selecting one virtual character from the virtual character family 700x associated with the object family 501x ​​to which the particular real-world object is associated.

[0057] In some embodiments, the mobile AR application 1000 may be configured to randomly select a virtual character from the virtual character family 700x that has an association with a particular real-world object.

[0058] In another embodiment, the mobile AR application 1000 is configured to select one virtual character from the virtual character family 700x to have association with the particular real-world object by implementing a selection / matching algorithm 1900 (described in further detail below). In yet another embodiment, the mobile AR application 1000 is configured to implement the selection / matching algorithm 1900 without implementing the virtual character selection algorithm 2200 described above.

[0059] Herein, we refer to the flowchart in Figure 19, which shows the steps of the selection / matching algorithm 1900 performed by at least one processor 12 of the mobile terminal 10. In some embodiments, the mobile AR app 1000 may be configured to perform the selection / matching algorithm 1900 to select one virtual character from the virtual character library 400 for association with a particular real-world object. In other embodiments, the mobile AR app 1000 may be configured to perform the selection / matching algorithm 1900 to select one virtual character from a subset of the virtual character library 400, i.e., from the virtual character family 700x. For the purposes of this specification, the selection / matching algorithm 1900 is described below with respect to selecting one virtual character from the virtual character library 400.

[0060] In step 1910, the mobile AR app 1000 is configured to identify matching parameters associated with a specific real-world object, and in step 1920, the mobile AR app 1000 is configured to identify corresponding matching parameters associated with a virtual character in the virtual character library 400. Details regarding these matching parameters are provided below. In step 1930, the mobile AR app 1000 is configured to compare and match the values ​​of the parameters associated with the specific real-world object with the values ​​of the corresponding parameters associated with a predetermined virtual character in the virtual character library 400. In step 1940, the mobile AR app 1000 is configured to select a predetermined virtual character in the virtual character library 400 based on the degree of match determined in step 1930, and that predetermined virtual character is selected to be associated with the specific real-world object.

[0061] Next, we will provide a more detailed explanation of an implementation example of the selection / matching algorithm 1900.

[0062] As described above, in step 1910, the mobile AR app 1000 is configured to identify matching parameters associated with the particular real-world object. In one example, the matching parameters may be an object label 60x assigned to the particular real-world object as a result of detection by the CVM 1040 in step 630 (as described above). During game design, the game designer may encode the specification of the object label 60x as matching parameters for the real-world object detected in the image captured by the ODM 1010. This specification may be encoded in game data 35 stored in memory 14.

[0063] Thus, in step 1910, the mobile AR app 1000 accesses memory 14 and identifies the object label 60x assigned to the particular real-world object as a matching parameter associated with that particular real-world object.

[0064] In step 1920, the mobile AR app 1000 is configured to identify corresponding matching parameters associated with multiple virtual characters in the virtual character library 400. In this example, the matching parameter is the candidate real-world object label 42x.

[0065] In the early stages of game design, the game designer can specify one or more candidate real-world objects for each virtual character in the virtual character library 400. For a given virtual character in the virtual character library 400, one or more candidate real-world objects correspond to real-world objects that can be detected by the CVM 1040 in images captured by the ODM 1010 and may be considered for association with the given virtual character.

[0066] During game design, the game designer may access the virtual character library 400 and encode one or more real-world objects from the real-world object library 300 as one or more candidate real-world objects for each virtual character in the virtual character library 400. This specification may also be encoded in the game data 35 stored in memory 14. In this way, one or more candidate real-world objects may be pre-set by the game designer / producer.

[0067] Each of the one or more candidate real-world objects may include identification information such as a candidate real-world object label 42x. For each virtual character in the virtual character library 400, the candidate real-world object label 42x of the one or more candidate real-world objects may be stored in the game data 35 as character data 37 stored in memory 14.

[0068] Furthermore, during game design, the game designer may encode the designation of candidate real-world object labels 42x as matching parameters for multiple virtual characters in the virtual character library 400. The game designer may also encode the designation of candidate real-world object labels 42x as the corresponding matching parameters for the object labels 60x of a particular real-world object. This designation may be encoded in the game data 35 stored in memory 14.

[0069] Returning to the selection / matching algorithm 1900, in step 1920, the mobile AR app 1000 accesses memory 14 and identifies candidate real-world object labels 42x as the corresponding matching parameters for the object labels 60x of the particular real-world object.

[0070] In step 1930, the mobile AR app 1000 is configured to compare and match the value of a parameter associated with the particular real-world object with the value of the corresponding parameter associated with a predetermined one of several virtual characters in the virtual character library 400. In this example, the mobile AR app 1000 is configured to match (i) the value of a candidate real-world object label 42x for a predetermined virtual character in the virtual character library 400 with (ii) the value of an object label 60x assigned to the particular real-world object by the CVM 1040.

[0071] For example, the mobile AR application 1000 may be configured to compare the aforementioned multiple labels by comparing a text string stored in memory 14 associated with an object label 60x with a text string stored in memory 14 associated with a candidate real-world object label 42x, and identifying the number of similarities between the text strings. The mobile AR application 1000 may be configured to require a threshold number of similarities to define the degree of match between the text strings. When a similarity number greater than or equal to the threshold number of similarities is detected, the mobile AR application 1000 may consider the candidate real-world object label 42x to match the object label 60x assigned to that particular real-world object. The threshold number of similarities may be set by the game designer during game design and stored in game data 35.

[0072] In step 1940, when matching the object label 60x of the particular real-world object with the candidate real-world object label 42x of a predetermined virtual character in the virtual character library 400, the mobile AR app 1000 is configured to select the predetermined virtual character to have an association with the particular real-world object.

[0073] The selection / matching algorithm 1900 will be illustrated by a non-restrictive example of selecting a virtual character 104 to associate with a real-world remote control 102 based on object labels 60x and candidate real-world object labels 42x.

[0074] In step 1910, the mobile AR app 1000 is configured to identify matching parameters associated with the real-world remote control 102. Thus, the mobile AR app 1000 accesses memory 14 and identifies the object label 60x assigned to the real-world remote control 102 as a matching parameter associated with the real-world remote control 102. In this case, detection by CVM 1040 assigns the object label 60x "remote control" to the real-world remote control 102. Therefore, in step 1910, the matching parameter relating to the real-world remote control 102 is identified as object label 60x "remote control".

[0075] In step 1920, the mobile AR app 1000 is configured to identify corresponding matching parameters associated with multiple virtual characters in the virtual character library 400. The mobile AR app 1000 accesses memory 14 and identifies candidate real-world object labels 42x as matching parameters for multiple virtual characters in the virtual character library 400 and as corresponding matching parameters for object labels 60x assigned to the real-world remote control 102.

[0076] In step 1930, the mobile AR app 1000 is configured to compare and match the value of 1 of the object label 60x assigned to the real-world remote control 102 with the value of a candidate real-world object label 42x for a predetermined 1 of several virtual characters in the virtual character library 400. In this example, the mobile AR app 1000 compares the text string "remote control" stored in memory 14 in association with the object label 60x of the real-world remote control 102 with the text strings stored in memory 14 in association with the candidate real-world object label 42x for each of the several virtual characters in the virtual character library 400 to identify similarities between the text strings.

[0077] In this example, the text string stored in memory associated with the candidate real-world object label 42x of the virtual character 104 is "remote control (multiple)". The mobile AR app 1000 detects the similarity between the text string "remote control" associated with object label 60x and the text string "remote control (multiple)" associated with the candidate real-world object label 42x of the virtual character 104. In this example, it is assumed that the number of similarities detected by the AR app 1000 is greater than or equal to the threshold number of similarities required to classify the degree of agreement between multiple text strings. Thus, the mobile AR app 1000 considers the candidate real-world object label 42x "remote control (multiple)" associated with the virtual character 104 to match the object label 60x "remote control" assigned to the real-world remote control 102.

[0078] In step 1940, when the object label 60x of the real-world remote control 102 and the candidate real-world object label 42x of the virtual character 104 are matched, the mobile AR app 1000 selects the virtual character 104 from the virtual character library 400 that has an association with the real-world remote control 102.

[0079] In some embodiments, the associated virtual character may be selected to have a visual association with the particular real-world object. In other words, the associated virtual character may have visual properties that correspond to the real-world visual properties of the particular real-world object.

[0080] The visual characteristics may include, but are not limited to, the shape of the particular real-world object, the color of the particular real-world object, or other visual characteristics of the particular real-world object. The mobile AR application 1000 is configured to display the particular real-world object on the display 11 and simultaneously display the visual characteristics of the associated virtual character on the display 11.

[0081] In the illustrated example, the virtual character 104 is selected to have a visual relevance to the real-world remote control 102, in which case the shape of the virtual character 104 corresponds to the real-world shape of the real-world remote control 102. As shown in Figures 1C and 1D, the mobile AR application 1000 is configured to display the shape of the real-world remote control 102 on the display 11 and the shape of the virtual character 104 on the display 11 at the same time.

[0082] In this example, the virtual character 104 is further selected for visual association with the real-world remote control 102 by a selection / matching algorithm 1900, which is performed by at least one processor 12, as described above. During game design, the game designer may designate "remote control(plural)" as one of the predetermined candidate real-world object labels 42x designated for the virtual character 104, based on the similarity observed between the shape of the virtual character 104 and the shape of the real-world remote control.

[0083] In other embodiments of this embodiment, the associated virtual character may be selected to visually associate with a specific real-world object, in addition to the selection / matching algorithm 1900 being executed by at least one processor 12 based on matching parameters different from those of the object label 60x and the candidate real-world object label 42x.

[0084] In this example, CVM1040 may be trained to detect the shape of a particular real-world object. For example, CVM1040 may be configured to extract shape data 64 related to the particular real-world object from camera data acquired in step 610 of method 600. The shape data 64 may be stored in memory 14 as part of a real-world object dataset 62. The shape data 64 may include, but is not limited to, boundary data, contour data, edge data, point cloud data, and / or curvature data of the particular real-world object. During game design, the game designer may encode a specification of the shape data 64 as a matching parameter for real-world objects that can be detected in images captured by ODM1010. This specification may be encoded in game data 35 stored in memory 14.

[0085] As described above, the shape data 44 of each virtual character in the virtual character family 400 is included in the game data 35 stored in the memory 14 of the mobile terminal 10. The shape data 44 of each virtual character in the virtual character family 400 may include, but is not limited to, boundary data, contour data, edge data, point cloud data, and / or curvature data of the virtual character.

[0086] During game design, the game designer may encode the specification of shape data 44 as a matching parameter for multiple virtual characters in the virtual character library 400. Alternatively, the game designer may encode the specification of the shape data 44 of the virtual character as a matching parameter corresponding to the shape data 64 of a particular real-world object. This specification may also be encoded in the game data 35 stored in memory 14.

[0087] In one example, the shape of the associated virtual character may correspond to the shape of a particular real-world object based on the matching of (i) the shape data 64 of the particular real-world object and (ii) the shape data 44 of a predetermined virtual character in the virtual character library 400.

[0088] In this example, in step 1910, the mobile AR application 1000 is configured to identify matching parameters associated with the specific real-world object. Thus, the mobile AR application 1000 accesses the memory 14 and identifies the shape data 64 as matching parameters associated with the specific real-world object.

[0089] In step 1920, the mobile AR app 1000 is configured to identify the corresponding matching parameters associated with the virtual character in the virtual character library 400. In this way, the mobile AR app 1000 accesses the memory 14 and identifies the shape data 44 as the matching parameters corresponding to the shape data 64 of the particular real-world object.

[0090] In step 1930, in this example, the mobile AR application 1000 is configured to compare and match the shape data 64 value of the specific real-world object with the shape data 44 value of the virtual character in the virtual character library 400. The mobile AR application 1000 may also be configured to match the shape data 44, 64 by comparing the data point cloud associated with the shape data 64 associated with the real-world object with the corresponding data point cloud associated with each shape data 44 of the virtual character in the virtual character library 400 and identifying the similarity between these data points. If a similarity count greater than or equal to a threshold is detected, the mobile AR application 1000 may consider that the shape data 64 of the real-world object matches one predetermined shape data 44 of multiple virtual characters in the virtual character library 400. The threshold similarity count may be set by the game designer during game design and stored in the game data 35.

[0091] In step 1940, when matching the shape data 64 of the specific real-world object with the shape data 44 of a predetermined virtual character in the virtual character library 400, the mobile AR application 1000 selects a predetermined virtual character from the virtual character library 400 and is configured to have a visual relationship with the specific real-world object.

[0092] The selection / matching algorithm 1900 is illustrated by a non-limiting example as one for selecting a virtual character 104 to visually associate with a real-world remote control 102 based on shape data 64, 44.

[0093] In step 1910, the mobile AR app 1000 is configured to identify matching parameters associated with the real-world remote control 102. Thus, the mobile AR app 1000 accesses the memory 14 and identifies the shape data 64 associated with the real-world remote control 102 as matching parameters associated with the real-world remote control 102. In this case, the CVM 1040 is configured to extract the shape data 64 of the remote control 102 from the camera data acquired in step 610 of method 600.

[0094] In step 1920, the mobile AR application 1000 is configured to identify corresponding matching parameters associated with virtual characters in the virtual character library 400. The mobile AR application 1000 accesses memory 14 and identifies the shape data 44 of the multiple virtual characters as matching parameters for the multiple virtual characters in the virtual character library 400, as well as corresponding matching parameters for the shape data 64 associated with the real-world remote control 102.

[0095] In step 1930, the mobile AR app 1000 is configured to compare and match the values ​​of the shape data 64 associated with the real-world remote control 102 with the values ​​of the shape data 44 for a predetermined one of several virtual characters in the virtual character library 400. In this example, the mobile AR app 1000 compares the data point cloud associated with the shape data 64 associated with the real-world remote control 102 with the corresponding data point clouds associated with each of the several virtual characters in the virtual character library 400 and detects the similarity between these data points.

[0096] In this example, the number of similarities detected by the AR app 1000 is assumed to be greater than or equal to the threshold number of similarities required to classify the matches between shape data 64 and 44. In this way, the mobile AR app 1000 considers the shape data 44 associated with the virtual character 104 to be a match with the shape data 64 associated with the real-world remote control 102.

[0097] In step 1940, when the shape data 64 associated with the real-world remote control 102 is matched with the shape data 44 associated with the virtual character 104, the mobile AR app 1000 selects the virtual character 104 so that it can be visually associated with the real-world remote control 102.

[0098] Therefore, the virtual character 104 is selected to have a visual relationship with the real-world remote control 102, such that the virtual character 104 has visual characteristics that correspond to the real-world visual characteristics of the real-world remote control 102. In this example, as shown in Figures 1C and 1D, the shape of the virtual character 104 corresponds to the real-world shape of the real-world remote control 102, and the mobile AR application 1000 is configured to display the shape of the virtual character 104 on the display 11 at the same time as displaying the shape of the real-world remote control 102 on the display 11.

[0099] In some embodiments, the associated virtual character is selected to have a visual association with a particular real-world object such that one color of the associated virtual character corresponds to one real-world color of the particular real-world object.

[0100] In one implementation of this embodiment, the CVM 1040 may be appropriately trained to detect and extract color data 66 associated with the particular real-world object from the camera data acquired by the ODM 1010 in step 610. The color data 66 may include data stored in memory 14 as part of a real-world object dataset 62. The color data 66 may include the color of the particular real-world object according to color rules known in the art, such as being represented by the Pantone® Matching System, cyan / magenta / yellow / black (CMYK) values, red / green / blue (RGB) values, or hexadecimal code (HEX) values. The color data 66 may also include other information such as saturation data and / or lightness data.

[0101] During game design, the game designer may encode color data 66 as matching parameters for real-world objects that can be detected in images captured by ODM1010. This specification may also be encoded in game data 35 stored in memory 14.

[0102] As described above, the color data 52 for each virtual character in the virtual character family 400 is included in the game data 35 stored in the memory 14 of the mobile terminal 10. The color data 52 for each virtual character in the virtual character family 400 may include, but is not limited to, the color of the particular real-world object represented according to color rules known in the art. The color data 52 may also include other information such as saturation data and / or brightness data.

[0103] During game design, the game designer can encode the color specification 52 as a matching parameter for a virtual character in the virtual character library 400. Alternatively, the game designer may encode the color data 52 specification for the virtual character as a matching parameter corresponding to the color data 66 of a specific real-world object. This specification may also be encoded in the game data 35 stored in memory 14.

[0104] In one example, the color of the associated virtual character may correspond to the color of the particular real-world object based on a match between (i) the color data 66 of the particular real-world object and (ii) the color data 52 of a predetermined virtual character in the virtual character library 400.

[0105] In this example, in step 1910, the mobile AR application 1000 is configured to identify matching parameters associated with the specific real-world object. Thus, the mobile AR application 1000 accesses memory 14 and identifies the color data 66 as matching parameters associated with the specific real-world object.

[0106] In step 1920, the mobile AR application 1000 is configured to identify corresponding matching parameters associated with a virtual character in the virtual character library 400. Thus, the mobile AR application 1000 accesses memory 14 and identifies color data 52 as matching parameters corresponding to the color data 66 of the particular real-world object.

[0107] In step 1930, in this example, the mobile AR app 1000 is configured to compare and match the value of 1 in the color data 66 of the specific real-world object with the value of 1 in the color data 52 of the virtual character in the virtual character library 400. The mobile AR app 1000 may also be configured to match the color data 52, 64 by comparing the data point cloud of 1 associated with the color data 66 associated with the real-world object with the corresponding data point clouds associated with each of the multiple virtual characters in the virtual character library 400 and identifying the similarity between these data points. If a similarity count greater than or equal to a threshold is detected, the mobile AR app 1000 may consider that the color data 66 of the real-world object matches a predetermined color data 52 of one of the multiple virtual characters in the virtual character library 400. The threshold similarity count may be set by the game designer during game design and stored in the game data 35.

[0108] In step 1940, when the color data 66 of the particular real-world object is matched with the color data 52 of a predetermined virtual character in the virtual character library 400, the mobile AR app 1000 is configured to select the predetermined virtual character from the virtual character library 400, and the predetermined 1 of the virtual character is selected to have a visual association with the particular real-world object.

[0109] The selection / matching algorithm 1900 is illustrated by a non-limiting example as one for selecting a virtual character 104 to visually associate with a real-world remote control 102 based on color data 66, 52.

[0110] In step 1910, the mobile AR app 1000 is configured to identify one matching parameter associated with the real-world remote control 102. Thus, the mobile AR app 1000 accesses memory 14 and identifies the color data 66 associated with the real-world remote control 102 as the matching parameter associated with the real-world remote control 102. In this case, the CVM 1040 is configured to extract the color data 66 of the remote control 102 from the camera data acquired by the ODM 1010 in step 610 of method 600.

[0111] In step 1920, the mobile AR app 1000 is configured to identify corresponding matching parameters associated with virtual characters in the virtual character library 400. The mobile AR app 1000 accesses memory 14 and identifies the color data 52 of the virtual character as a matching parameter for multiple virtual characters in the virtual character library 400, and as a corresponding matching parameter for color data 66 associated with the real-world remote control 102.

[0112] In step 1930, the mobile AR app 1000 is configured to compare and match a value of 1 in the color data 66 associated with the real-world remote control 102 with a value of 1 in the color data 52 for a predetermined number of virtual characters in the virtual character library 400. In this example, the mobile AR app 1000 compares the data point cloud associated with the color data 66 associated with the real-world remote control 102 with the corresponding data point cloud associated with each of the multiple virtual characters in the virtual character library 400 to detect similarity between these data points.

[0113] In this example, it is assumed that the number of similarities detected by the AR app 1000 is greater than or equal to the threshold number of similarities required to classify the match between color data 66 and 52. Thus, the mobile AR app 1000 considers the color data 52 associated with the virtual character 104 to be a match with the color data 66 associated with the real-world remote control 102.

[0114] In step 1940, when the color data 66 associated with the real-world remote control 102 and the color data 52 associated with the virtual character 104 are matched, the mobile AR app 1000 selects the virtual character 104 for visual association with the real-world remote control 102.

[0115] Therefore, the virtual character 104 is selected to have a visual association with the real-world remote control 102, such that it has visual characteristics corresponding to the real-world visual characteristics of the real-world remote control 102. In this example, the mobile AR application 1000 is configured to display the colors of the real-world remote control 102 on the display 11, and at the same time, to display the colors of the virtual character 104 on the display 11.

[0116] In addition to, or selectively, the visual characteristics described above, in some embodiments, the associated virtual character may be represented as having other characteristics related to the particular real-world object, such as other visual characteristics, locomotion characteristics, audible characteristics, and / or tactile characteristics. Examples of these characteristics are described in further detail below.

[0117] In some embodiments, the associated virtual character may have other visual characteristics related to the particular real-world object. In a non-limiting example, if the particular real-world object is a light source (e.g., a lamp, light bulb, candle, flashlight, etc.), the associated virtual character may be represented by a glowing effect.

[0118] In some embodiments, the associated virtual character may have movement in the AR game environment 1005, and such movement may be represented as substantially similar to the expected movement of the particular real-world object (for example, movement acting under some physical force). Thus, the associated virtual character may have movement characteristics that correspond to the real-world movement characteristics of the particular real-world object. The mobile AR application 1000 is configured to display the particular real-world object on the display 11 and, at the same time, display the movement characteristics of the associated virtual character on the display 11.

[0119] As a non-limiting example, if the particular real-world object is a pendulum, the generated associated virtual character may be represented to have the appearance of swinging around a fixed point. Other movements of the associated virtual character may be possible so that it appears to move around the AR game environment 1005. A suitable collision detection model may be implemented to govern the interaction between the associated virtual character and several other real-world objects within the AR game environment 1005.

[0120] In some embodiments, the associated virtual character may have an audible association with the particular real-world object. Thus, the associated virtual character may have audible characteristics corresponding to the real-world audible characteristics of the particular real-world object.

[0121] In a non-limiting example, if the particular real-world object is a balloon, the associated virtual character may be represented to have a voice that mimics the effect of a voice under the influence of helium gas. The sound of the voice may be emitted by a mobile device 10 (for example, via a speaker 20).

[0122] In some embodiments, the associated virtual character may have a haptic connection to the specific real-world object. Thus, the associated virtual character may have haptic properties that correspond to the real-world haptic properties of the specific real-world object. The mobile AR application 1000 is configured to display the specific real-world object on the display 11 and, at the same time, display the haptic properties of the associated virtual character on the display 11.

[0123] For example, the associated virtual character may be represented by haptic feedback that describes the characteristics of the particular real-world object, and the haptic feedback is emitted by the mobile device 10 (e.g., via the haptic feedback module 54 and / or the display 11 / touchscreen 16 of the mobile device 10). For example, the mobile device 10 may be configured to transmit signals representing texture through interaction between the player 1 and the display 11 / touchscreen 16 of the mobile device 10. In a non-limiting example, if the particular real-world object has a spiky texture (e.g., a real-world cactus), the associated virtual character may be visually represented so that it is perceived as "spiky," and the player 1 can experience a spiky sensation when touching the display 11 / touchscreen 16 of the mobile device 10.

[0124] The associated virtual character 104 may be selected, as described above, so as to be visually, dynamically, audibly, and / or tactilely associated with the particular real-world object by the selection / matching algorithm 1900 being executed by at least one processor 12. In addition to specifying one or more candidate real-world objects for each of the multiple virtual characters in the virtual character library 400 during game design, the game designer may further encode one or more visual, movement, auditory, and / or tactile characteristics of the multiple virtual characters in the virtual character library 400 based on the perceived visual, movement, auditory, and / or tactile characteristics of the one or more candidate real-world objects specified for each of the multiple virtual characters in the virtual character library 400. These characteristics may consist of one or more candidate characteristics 56 encoded by the game designer in the game data 35.

[0125] Once the selection / matching algorithm 1900 selects a virtual character to associate with a particular real-world object, the mobile AR application 1000 may further configure itself to select one or more visual, motion, auditory, and tactile characteristics of one or more candidate characteristics 56 and implement characteristics relating to the associated virtual character. Thus, the visual, motion, auditory, and tactile characteristics of the associated virtual character may be displayed on the display 11 / emitted by the speaker 20 / implemented by the tactile feedback module 54 at the same time as the particular real-world object is displayed on the display 11.

[0126] In other embodiments, the mobile AR application 1000 may be configured to present a display of one or more candidate characteristics 56 to the player 1 via the display 11 of the mobile terminal 10, and the mobile AR application 1000 may be configured to give the player 1 the opportunity to provide player input to select one or more candidate characteristics 56 to implement for the associated virtual character. The mobile AR application 1000 may also provide an opportunity to input via a GUI 48.

[0127] In other embodiments, instead of implementing a selection / matching algorithm 1900 to select a virtual character from the virtual character library 400 to have a visual relevance to the particular real-world object, the mobile AR app 1000 may be configured to randomly select a virtual character from the virtual character library 400 to have a relevance to the particular real-world object.

[0128] In yet another embodiment, the mobile AR application 1000 may be configured to modify the character data 37 of a virtual character (randomly or through an implementation of a selection / matching algorithm 1900) from a virtual character library 400 selected to associate with the particular real-world object.

[0129] For example, the mobile AR app 1000 may be configured to modify one or more representations 88 of the virtual character, such as the graphic representation, the numerical representation, the geometric representation, or the mathematical representation. In addition or optionally, the mobile AR app 1000 may be configured to modify the attributes 90 of the virtual character, such as shape data 44, color data 52, image data, position data, material / texture data, physical state data, field of view data, lighting data (e.g., direction, position, color and / or intensity), sound data, motion data, collision data, environment data, timer data and / or other data related to the virtual character.

[0130] In one embodiment, instead of modifying the character data 37 of a predetermined virtual character from the virtual character library 400 selected by the mobile AR application 1000, a copy of the character data 37 may be created and stored in the memory 14, and then the copy of the character data 37 may be modified.

[0131] In another embodiment of this embodiment, one or more predetermined character data 37 of the virtual characters in the virtual character library 400 may be modified such that one of the predetermined characters has visual properties corresponding to the real-world visual properties of the particular real-world object. Thus, one predetermined character data 37 of the plurality of virtual characters may be modified such that one of the plurality of virtual characters has a visual relationship with the particular real-world object.

[0132] For example, a predetermined character data 37 of multiple virtual characters in the virtual character library 400 may be modified so that a predetermined shape data 44 and / or color data 52 of multiple virtual characters in the virtual character library 400 corresponds to the shape data 64 and / or color data 66 of the specific real-world object.

[0133] Next, one embodiment of this embodiment will be described in more detail. This embodiment will be described in relation to the modification of shape data 44, but it should be understood that this embodiment may additionally or selectively include the modification of a predetermined single color data 52 of multiple virtual characters in the virtual character library 400.

[0134] In this example, the CVM1040 may be appropriately trained to detect the specific real-world shape. For example, the CVM1040 may be configured to extract shape data 64 associated with the specific real-world object from the camera data acquired in step 610 of method 600.

[0135] The mobile AR application 1000 may be further configured to select a predetermined one of several virtual characters in the virtual character library 400 to associate with the particular real-world object. In this example, it is assumed that the selection is random. In other examples, the selection may be made by implementing the selection / matching algorithm 1900 described above.

[0136] The mobile AR application 1000 is configured to access shape data 44 stored in memory 14 associated with a predetermined one of several virtual characters in the virtual character library 400. In this example, the mobile AR application 1000 is configured to modify the shape data 44 stored in memory 14 associated with a predetermined one of several virtual characters in the virtual character library 400 so that the shape data 44 is identical to the shape data 64 associated with the particular real-world object.

[0137] Therefore, a predetermined one is selected from among multiple virtual characters in the virtual character library 400, and the character data 37 associated with the predetermined one among the multiple virtual characters is modified so that the predetermined one among the multiple virtual characters is visually associated with the specific real-world object.

[0138] In other embodiments, the mobile AR app 1000 may be configured to "create" the relevant virtual character rather than selecting it from the virtual character library 400.

[0139] In one example, the associated virtual character may be "created" by selecting character attributes from a character attribute database 68. In this example, during game design, the game designer may encode the character attributes from the character attribute database 68 into game data 35. The character attributes may correspond to the character data 37 of one virtual character. Thus, the character attribute database 68 may consist of multiple representations 88, including graphic representations or numerical, geometric, or mathematical representations of the virtual character. The character attribute database 68 may also store multiple attributes 90, such as shape data 44, color data 52, image data, position data, material / texture data, physical state data, field of view data, lighting data (e.g., direction, position, color, and / or intensity), sound data, motion data, collision data, environment data, timer data, and / or other data associated with the virtual character.

[0140] In this example, the mobile AR application 1000 is configured to select one or more attributes from a character attribute database 68 to store as character data 37 in memory 14 associated with the virtual character that is “created” to be associated with the particular real-world object. In this example, the mobile AR application 1000 randomly selects one or more attributes from the character attribute database 68.

[0141] When the mobile AR app 1000 selects one or more attributes from the character attribute database 68 and stores them as character data 37, one associated virtual character is "created" to visually associate with the specific real-world object.

[0142] In some embodiments, the associated virtual character may be “created” to have a visual association with the particular real-world object. In other words, the associated virtual character may have visual properties that correspond to the real-world visual properties of the particular real-world object.

[0143] For example, the character data 37 of the associated virtual character may be selected by the mobile AR application 1000 such that the shape data 44 and / or color data 52 of the associated virtual character correspond to the shape data 64 and / or color data 66 of the specific real-world object, respectively.

[0144] Next, one embodiment of this specification will be described in more detail. This implementation will be described with respect to the determination of the shape data 44 of the associated virtual character "created" by the mobile AR application 1000, but it should be understood that this implementation may additionally or selectively include the determination of the color data 52 of the associated virtual character "created" by the mobile AR application 1000.

[0145] In this example, the CVM 1040 may be appropriately trained to detect the shape of a particular real-world object. For example, the CVM 1040 may be configured to extract shape data 64 associated with the particular real-world object from the camera data acquired in step 610 of method 600. The shape data 64 of the particular real-world object may be stored in memory 14.

[0146] The mobile AR application 1000 is configured to access shape data 64 of a specific real-world object stored in memory 14, and to generate shape data 44 of an associated virtual character such that the shape data 44 is identical to the shape data 64 associated with the specific real-world object. The shape data 44 of the associated virtual character is stored in memory 14.

[0147] When the mobile AR app 1000 generates character data 37, one associated virtual character is "created" to visually associate it with the specific real-world object.

[0148] Furthermore, the mobile AR application 1000 may be configured to maintain records of the specific real-world object and the associated virtual character in memory 14. Thus, as shown in Figures 11 to 13, the various tables stored in memory 14 may include information about the discoverable real-world object set 58, the associated virtual character, and the relationship between the specific real-world object and the associated virtual character.

[0149] Specifically, as shown in Figure 11, the table 1100 stored in memory 14 may include information about a real-world object library 300 that contains information about each real-world object in the discoverable real-world object set 58. Each real-world object in table 1100 may be identified by an object name 50x, as shown in column 1101. Furthermore, real-world objects in the discoverable real-world object set 58 detected by ODM 1010 may be identified by an object ID 1111.

[0150] Table 1100 may include a column 1105 indicating the object family 501x ​​to which each of the real-world objects belongs. The object labels 60x assigned to the real-world objects by CVM 1040 may also be included in a column of Table 1100 (for example, column 1107 in Figure 11). The object labels 60x may be assigned by CVM 1040 as a result of the detection of the given real-world object. For example, the real-world remote control 102 may be part of the "living room" object family 501x ​​and may be assigned the "remote control" object label 60x. Other appropriate forms of classification of the real-world objects may be implemented.

[0151] As will be further explained below, the output of CVM1040 may be configured to include a confidence level 800 indicating the level of certainty that CVM1040 correctly identified the real-world object. The confidence level 800 may be stored in memory 14, and this information may be included in column 1113 of table 1100.

[0152] Furthermore, as will be described later, the composition of the discoverable real-world object set 58 may change over time. Therefore, table 1100 may include an indication of whether a real-world object is currently part of the discoverable real-world object set 58. For example, this information may be stored in column 1103 of table 1100.

[0153] Table 1100 may also include an indication of whether a real-world object in the real-world object library 300 has been detected, and / or whether the associated virtual character was emitted from or generated by that real-world object. Furthermore, Table 1100 may include information regarding the status of each of several real-world objects in the real-world object library 300 (for example, whether they have been detected, not detected, or are undetectable because they are not part of the detectable real-world object set 58).

[0154] With respect to Figure 12, the table 1200 stored in memory 14 may contain information about each virtual character in the virtual character library 400. Each of the multiple virtual characters in table 1200 may be identified by a name / object ID 1201.

[0155] As described above, in some embodiments, the mobile AR application 1000 may be configured to classify each virtual character into a virtual character family 700x ("virtual character family"). Table 1200 may show the virtual character family 700x to which each virtual character belongs. This information may be included in Table 1200, for example, in column 1203.

[0156] Furthermore, the table 1200 may include information related to the candidate real-world object labels 42x, character data 37, and candidate characteristics 58 mentioned above. This information may be included in columns 1205, 1207, and 1209 of the table 1200, respectively.

[0157] Table 1200 may also include a display of the status of each virtual character, for example, if the virtual character was generated from / created by the specific real-world object, or if it is considered locked by the mobile AR app 1000 and therefore cannot / is not generated from the specific real-world object (as shown in column 1207 of Table 1200 in Figure 12).

[0158] As shown in Figure 13, the table 1300 stored in memory 14 may include information about the connection or association between the particular real-world object and the associated virtual character. The table 1300 may include a representation of the particular real-world object and the associated virtual character generated from / by the particular real-world object. For example, the table 1300 may associate the object IDs of the real-world object and the virtual object (as shown in columns 1301 and 1303 of the table 1300 in Figure 13). The table 1300 may also include additional information such as the object label 60x of the real-world object and the object family 501x ​​to which the real-world object belongs (as shown in columns 1305 and 1307 of the table 1300 in Figure 13).

[0159] As will be further described below, the mobile AR application 1000 may be configured to attribute point values ​​to the specific real-world object and / or the associated virtual character. Information regarding the point values ​​may be stored in table 1300 (for example, in column 1311).

[0160] Table 1300 may also include a display of the player who detected the real-world object (column 1313) and the time at which the associated virtual character was generated in the AR game environment 1005 (column 1315).

[0161] Table 1300 also shows the spatial coordinates (XYZ) of the specific real-world object. RW And, when the related virtual character is generated, t=t gen The display may include the player's spatial coordinates (XYZ) 1. The spatial coordinates (XYZ) of the specific real-world object. RW This may be stored in memory 14 as part of the real-world object data 62. Also, table 1300 is used when the real-world object is detected t=t det The display may include the distance between player 1 and the specific real-world object.

[0162] Also, the table 1300 includes the current spatial coordinates (XYZ) of the specific real-world object RW display and the current spatial coordinates (XYZ) of the related virtual character emitted by / generated by the specific real-world object VC may also be included.

[0163] The mobile AR app 1000 may also be configured to generate a visual effect corresponding to the fact that the related virtual character is emitted by / generated by the specific real-world object. Non-limiting examples of visual effects include "virtual flash" indicating that the related virtual character is about to be emitted by / generated by the specific real-world object. Such virtual flash includes a computer-generated image (CGI) element 1 displayed on the display 11 near the specific real-world object, an auditory signal and / or a tactile signal generated on the mobile terminal 10, or other signals suitable for indicating to the player 1 that the related virtual character is about to be emitted by / generated by the specific real-world object, but is not limited thereto.

[0164] The mobile AR app 1000 may be configured to represent the related virtual character by a CGI element 1 that appears together with a motion line in order to provide an effect for the purpose of indicating to the player 1 that the related virtual character is emitted by / generated by the specific real-world object. The related virtual character may be released from the anchor 46 created in step 640. In some embodiments, this effect may be further enhanced by an effect of increasing the size of the related virtual character as the distance between the related virtual character and the specific real-world object increases. In some embodiments, other effects can also be added, including but not limited to changing the "facial expression" of the related virtual character, such as smiling, opening or closing the eyes.

[0165] In the illustrated exemplary embodiment, as shown in Figure 1B, the mobile AR app 1000 may be configured to generate one virtual flourish 103 (e.g., a “sparkle”) indicating that the virtual character 104 is about to be emitted from the real-world remote control 102. As shown in Figure 1C, the virtual character 104 appears with a motion line 105, providing an effect intended to indicate to the player 1 that the virtual character 104 is “generated from” the real-world remote control 102. Also, between Figure 1C and Figure 1D, the size of the virtual character 104 increases as the distance between the virtual character 104 and the real-world remote control 102 increases. Furthermore, the virtual character 104 is depicted with “eyes” that are shown changing from a closed state to an open state between images 100C and 100D of Figure 1C and Figure 1D. Finally, Figure 1D shows the virtual character 104 fully emitted from the real-world remote control 102.

[0166] Furthermore, other effects can be added with respect to the specific real-world object and the associated virtual character in other embodiments described later.

[0167] For example, an effect may be added that conveys how the associated virtual character is emitted from / generated from the particular real-world object. For example, the mobile AR app 1000 may be configured to create an effect by making it appear as if the associated virtual character is "pulling out" of an opening in the particular real-world object. In yet another example, a resistance effect may be created such that player input may be required to "pull" the associated virtual character from the particular real-world object. In other embodiments, any or a combination of the aforementioned effects can be further enhanced by an effect in which the associated virtual character is stretched as it is gradually "pulled out" from the particular real-world object and then returns to its normal shape. As an additional effect, the particular real-world object may be transformed into the associated virtual character. For example, the particular real-world object may appear as if it is folded or unfolded into the associated virtual character.

[0168] In other embodiments, the mobile AR application 1000 may be configured to display the associated information on the display 11 of the mobile terminal 10 when the associated virtual character is generated or after it has been emitted / generated from the particular real-world object. Such information may include, but is not limited to, the name of the associated virtual character, the point value of the associated virtual character, or the name of the "virtual character collection" to which the virtual character belongs (described in more detail below).

[0169] To emphasize the appearance of the effects described above, the mobile AR app 1000 may be configured to modify the image presented on the display 11 to appear blurred or darker while the associated virtual character is being emitted from / generated by the particular real-world object.

[0170] When the associated virtual character is emitted from / generated from the particular real-world object, the anchor 46 roughly positioned at the location of the particular real-world object may be removed. Next is step 660.

[0171] Step 66 Medicine In some embodiments, the mobile AR application 1000 may be configured such that the associated virtual character is emitted from / generated from a particular real-world object to further advance the player's gameplay.

[0172] For example, the associated virtual character may be considered "collected," the mobile AR app 1000 may increase the player's score, or add the associated virtual character to a "virtual character collection" associated with player 1. As shown in Figure 5, the virtual character collection 900 may be a data container in which various aspects of gameplay are recorded and may be presented to player 1, for example, on a "virtual character collection screen" 500.

[0173] In some embodiments, the mobile AR application 1000 may be configured to provide the player 1 with an opportunity to provide input for accessing the virtual character collection screen 500. For example, the input for accessing the virtual character collection screen 500 may be provided by the player 1 via the GUI 48. In response to the player's input, the mobile AR application 1000 may be configured to stop displaying the image of the scene captured and presented by the mobile AR application 1000 on the display 11 of the mobile terminal 10 and start displaying the virtual character collection screen 500. Similarly, when the virtual character collection screen 500 is displayed on the display 11 of the mobile terminal 10, the player 1 may provide player input via the GUI 48 to stop displaying the virtual character collection screen 500 and start displaying the image of the scene captured and presented by the mobile AR application 1000 on the display 11.

[0174] The collected virtual characters may be highlighted in some way, or displayed in bold (for example, as shown in Figure 5). Other methods may be used to inform Player 1 that a particular virtual character has been collected.

[0175] During a game session, the mobile AR app 1000 may also be configured to maintain a record of all real-world objects associated with a virtual character. The mobile AR app 1000 may also be configured to maintain a record of real-world objects that have not been detected during the game session from the set of detectable real-world objects 58. The mobile AR app 1000 may also be configured to maintain a record of multiple virtual characters that could be associated with the undetected real-world objects in the set of detectable real-world objects 58 during the game session. Any combination of similar information may be displayed on the virtual character collection screen 500 and presented to the player 1.

[0176] The mobile AR application 1000 may also be configured to present each virtual character family 700x to the player 1 through a virtual character collection screen 500. In particular, the virtual character collection screen 500 may present information related to the player's virtual character collection 900, and may further present virtual characters 71x, 72x, 73x, 74x, and 75x of object families 501x ​​(for example, virtual character 71x of object family 5011 and virtual character 72x of object family 72x), which may be related to physical locations in the real world, as described above.

[0177] In some embodiments, gameplay can be further advanced by unlocking additional virtual character families 700x. For example, unlocking an additional virtual character family 700x may be a result of player 1 collecting a predetermined number of virtual characters and / or player 1 collecting all virtual characters within a predetermined virtual character family 700x. In another example, unlocking an additional virtual character family 700x may also be a result of detecting a particular combination of real-world objects. For example, the particular combination of real-world objects may be identified as a particular combination of object labels 60x (assigned to the real-world objects), or a threshold of some real-world objects in one or more object families 501x ​​may be required for progression.

[0178] Here, returning to the situation in which the associated virtual character was emitted / generated from the particular real-world object (see step 650), the associated virtual character may be added to the virtual character collection 900. This may be done autonomously by the mobile AR app 1000 without player input, for example, immediately following its emission / generation or after a predetermined time delay. In other embodiments, input from player 1 may be required for the associated virtual character to be added to the virtual character collection 900. For example, player 1 may be required to touch the associated virtual character on the display 11 / touchscreen 16, or to touch the particular real-world object associated with the associated virtual character (which would be detected by the camera), or to touch another part of the display 11 / touchscreen 16.

[0179] In some embodiments, real-world objects may be designated as real-world representations of the virtual character collection 900. For example, real-world objects from an object family 501x ​​classified as "storage" (e.g., drawers, cabinets, boxes, etc.) may be associated with the virtual character collection 900. Thus, the virtual character collection screen 500 that constitutes the virtual character collection 900 may appear to be emitted from / generated by the real-world object. Furthermore, when the mobile AR application 1000 collects virtual characters, it may be configured to have the effect of representing the collected virtual characters as moving toward and / or into the real-world object when adding them to the virtual character collection 900.

[0180] In other embodiments, a point-based system can be used to further advance the player's gameplay. For example, player 1 may be required to accumulate points to unlock further virtual character families 700x once a point threshold is met.

[0181] Now, returning to the situation where the associated virtual character was emitted from / generated from the particular real-world object (see step 650), the mobile AR app 1000 may be configured to attribute point values ​​to the particular real-world object and / or the associated virtual character. In some embodiments, the point values ​​may be attributed based on the density of real-world objects in the scene. For example, if the particular real-world object is located in an area with a higher density of detectable real-world objects (in terms of the number of detectable real-world objects per unit area or per unit volume exceeding a predetermined threshold), the point value to which it belongs may be lower than if the particular real-world object is located in an area with a lower density of detectable real-world objects.

[0182] In other embodiments, the point value may depend on the physical distance between player 1 and a particular real-world object relative to other detectable real-world objects in the scene. For example, given two detectable real-world objects at different physical distances from the player, a lower point value would be attributed to the associated virtual character of 1 emitted / generated from the closer of the two detectable real-world objects. This means that player 1 may move physically (in the real world) to obtain a higher score.

[0183] In other embodiments, the point value belonging to the associated virtual character emitted from / generated by the particular real-world object may change as a function of time. For example, the point value may decrease as a function of the time required for player 1 to find / detect the particular real-world object. The duration of time may be calculated from the startup time of the mobile AR app 1000 or from a set amount of gameplay time.

[0184] In another example, if there are multiple detectable real-world objects detected in the camera image, the point values ​​of virtual characters emitted from / generated by each of the real-world objects may be based on the count of a specific object label 60x assigned to each of the multiple real-world objects. For example, given three detectable real-world objects assigned object label 601 and one detectable real-world object assigned object label 602, the point values ​​associated with the detectable real-world object assigned object label 602 may be higher than the point values ​​associated with the detectable real-world object assigned object label 601.

[0185] Those skilled in the art will understand that the method for generating the virtual character 104 described above and the effects described with respect to the virtual character (i.e., steps 610 to 660) can be applied to the virtual characters 204, 304, and 404 in Figures 2A to 2B, 3A to 3E, and 4A to 4C. As shown in Figures 1A to 1D, 2A to 2B, 3A to 3E, and 4A to 4C, each virtual character (or virtual character) 104, 204, 304, and 404 can be emitted from and associated with different real-world objects 102, 202, 302, and 402, respectively.

[0186] For example, Figure 2A shows an image 200A of a scene that may be captured and presented on the display 11 of a mobile device 10 by a mobile AR application 1000. Image 200A is shown to include a real-world cabinet 201 and a real-world spatula 202.

[0187] In the illustrated exemplary embodiment, the camera image includes a real-world cabinet 201 and a real-world spatula 202. For this example, we assume that only the real-world remote spatula 202 is included in the detectable real-world object set 58.

[0188] In step 610, the ODM 1010 captures an image from the camera feed (i.e., a video frame) of the mobile device 10. In step 620, the mobile AR app 1000 is configured to process the received camera image, and finally, in step 630, the real-world spatula 202 is detected in the scene as a result of a positive output of the CVM 1040. In step 640, the AR API 1020 of the mobile AR app 1000 is configured to provide an anchor 46 roughly established at the location of the real-world spatula 202, and to provide an anchor for the final virtual character 204 associated with the real-world spatula 202.

[0189] In step 650, the mobile AR app 1000 creates an association between the real-world spatula 202 and the virtual character 204. In this example, the associated virtual character 204 is selected to have a visual association with the real-world spatula 202 such that the shape of the virtual character 204 corresponds to the shape of the real-world spatula 202. The virtual character 204 may be selected from the virtual character library 400 by a selection / matching algorithm 1900 performed by at least one processor 12, or it may be "created" by the mobile AR app 1000 as described above.

[0190] A visual effect corresponding to the virtual character 204 being emitted from / generated from the real-world spatula 202, namely a virtual flourish 203, is shown in Figure 2A. The virtual character 204 is displayed along with motion lines 205 to provide an effect intended to show player 1 that the virtual character 204 is being "generated" from the real-world spatula 202.

[0191] As shown in the exemplary embodiment presented in Figure 2B, attributes associated with the virtual character 204 are displayed on the display 11 of the mobile terminal 10 after it has been emitted from / generated by the real-world spatula 202. In this embodiment, the mobile AR app 1000 is configured to display the name / identifier of the virtual character 204 (i.e., “Spatch” in Figure 2B), the point value of the virtual character 204 (i.e., “500pt” as shown in Figure 2B), and the name of the “virtual character collection” to which the virtual character 204 belongs (i.e., “Kitchen” as shown in Figure 2B).

[0192] When virtual character 204 is emitted from / generated by real-world spatula 202, virtual character 204 may be added to virtual character collection 900.

[0193] In another example, Figure 3A shows an image 300A of a different scene that may be captured and presented on the display 11 of a mobile device 10 by a mobile AR application 1000. Image 300A is shown to include a real-world cabinet 301 and a real-world stapler 302.

[0194] In the illustrated exemplary embodiment, the camera image includes a real-world cabinet 301 and a real-world stapler 302. For this example, it is assumed that both the real-world stapler 302 and the real-world cabinet 301 are included in the detectable real-world object set 58.

[0195] In step 610, the ODM 1010 captures an image from the camera feed (i.e., a video frame) of the mobile device 10. In step 620, the mobile AR app 1000 is configured to process the received camera image, and finally, in step 630, the real-world stapler 302 is detected in the scene as a result of a positive output of the CVM 1040. In step 640, the AR API 1020 of the mobile AR app 1000 is configured to provide an anchor 46 roughly established at the location of the real-world stapler 302 and to provide an anchor for the final virtual character 304 associated with the real-world stapler 302.

[0196] In step 650, the mobile AR app 1000 creates an association between the real-world stapler 302 and the virtual character 304. In this example, the associated virtual character 304 is selected to have a visual association with the real-world stapler 302 such that the shape of the virtual character 304 corresponds to the shape of the real-world stapler 302. The virtual character 304 may be selected from the virtual character library 400 by a selection / matching algorithm 1900 performed by at least one processor 12, or it may be "created" by the mobile AR app 1000 as described above.

[0197] A visual effect corresponding to the virtual character 304 being emitted from / generated by the real-world stapler 302, namely a virtual flourish 303, is shown in Figure 3A. The virtual character 304 is displayed along with motion lines 305 to provide an effect intended to show player 1 that the virtual character 304 is being "generated" from the real-world stapler 302.

[0198] Furthermore, in this embodiment, in order to enhance the appearance of the virtual flourish 303 and motion line 305, the mobile AR app 1000 is configured to darken the image presented on the display 11 while the virtual character 304 is being emitted from / generated by the real-world stapler 302.

[0199] When virtual character 304 is emitted from / generated by real-world stapler 302, virtual character 304 may be added to virtual character collection 900. Figure 3E shows a virtual character collection screen 500 displaying virtual character collection 900. The virtual character collection screen 500 displays virtual character 304, which is part of object family 5015. Virtual character 304 is virtual character 304, which is part of virtual character family 7005.

[0200] In the embodiment shown in Figure 3B, one real-world object is associated with one virtual object representing the virtual character collection 900. Thus, in this embodiment, the real-world cabinet 301 is configured to have an association with one virtual object representing the virtual character collection 900. Therefore, as shown in Figure 3C, the virtual character collection screen 500 that constitutes the virtual character collection 900 is displayed as if it were generated by being emitted from the real-world cabinet 301, as will be described later.

[0201] Therefore, in step 610, the ODM 1010 captures an image from the camera feed (i.e., a video frame) of the mobile terminal 10. In step 620, the mobile AR app 1000 is configured to process the received camera image, and finally, in step 630, the real-world cabinet 301 is detected in the scene as a result of a positive output of the CVM 1040. Before proceeding to step 640, the mobile AR app 1000 may be configured to determine whether the object label 60x assigned to the real-world cabinet 301 corresponds to the object label 60x assigned to the real-world object portion of the discoverable real-world object set 58 classified as a candidate object family (e.g., the object family 501x ​​classified as "storage"). If affirmative, the method proceeds to step 640, in which the AR API 1020 of the mobile AR app 1000 is configured to provide an anchor 46 that is roughly established at the location of the real-world cabinet 301 in order to provide an anchor for the final virtual object 804 representing the virtual character collection 900.

[0202] In step 650, the mobile AR app 1000 creates a record in memory 14 that associates the real-world cabinet 301 with a virtual object 804 representing the virtual character collection 900, as shown in Figure 3C.

[0203] A visual effect corresponding to the virtual object 804 being emitted from / generated from the real-world cabinet 301, namely a virtual flourish 803, is shown in Figure 3A. The virtual character 803 is displayed along with a motion line 805 to provide an effect intended to indicate to player 1 that the virtual object 804 is being "generated" from the real-world cabinet 301.

[0204] The mobile AR application 1000 may be configured to display the virtual character collection screen 500 to the player 1 and provide the player 1 with an opportunity to make an activation input via the virtual object 804 in order to display the virtual character collection 900. For example, the player 1 can make an activation input by touching the virtual object 804, as shown in Figure 3D. Upon detecting an activation input from the player 1, the mobile AR application 1000 may be configured to display the virtual character collection screen 500 and present the virtual character collection 900 to the player 1, as shown in Figure 3E.

[0205] In yet another example, Figure 4A shows an image 400A of a scene that may be captured and presented on the display 11 of a mobile device 10 by a mobile AR application 1000. Image 400A is shown to include a real-world cabinet 401 and a real-world toothbrush 402.

[0206] In the illustrated exemplary embodiment, the camera image includes a real-world cabinet 401 and a real-world toothbrush 402. For this example, only the real-world toothbrush 402 is included in the detectable real-world object set 58.

[0207] In step 610, the ODM 1010 captures an image from the camera feed (i.e., a video frame) of the mobile device 10. In step 620, the mobile AR app 1000 is configured to process the received camera image, and finally, in step 630, the real-world toothbrush 402 is detected in the scene as a result of a positive output of the CVM 1040. In step 640, the AR API 1020 of the mobile AR app 1000 is configured to provide an anchor for the final virtual character 404 associated with the real-world toothbrush 402, by providing an anchor 46 that is roughly established at the location of the real-world toothbrush 402.

[0208] In step 650, the mobile AR app 1000 creates an association between the real-world toothbrush 402 and the virtual character 404. In this example, the associated virtual character 404 is selected to have a visual association with the real-world toothbrush 402 such that the shape of the virtual character 404 corresponds to the shape of the real-world toothbrush 402. The virtual character 404 may be selected from the virtual character library 400 by a selection / matching algorithm 1900 performed by at least one processor 12, or it may be "created" by the mobile AR app 1000 as described above.

[0209] A visual effect corresponding to the virtual character 404 being emitted from / generated from the real-world toothbrush 402, namely a virtual flourish 403, is shown in Figure 4A. The virtual character 404 is displayed along with motion lines 405 to provide an effect intended to show player 1 that the virtual character 404 is "generated" from the real-world toothbrush 402.

[0210] As shown in Figure 4C, even if the real-world toothbrush 402 is no longer in the field of view of the camera 26 and as a result is no longer displayed on the display 11, the virtual character 404 may remain in the image of the scene displayed on the display 11. In such a case, the spatial coordinates (XYZ) of the real-world toothbrush 402 are RW and the spatial coordinates (XYZ) of virtual character 404 VC These may be different. When virtual character 404 is emitted from / generated by real-world toothbrush 402, virtual character 404 may be added to virtual character collection 900.

[0211] In some embodiments, in addition to the first virtual character being emitted from / generated from a first real-world object, a second real-world object may be detected by the CVM 1040 of the mobile AR application 1000. When the second real-world object is detected, the second virtual character may be emitted from / generated from the second real-world object. In this way, the mobile AR application 1000 creates an association between the first real-world object and the first virtual character, and also creates another association between the second real-world object and the second virtual character. It is also possible that the second real-world object is detected before the first virtual character is emitted from / generated from the first real-world object, resulting in two simultaneous generation operations.

[0212] Related real-world objects and related virtual characters In other embodiments, in step 630 of method 600, a plurality of related real-world objects may be detected by the CVM 1040. The plurality of related real-world objects may be related by a visible relationship in the real world. For example, the plurality of real-world objects may be in physical contact in the real world (e.g., physically connected). For example, the plurality of real-world objects may be visibly related in the real world. In such embodiments, the mobile AR application 1000 may be configured to create relationships between the plurality of related real-world objects and a plurality of related virtual characters, each virtual character being associated with its respective real-world object.

[0213] In one example of the implementation of this embodiment, Figure 17A shows an image 1700A of a scene that may be captured and presented on the display 11 of a mobile terminal 10 by a mobile AR application 1000. Image 1700A is shown to include a real-world straw 902 and a table 1801 located inside a real-world cup 1702. For this embodiment, it is assumed that only the real-world straw 902 and the real-world cup 1702 are included in the detectable real-world object set 58.

[0214] In step 610, the ODM 1010 captures an image from the camera feed (i.e., a video frame) of the mobile device 10. In step 620, the mobile AR app 1000 is configured to process the received camera image, and finally, in step 630, the real-world straw 902 is detected in the scene as a result of a positive output from the CVM 1040.

[0215] While step 630 continues to be executed, the CVM 1040 similarly detects a second real-world object, namely the real-world cup 1702. Thus, step 630 may include the detection of multiple real-world objects.

[0216] In this example, when detecting the real-world cup 1702, in step 640, the AR API 1020 of the mobile AR app 1000 is configured to provide an anchor 461 that provides an anchor for the final virtual character 904 associated with the real-world straw 902, and an anchor 462 that provides an anchor for the final virtual character 1704 associated with the real-world cup 1702.

[0217] In step 650, the mobile AR app 1000 creates associations between the real-world straw 902 and the virtual character 904, and between the real-world cup 1702 and the virtual character 1704. As shown in Figure 17B, the shape of the virtual character 904 corresponds to the shape of the real-world straw 902, and the shape of the virtual character 1704 corresponds to the shape of the real-world cup 1702. That is, the virtual character 904 is straw-shaped, and the virtual character 1704 is cup-shaped. The virtual characters 904 and 1704 may be selected from the virtual character library 400 by a selection / matching algorithm 1900 performed by at least one processor 12, or they may be "created" by the mobile AR app 1000 as described above.

[0218] In the illustrated exemplary embodiment, as shown in Figure 17B, the mobile AR app 1000 may be configured to generate a virtual flourish 903 (e.g., "sparkle") indicating that the virtual character 904 is about to be released from the real-world straw 902 and a virtual flourish 1703 indicating that the virtual character 1004 is about to be released from the real-world cup 1702. As shown in Figure 17B, each of the virtual characters 904, 1704 can appear with motion lines 905, 1705, and perform a performance intended to show player 1 that the virtual characters 904, 1704 are being "generated" from the real-world remote straw 902 and the real-world cup 1702. Finally, Figure 17C shows that the virtual characters 904, 1704 have been fully released from the real-world remote straw 902 and the real-world cup 1702.

[0219] When associated virtual characters 904, 1704 are emitted from / generated by specific real-world objects 902, 1702, associated virtual characters 904, 1704 may be added to the virtual character collection 900.

[0220] A single real-world object and multiple virtual characters In some embodiments, the mobile AR app 1000 may be configured to create an association between a single real-world object and a single virtual character, while in other embodiments, the mobile AR app 1000 may be configured to create an association between a single real-world object and multiple virtual characters.

[0221] In a non-limiting example, Figure 18A shows an image 1800A of a scene that may be captured and presented on the display 11 of a mobile device 10 by a mobile AR application 1000. Image 1800A is shown to include a bunch of real-world grapes 1802 and a table 1701. For this embodiment, we assume that only the bunch of real-world grapes 1802 is included in the detectable real-world object set 58.

[0222] In step 610, the ODM 1010 captures an image from the camera feed (i.e., a video frame) of the mobile device 10. In step 620, the mobile AR app 1000 is configured to process the received camera image, and finally, in step 630, the real-world grape cluster 1802 is detected in the scene as a result of a positive output from the CVM 1040. In this embodiment, a well-trained CVM 1040 may detect that the real-world grape cluster 1802 consists of individual real-world grapes 1902.

[0223] In step 640, the AR API 1020 of the mobile AR app 1000 is configured to provide multiple anchors 46x roughly located at the positions of the real-world grape cluster 1802, and ultimately to provide anchors to multiple virtual characters 1804x associated with the real-world grape cluster 1802.

[0224] In step 650, the mobile AR app 1000 creates associations between the real-world grape cluster 1802 and the multiple virtual characters 1804x. The mobile AR app 1000 also creates associations between the individual real-world grapes 1902 and each of the multiple virtual characters 1804x. The virtual characters 1804x may be selected from the virtual character library 400 by a selection / matching algorithm 1900 performed by at least one processor 12, or they may be "created" by the mobile AR app 1000 as described above.

[0225] In the illustrated exemplary embodiment, as shown in Figure 17B, the mobile AR app 1000 may be configured to generate a virtual flourish 1803 (e.g., "sparkle") indicating that the virtual character 1804x is about to be released from a bunch of real-world grapes 1802 / individual grapes 1902. Also, as shown in Figure 18B, each of the virtual characters 1804x appears with a motion line 1805, providing a performance intended to show player 1 that the virtual character 1804x is being "generated" from a bunch of real-world grapes 1802 / individual grapes 1902. Finally, Figure 18C shows the virtual character 1804x fully released from a bunch of real-world grapes 1802 / individual grapes 1902.

[0226] Once the associated virtual character 1804x is generated by being released from / a real-world bunch of grapes 1802 / individual grapes 1902, the associated virtual character 1804x may be added to the virtual character collection 900.

[0227] Detectable real-world objects The size of the detectable real-world object set 58 may be quantified as the total number of real-world objects in the detectable real-world object set 58 at a predetermined time during a game session. In some embodiments, the size of the detectable real-world object set 58 may change (e.g., increase or decrease) as the game session progresses. For example, the size of the detectable real-world object set 58 may change when a particular combination of real-world objects is detected in the image data. The combination of real-world objects may be characterized as a particular combination of object labels 60x assigned to the real-world objects by the CVM 1040. For example, the mobile AR application 1000 may be configured to increase the size of the detectable real-world object set 58 when a threshold of real-world objects in a particular object family 501x ​​is detected. According to a non-limiting example of this embodiment, the size of the detectable real-world object set 58 may increase when it is detected that the number of real-world objects in a particular object family 501x ​​(e.g., kitchen object family 5011) meets or exceeds a threshold of real-world objects encoded in the game data 35.

[0228] In some embodiments, the configuration of the discoverable real-world object set 58 may change over time (i.e., the specific real-world objects within the discoverable real-world object set 58 may change over time).

[0229] In some embodiments, the real-world object may be discoverable only for a limited period of time. For example, the real-world object may be part of a discoverable real-world object set 58 for a limited period of time. Thus, the mobile AR app 1000 may be configured to maintain membership in the discoverable real-world object set 58 based on a timer for each real-world object. In this way, the mobile AR app 1000 may be configured to assign an expiration date to each real-world object, monitor the expiration date, and remove the real-world object from the discoverable real-world object set 58 when the expiration date is reached or exceeded. Similarly, the mobile AR app 1000 may be configured to remove a real-world object from the discoverable real-world object set 58 after emitting / generating a virtual character.

[0230] In some embodiments, the expiration date assigned to a real-world object may change during a game session. For example, the expiration date of a real-world object may increase or decrease as a function of the object label 60x assigned to the real-world object and the total number of real-world objects detected and assigned to this same object label 60x. For example, a variable may be stored in memory 14 to track the number of objects detected and assigned a particular object label 60x. The mobile AR app 1000 may be configured to change the expiration date assigned to this particular object label 60x based on the value of the variable. For example, if a threshold is met or exceeded for a real-world object assigned a particular object label 60x, the mobile AR app 1000 may be configured to shorten the expiration date assigned to this particular object label 60x. Thus, the period during which an object assigned a particular object label 60x can be part of a discoverable set of real-world objects 58 may change during a game session.

[0231] Therefore, the mobile AR app 1000 may be configured to change the expiration date of a real-world object in the discoverable real-world object set 58 based on the object label 60x assigned to the real-world object when a real-world object is detected in step 630.

[0232] In some embodiments, a virtual character may not be emitted from / generated from the real-world object unless a confidence threshold 800x is met or exceeded in the object detection step (step 630 of method 600). As described above, confidence 800 represents the level of certainty that the CVM 1040 has identified the object. Specifically, confidence 800 represents the level of certainty that the object label 60x assigned by the CVM 1040 actually corresponds to the particular real-world object detected. The real-world object data 62 for each real-world object includes the object label 60x (e.g., "remote control", "spatula") and confidence 800 (e.g., 0-100%).

[0233] In this example, the generation of a virtual character in the AR game environment 1005 occurs when a confidence threshold 800x is met or exceeds it. Therefore, the mobile AR application 1000 is configured to identify real-world objects in the captured image by processing the image data with the CVM 1040. The mobile AR application 1000 is configured to determine whether the confidence level 800 belonging to a given real-world object exceeds a predetermined threshold 800x (e.g., 65%). If the confidence level 800 of a given real-world object is below the confidence threshold 800x, the method does not proceed to step 640, and the virtual character is not generated. However, if it is determined that the confidence level 800 of a particular real-world object exceeds the predetermined confidence threshold 800x, the virtual character may be generated.

[0234] Detection event In some embodiments, after the associated virtual character is emitted from / generated from a particular real-world object, that particular real-world object may then be included in an image from the camera feed (i.e., video frame) of the mobile terminal 10, so that the particular real-world object may be detected a second time by the CVM 1040.

[0235] In one implementation example of this embodiment, when a specific real-world object is detected in the image for the second time, the mobile AR application 1000 is configured to maintain the association between the associated virtual character and the specific real-world object without generating a new virtual character associated with the specific real-world object. In this way, no second virtual character associated with the specific real-world object is generated, and the mobile AR application 1000 is configured to maintain the association between the associated virtual character and the specific real-world object as long as the associated virtual character continues to be displayed on the display 11 (or even if the associated virtual character is no longer displayed on the display 11).

[0236] Therefore, before proceeding to step 640, the method may additionally include a step of accessing a record stored in memory 14 with respect to determining whether the associated virtual character was emitted from / generated by the real-world object detected in step 630. If affirmative, in this embodiment the method does not proceed to step 640. If negative, the method may proceed to step 640.

[0237] In another embodiment of this design, when a specific real-world object in the image is detected a second time (possibly after a threshold time has elapsed), the mobile AR application 1000 may be configured to create an association between the specific real-world object and a second virtual character. Thus, information regarding a first association between the specific real-world object and a first associated virtual character, and information regarding a second association between the specific real-world object and a second associated virtual character, will be contained in a table 1300 stored in the memory 14 of the mobile terminal 10.

[0238] In some embodiments, the mobile AR application 1000 is configured to maintain an association between the first related virtual character and the specific real-world object, and to generate a second related virtual character as the first related virtual character continues to be displayed on the display 11. In other embodiments, the mobile AR application 1000 is configured to maintain an association between the first related virtual character and the specific real-world object, and to generate a second related virtual character when the first related virtual character is no longer displayed on the display 11.

[0239] In some embodiments, the second related virtual character may have the same point value as the first related virtual character. In other embodiments, the second related virtual character may have a different point value from the first related virtual character.

[0240] In some embodiments, after an associated virtual character 1 is emitted from / generated from a first real-world object 1 that has been assigned a specific object label 60x of 1 by the CVM 1040, a second real-world object 1 may appear in an image of 1 from the camera feed (i.e., video frame) of the mobile terminal 10 and be assigned the same specific object label 60x that was assigned to the first real-world object.

[0241] For example, considering one first real-world object and one second real-world object detected by the CVM 1040 and assigned a specific object label 601, the mobile AR app 1000 may be configured not to create an association between one new virtual character and the second real-world object. In this way, the mobile AR app 1000 is configured to maintain an association between the associated virtual character and the first real-world object without generating a new virtual character that has an association with the second real-world object. This association is maintained whether the associated virtual character continues to be displayed on the display 11 or whether the associated virtual character is no longer displayed on the display 11.

[0242] Therefore, before proceeding to step 640, the method may further include accessing a record stored in memory 14 with respect to determining whether the associated virtual character 1 was emitted from / generated by the real-world object to which the object label 601 assigned to the second real-world object was assigned. If positive, the method does not proceed to step 640. If negative, the method proceeds to step 640.

[0243] In another embodiment of this design, the mobile AR application 1000 may be configured to create an association between a new virtual character and a second real-world object to which an object label 601 has been assigned.

[0244] Thus, the mobile AR application 1000 is configured to maintain an association between the associated virtual character and the first real-world object, and to generate a new virtual character that has an association with the second real-world object. This association is maintained whether the associated virtual character continues to be displayed on the display 11 or whether the associated virtual character is not displayed on the display 11.

[0245] In some embodiments, the new virtual character may have the same point value as the first related virtual character. In other embodiments, the new virtual character may have a different point value from the first related virtual character.

[0246] In some embodiments, the mobile AR app 1000 may be configured to limit the number of times a particular real-world object can be detected. In other embodiments, the mobile AR app 1000 may be configured to limit the number of virtual characters associated with a particular real-world object. In yet another embodiment, the mobile AR app 1000 may be configured to limit the number of virtual characters associated with a real-world object to which a particular object label 60x has been assigned.

[0247] Therefore, the mobile AR app 1000 may be configured to store variables in memory 14 to track these values. For example, the mobile AR app 1000 may store a variable in memory 14 to track the number of times a particular real-world object 1 has been detected. For another example, the mobile AR app 1000 may store a variable in memory 14 to track the number of virtual characters associated with a particular real-world object 1. For yet another example, the mobile AR app 1000 may store a variable in memory 14 to track the number of real-world objects to which a particular object label 60x has been associated. The mobile AR app 1000 may be configured to increment these variables during a game session when a real-world object is detected and / or when an association is made between a real-world object and a virtual character.

[0248] Before proceeding to step 640, the method may include a step of determining, based on the current value of the variable, whether a threshold limit has been met or exceeded. If the threshold limit has been met or exceeded, the method does not need to proceed to step 640. If the threshold limit has not been met, the method may proceed to step 640.

[0249] In some embodiments, the ODM 1010 may capture one image from the camera feed of the mobile terminal 10 in which multiple real-world objects from a set of detectable real-world objects 58 may be present simultaneously. In this way, when processing the camera image, the mobile AR application 1000 can ultimately detect multiple real-world objects.

[0250] In one example, upon detection of multiple real-world objects, the mobile AR app 1000 may be configured to give player 1 the opportunity to select a particular one of the multiple real-world objects such that one virtual character is emitted / generated from that real-world object. For example, player 1 may make the selection by touching a portion of the display 11 / touchscreen 16 that displays the particular (i.e., the real-world object to be selected) object. In another example, player 1 may make the selection by touching another portion of the display 11 / touchscreen 16 or through another suitable mechanism.

[0251] In another example, upon detecting the multiple real-world objects, the mobile AR app 1000 may be configured to randomly select one of the multiple real-world objects as the object from which a virtual character is emitted / generated.

[0252] In yet another example, when multiple real-world objects are detected, the mobile AR app 1000 may be configured to execute a real-world object selection algorithm 1400 to select a specific one of the multiple real-world objects as the real-world object from which the virtual character is emitted / generated.

[0253] Reference is made to the flowchart of FIG. 14 showing the steps of the real-world object selection algorithm 1400 executed by the mobile AR app 1000. In step 1410, each of the plurality of real-world objects is ranked based on one or more criteria.

[0254] · Counting the number of occurrences of a specific object label 60x The mobile AR app 1000 may be configured to store in the memory 14 the number of occurrences of a specific object label 60x assigned to one real-world object during a game session. The mobile AR app 1000 may be configured to access the memory 14 to determine the count of the occurrence of a specific object label 60x, e.g., object label 601, and further to assign a ranking to each of the plurality of real-world objects detected in the scene based on the count of the specific object label 60x. The lower the count, the higher the ranking may be associated.

[0255] · Degree of reliability 800 associated with a specific real-world object As described above, the mobile AR app 1000 is configured to store in the memory 14 the reliability 800 belonging to one real-world object at the time of detection by the CVM 1040. The mobile AR app 1000 may be configured to access the memory 14 to determine the reliability 800 for each of the real-world objects detected in the scene, and further to assign a ranking to each of the plurality of real-world objects detected in the scene based on the reliability 800. The higher the reliability 800, the higher the ranking may be associated.

[0256] · Proximity of a predetermined real-world object to other real-world objects The mobile AR app 1000 may be configured to determine the distance between each of the real-world objects detected in the scene. Real-world objects that are farther away from other real-world objects may receive a higher rating than real-world objects that are closer to each other.

[0257] Other criteria may be considered when evaluating multiple real-world objects.

[0258] In some embodiments, the mobile AR application 1000 may be configured to assign a weight W of 1 to each of the criteria. In some embodiments, the weights W of each criterion may be the same. In other embodiments, the weights of each criterion may be different.

[0259] Thus, in step 1420, the mobile AR app 1000 may be configured to determine the sum of weighted rankings for each of the multiple real-world objects.

[0260] In step 1430, the real-world object with the highest rating among the multiple real-world objects is determined, and in step 1440, this real-world object is selected for association with one virtual character.

[0261] Conditions for generating virtual text As described above, in some embodiments, when the mobile AR app 1000 detects a particular real-world object, the mobile AR app 1000 may be configured to provide the player 1 with an opportunity to provide input related to that particular real-world object.

[0262] For example, in some embodiments, for method 600 to proceed to step 640, the face 3 of player 1 may be present in the image data captured from the camera feed at the same time as the particular real-world object, so that it can be detected by the CVM 1040.

[0263] For example, Figure 20A shows an image 2000A of a scene that may be captured and presented on the display 11 of a mobile terminal 10 by a mobile AR application 1000. Image 2000A is shown to include a real-world table 2001 and a real-world remote control 102. In the illustrated exemplary embodiment, the camera image includes a real-world cabinet 201 and a real-world remote control 102. For this example, we assume that only the real-world remote control 102 is included in the detectable real-world object set 58.

[0264] In step 610, the ODM 1010 captures an image from the camera feed (i.e., a video frame) of the mobile device 10. In step 620, the mobile AR app 1000 is configured to process the received camera image, and finally, in step 630, the real-world remote control 102 is detected in the scene as a result of a positive output from the CVM 1040.

[0265] In this embodiment, the mobile AR application 1000 is configured to request that the player include their face 3 in the captured image. The mobile AR application 1000 can make this request by a message displayed on the mobile device's display 11 and / or by an audible cue emitted by the mobile device's speaker 20 and / or by haptic feedback emitted by the haptic feedback module 54.

[0266] Figure 20B is an image 2000B of a scene that can be captured and presented on the display 11 of a mobile device 10 by a mobile AR application 1000, and includes both the player's face 3 and a real-world remote control 102.

[0267] The mobile AR app 1000 is configured to process the received camera image, and ultimately, as a result of a positive output of the CVM 1040, in step 630, both the face 3 of player 1 and the real-world remote control 102 are detected in the scene. In step 640, the AR API 1020 of the mobile AR app 1000 is configured to provide an anchor 46 to be roughly established at the location of the real-world remote control 102 in order to provide an anchor for the final virtual character 104 of 1 associated with the real-world remote control 102.

[0268] In step 640, the AR API 1020 of the mobile AR app 1000 is configured to provide an anchor 46 that is roughly established at the location of the real-world spatula 202 in order to provide an anchor for the final virtual character 204 associated with the real-world remote control 102. A visual effect corresponding to the virtual character 204 being emitted from / generated by the real-world remote control 102, namely a virtual flourish 103, is shown in Figure 20D. The virtual character 104 is displayed with motion lines 105 to provide an effect intended to show player 1 that the virtual character 104 is "generated from" the real-world remote control 102.

[0269] In step 650, the mobile AR app 1000 creates an association between the real-world remote control 102 and the virtual character 104.

[0270] In other embodiments, when a particular real-world object is detected, the mobile AR app 1000 may provide player 1 with an opportunity to provide input to prevent an association from being made between the particular real-world object and the virtual character. For example, player 1 may touch a portion of the display 11 / touchscreen 16 on which the particular real-world object is displayed. In other examples, player 1 may provide input by touching another portion of the display 11 / touchscreen 16 or through other suitable mechanisms.

[0271] Thus, before proceeding to step 640, method 600 may include a step of detecting player input that requests to prevent the creation of an association between the real-world object and the virtual character of 1. If such input is detected, the method does not proceed to step 640. If there is no such input, the method proceeds to step 640.

[0272] In some embodiments, at step 640, mobile AR app 1000 may be configured to select the position of anchor 46 so as to minimize the appearance of the associated underlying character that hides or obstructs other detectable real-world objects where the associated virtual character appears in the real-world scene.

[0273] FIG. 15 is a flowchart showing the steps of an algorithm 1500 that can be implemented by at least one processor 12 of mobile device 10 to position anchor 46.

[0274] At step 1510, mobile AR app 1000 is configured to detect other real-world objects from the set of detectable real-world objects 58 in the image data. At step 1520, mobile AR app 10 is configured to detect at least one region ("low-density region") within the image data of the real-world scene that has a low density of the other real-world objects. At step 1530, mobile AR app 1000 establishes an anchor 46 for the associated virtual character at a position associated with the low-density region.

[0275] Interaction within an AR game environment In some embodiments, the mobile AR application 1000 may be configured to provide player 1 with an opportunity to provide input for manipulating an associated virtual character (i.e., character manipulation input). For example, player 1 may touch a portion of the display 11 / touchscreen 16 on which the associated virtual character is displayed. In other examples, player 1 may provide input by touching other portions of the display 11 / touchscreen 16 or through other suitable mechanisms.

[0276] The associated virtual character may be manipulated in various different ways, such as by changing its position or modifying it (for example, by modifying one or more of its color, shape, size, scale, direction / orientation, etc.).

[0277] In one implementation of this embodiment, in response to detecting character operation input from player 1, the mobile AR application 1000 may be configured to represent the associated virtual character with an effect intended to indicate to player 1 that the associated virtual character is responding to the character operation input. In some embodiments, the effect may include, but is not limited to, changes in the associated virtual character's "facial expression," such as smirking, opening or closing its eyes, or changes in the associated virtual character's posture.

[0278] In some embodiments, the player 1 and the associated virtual character may "interact". For example, the associated virtual character may give the player 1 the impression that it is "following" the player 1 in the real world. For example, when the player 1 moves in the real world, the mobile AR application 1000 may be configured to maintain a predetermined distance between the associated virtual character and the player 1. For example, the spatial coordinates (XYZ) 1 of the player 1 may be recorded in memory 14. Also, as mentioned above, the spatial coordinates (XYZ) of the associated virtual character VCThe following may be recorded in memory 14: the spatial coordinates (XYZ) of player 1 and the spatial coordinates (XYZ) of the associated virtual character. VC The distance between player 1 and the associated virtual character may be maintained by maintaining a predetermined distance between them.

[0279] In some embodiments, a shared AR experience may be established such that two or more devices run the mobile AR app 1000 in the same physical space, and two or more players operating these devices experience substantially the same virtual world in real time from different viewpoints.

[0280] For example, Player 1, the first player, may operate the mobile device 10 running the mobile AR application 1000, and Player 2, the second player, may operate Player 1's mobile device 82 running the mobile AR application 1000. In this example, Players 1 and 2 are in the same physical space and a shared AR experience is established. In this embodiment, Player 1's mobile device 10 may have detected a particular real-world object, but the associated virtual character is displayed on both mobile devices 10 and 82.

[0281] In one example of implementing this embodiment, if player 2 performs character operation input to control the associated virtual character via the touchscreen 86 of one of the displays 84 / 1 of the mobile terminal 82, the mobile AR application 1000 may be configured to avoid implementing such character operation.

[0282] In this embodiment, the associated virtual character may give player 2 the impression that it is "running away" from player 2 in the real world. For example, when player 2 moves in the real world, the mobile AR application 1000 may be configured to maintain a predetermined distance between the associated virtual character and player 2. For example, the spatial coordinates (XYZ)2 of player 1 may be recorded in memory 14. The spatial coordinates (XYZ)2 of player 2 and the spatial coordinates (XYZ) of the associated virtual character VCThe distance between player 2 and the associated virtual character may be maintained by maintaining a predetermined distance between them. Furthermore, the orientation of the associated virtual character may be such that it appears as if the associated virtual character is "turning its face away" from player 2.

[0283] In some embodiments, the mobile AR application 1000 may be configured so that multiple related virtual characters can be displayed together on the display 11 of the mobile terminal 10. In such embodiments, the mobile AR application 1000 may be configured to represent the related virtual characters interacting with each other. For example, the multiple related virtual characters may "battle" each other. This interaction may be triggered and / or controlled as a result of or independently of player input.

[0284] Therefore, as can be seen from Figure 16, a method 1600 for execution by at least one processor is provided. Step 1610 includes maintaining one augmented reality (AR) game environment during a game session. Step 1620 includes receiving image data representing a real-world scene including one or more real-world objects. Step 1630 includes detecting at least one real-world object in the image data from a set of discoverable real-world objects. Step 1640 includes generating at least one virtual character in the AR game environment when the at least one real-world object is detected in the image data, the at least one virtual character having association with the at least one real-world object.

[0285] As a result of the methods, systems, and applications described above, the mobile AR application 1000 has a concrete and tangible effect in the real world by forcing the player to physically move their mobile device to find multiple real-world objects that trigger the generation of the multiple virtual characters in the game environment. Furthermore, since geographical location is not a requirement for gameplay or progression, the game can be played even offline, such as under unreliable mobile network conditions or in a cottage in the woods or on an airplane.

[0286] This specification and drawings merely illustrate the principles of the present invention. Those skilled in the art will therefore understand that various arrangements embodying the principles of the present invention, and which fall within the scope defined in the appended claims, are not expressly described or shown herein.

[0287] Furthermore, all examples described herein are primarily intended to help the reader understand the principles of the invention and are construed as not being limited to such specifically described examples and conditions. For example, this disclosure describes embodiments of the invention in relation to the analysis of various desktop objects. However, it will be understood by those skilled in the art that the invention can also be advantageously used to search for other types and forms of objects in other environments.

[0288] Furthermore, all descriptions herein describing the principles, aspects, and embodiments of the present invention, as well as specific examples thereof, are intended to encompass their equivalents. For example, the functions of the various elements shown in the figure, including any functional blocks labeled as “modules,” “plugins,” or “application program interfaces,” may be provided not only through dedicated hardware but also through the use of hardware capable of running software in conjunction with appropriate software.

[0289] Furthermore, although the aforementioned specification refers to mobile devices, those skilled in the art will readily recognize that various steps of the above-described method can be performed by any number of computer devices, such as video cameras, digital cameras, infrared cameras, desktop computers, laptop computers, tablets, smartphones, smartwatches, or other wearables. Herein, some embodiments are also intended to cover program storage devices, for example, digital data storage media that are machine- or computer-readable and encode machine-executable or computer-executable instruction programs, the instructions which perform some or all of the steps of the above-described method. These embodiments are also intended to cover computers programmed to perform the steps of the above-described method.

[0290] Those skilled in the art will understand that when a processor is described as “configured” to perform an action or process, this may mean that the processor performs an action or process by executing computer-readable instructions read from device memory in which these computer-readable instructions are stored.

[0291] Those skilled in the art should understand that any feature of any embodiment disclosed herein may be used in combination (e.g., instead of or in addition to) any feature of any other embodiment disclosed herein in some embodiments. Certain additional elements that may be required for the operation of some embodiments are assumed to be within the ordinary range of those skilled in the art and are therefore not described or illustrated. Furthermore, some embodiments may not include, lack, or function any elements not specifically disclosed herein.

[0292] Various embodiments and examples have been presented, but these are for illustrative purposes only and should not be considered limiting. Various modifications and extensions will be apparent to those skilled in the art.

Claims

1. A method performed by the processor of a computing system, Maintaining an augmented reality (AR) gaming environment during a game session, Receiving image data representing a real-world scene containing one or more real-world objects, To detect at least one real-world object from the image data, After receiving image data, an object identifier is assigned as a detection result indicating that at least one real-world object has been detected. Determining whether the object identifier assigned to the at least one real-world object is part of a set of discoverable real-world objects classified into a candidate object family, If it is determined that it is part of the set of discoverable real-world objects, then the at least one real-world object is associated with a discoverable real-world object from the set of discoverable real-world objects based on the object identifier assigned to the at least one real-world object, To generate at least one virtual character in the aforementioned AR game environment It has, The at least one virtual character to be generated is associated with the at least one real-world object based on the association between the at least one real-world object and the real-world object detectable from the set of detectable real-world objects, A method wherein each real-world object included in the detectable real-world object set is assigned at least one object family, and the at least one object family assigned to a given real-world object included in the detectable real-world object set is based on the expected physical location of the given real-world object.

2. The method according to claim 1, further comprising storing in memory the association between the at least one virtual character and the at least one real-world object.

3. The method according to claim 1, further comprising simultaneously displaying the at least one virtual character and the at least one real-world object on a display screen.

4. The method according to claim 1, The aforementioned at least one real-world object includes one specific real-world object, The at least one virtual character includes one specific virtual character, The at least one virtual character having an association with the at least one real-world object includes the particular virtual character having visual characteristics corresponding to the real-world visual characteristics of the particular real-world object.

5. The method according to claim 4, further comprising displaying the visual characteristics of the specific virtual character on a display screen and simultaneously displaying the specific real-world object on the display screen.

6. The method according to claim 1, The aforementioned at least one real-world object includes one specific real-world object, The at least one virtual character includes one specific virtual character, The at least one virtual character associated with the at least one real-world object includes the specific virtual character having movement characteristics corresponding to the real-world movement characteristics of the specific real-world object.

7. The method according to claim 6, further comprising displaying the movement characteristics of the specific virtual character on the display screen and simultaneously displaying the specific real-world object on the display screen.

8. The method according to claim 1, The aforementioned at least one real-world object includes one specific real-world object, The at least one virtual character includes one specific virtual character, The at least one virtual character associated with the at least one real-world object includes the particular virtual character having audible characteristics corresponding to the real-world audible characteristics of the particular real-world object.

9. The method according to claim 1, wherein the at least one virtual character includes one single virtual character.

10. The method according to claim 1, wherein the at least one virtual character comprises a plurality of virtual characters, and the plurality of virtual characters have associations with the at least one real-world object.

11. A method according to claim 1, wherein the at least one real-world object comprises at least one first real-world object, and the at least one virtual character comprises at least one first virtual character, and the method further comprises To detect at least one second real-world object from the aforementioned set of detectable real-world objects using image data, When at least one second real-world object is detected from the image data, at least one second virtual character is generated in the AR game environment, and the at least one second virtual character has an association with the at least one second real-world object. It has.

12. A method according to claim 1, the method further comprising storing in memory the marks of the at least one real-world object and the marks of the at least one virtual character that are generated in the AR game environment when the at least one real-world object is detected.

13. A method according to claim 1, wherein the at least one virtual character is generated in the AR game environment, From the detectable set of real-world objects, determine at least one object family associated with at least one real-world object, and the at least one object family is associated in memory with a virtual character set. Selecting the at least one virtual character from the virtual character set associated with the at least one object family. It has.

14. The method according to claim 1, wherein each real-world object from the detectable real-world object set is assigned at least one object label.

15. A method according to claim 1, wherein generating the at least one virtual character in the AR game environment further comprises establishing anchors that fix the at least one virtual character to the at least one real-world object in relation to the position of the at least one real-world object.

16. The method according to claim 1, wherein the at least one virtual character includes one first virtual character, and the method further comprises The process involves detecting the at least one real-world object in the image a second time, In response to the detection of the at least one real-world object for the second time, the association between the at least one virtual character and the at least one real-world object is maintained without generating a new virtual character that has an association with the at least one real-world object. It has.

17. The method according to claim 1, wherein the at least one virtual character includes one first virtual character, and the method further comprises The process involves detecting the at least one real-world object in the image a second time, In response to detecting the at least one real-world object for the second time, the at least one virtual character maintains the association between the at least one real-world object and the at least one virtual character, and generates one additional virtual character that has an association with the at least one real-world object. It also possesses.

18. The method according to claim 1, wherein the at least one real-world object includes one first real-world object, the object identifier includes a first object identifier, the at least one virtual character includes one first virtual character, and the method further includes To detect at least one second real-world object from the image data, After receiving image data, the detection result indicates that at least one second real-world object has been detected, and assigns a second object identifier that is the same as the first object identifier. In the AR game environment, without generating one additional virtual character, the at least one second real-world object is associated with a real-world object included in the set of discoverable real-world objects, and the association is performed based on a second object identifier assigned to the at least second real-world object. It has.

19. A method according to claim 1, wherein the method further comprises To maintain a first record of real-world objects detected during the aforementioned game session, To provide the player with a representation of the content of the aforementioned first record. It has.

20. The method according to claim 19, wherein the method further comprises Maintain a second record of real-world objects within the set of detectable real-world objects that were not detected during the aforementioned game session. To provide the player with a representation of the content of the aforementioned second record. It has.

21. A method according to claim 1, wherein the method further comprises To maintain the first record of the virtual character generated during the aforementioned game session, To provide the player with a representation of the content of the aforementioned first record. It has.

22. A method according to claim 21, wherein the method further comprises Maintain a second record of the virtual character associated with the real-world object in the set of detectable real-world objects that was not detected during the game session, To provide the player with a representation of the content of the aforementioned second record. It has.

23. The system comprises at least one processor, a memory for storing instructions to be executed by the processor, at least one input device configured to accept input from a user, and at least one output device configured to provide output to a user. The at least one processor is configured to execute instructions in memory to implement an interactive computer program that generates the output in response to the input it receives, and the interactive computer program is Maintaining an augmented reality (AR) gaming environment during a game session, Receiving image data representing a real-world scene containing one or more real-world objects, To detect at least one real-world object from the image data, After receiving image data, an object identifier is assigned as a detection result indicating that at least one real-world object has been detected. Determining whether the object identifier assigned to the at least one real-world object is part of a set of discoverable real-world objects classified into a candidate object family, If it is determined that it is part of the set of discoverable real-world objects, then the at least one real-world object is associated with a discoverable real-world object from the set of discoverable real-world objects based on the object identifier assigned to the at least one real-world object, To generate at least one virtual character in the aforementioned AR game environment, The process includes at least one of the following steps: The at least one virtual character to be generated is associated with the at least one real-world object based on the association between the at least one real-world object and the real-world object detectable from the set of detectable real-world objects, A gaming device in which each real-world object included in the detectable real-world object set is assigned at least one object family, and the at least one object family assigned to a given real-world object included in the detectable real-world object set is based on the expected physical location of the given real-world object.

24. A computer-readable storage medium on which computer-readable instructions are written, which, when read and executed by at least one processor of a gaming device, cause the gaming device to perform a method included in an interactive computer program, The method is Maintaining an augmented reality (AR) gaming environment during a game session, Receiving image data representing a real-world scene containing one or more real-world objects, To detect at least one real-world object from the image data, After receiving image data, an object identifier is assigned as a detection result indicating that at least one real-world object has been detected. Determining whether the object identifier assigned to the at least one real-world object is part of a set of discoverable real-world objects classified into a candidate object family, If it is determined that it is part of the set of discoverable real-world objects, then the at least one real-world object is associated with a discoverable real-world object from the set of discoverable real-world objects based on the object identifier assigned to the at least one real-world object, In the aforementioned AR game environment, at least one virtual character is generated. It has, The at least one virtual character to be generated is associated with the at least one real-world object based on the association between the at least one real-world object and the real-world object detectable from the set of detectable real-world objects, A computer-readable storage medium wherein each real-world object included in the detectable real-world object set is assigned at least one object family, and the at least one object family assigned to a given real-world object included in the detectable real-world object set is based on the expected physical location of the given real-world object.

Images