System and method for generating an augmented reality object

Abstract

The user can capture video of the user's surroundings using the smart device's camera, and the smart device can overlay virtual objects onto the image being captured by the smart device. The virtual objects can be selected based on real objects in the smart device's environment, such as chalk drawings on the floor. The application may take into account other factors when selecting items, such as the user's profile and geolocation, as well as features downloaded by the user of the application (such as characters, accessories, masks, avatars, etc.). The application allows the user to interact with the virtual objects by customizing the objects and adding features to them.

Description

Technical Field

[0001] This application claims the benefit of U.S. Non-Provisional Application No. 17 / 102,721, filed on Nov. 24, 2020, and patented on May 25, 2021, as U.S. Patent No. 11,017,609, the entire disclosure of which is incorporated herein by reference.

[0002] The present invention relates to a system and method for generating augmented reality objects.

Background Art

[0003] Augmented reality is an augmented version of reality, where a live direct or indirect view of the physical environment of the real world is augmented by computer-generated images superimposed on the view of the user of the real world. In some cases, augmented reality includes computer-generated perceptual information across multiple sensory modalities such as vision, hearing, touch, proprioception, smell, etc. Augmented reality can enhance a person's current perception of reality.

[0004] In machine learning, statistical techniques are used to train a computer to perform a specific task using data, even if it is not explicitly programmed to do so. The goal of machine learning is to build algorithms that can learn from data and make predictions about the data. These algorithms function by creating mathematical models that can classify data. The process of creating a model may include training and fine-tuning model parameters using input data.

Summary of the Invention

[0005] Augmented reality provides unique learning and entertainment tools for children to interact with their surrounding environment, enhance their creative senses, and explore objects. Specifically, using augmented reality, virtual objects (or augmented reality objects) can be superimposed on a display of an environmental image being captured (e.g., in real time). For example, a user of a smart device (e.g., a smartphone, tablet, smart glasses, wearable device, etc.) can capture a video of the user's surroundings using the camera of the smart device, and the smart device (e.g., an augmented reality application running on the smart device) can superimpose a virtual object (e.g., an augmented reality object) on the image being captured by the smart device.

[0006] The virtual object can be selected based on an actual object in the child's surrounding environment, such as a chalk drawing on the floor. The application may consider other elements, such as the user's profile and geographical location information, and features downloaded by the user of the application (e.g., characters, accessories, masks, avatars, etc.) when selecting an item. Through the application, a child can interact with the virtual object by customizing the object and adding features to the object. In one exemplary embodiment, the user can record his or her own interaction with the virtual object, and the application can post this interaction to a social media application. The augmented reality application can display features and functions of the virtual object that a child cannot otherwise see or experience. Therefore, the virtual object can enhance the user's learning experience.

[0007] The summary of the invention is provided to introduce a selection of concepts in a simplified form that will be further described in the following modes for carrying out the invention. The summary of the invention is not intended to identify the key features or essential features of the subject matter recited in the claims, nor is it intended to be used as an aid in determining the scope of the subject matter recited in the claims.

[0008] Additional features and advantages will be described in the following description, and in part will be apparent from the following description, or may be learned by practice of the teachings herein. The features and advantages of the invention may be realized and obtained by means of the means and combinations particularly pointed out in the appended claims. The features of the invention will become more fully apparent from the following description and appended claims, or may be learned by practice of the invention described below.

Brief Description of the Drawings

[0009] To explain the manner in which the above and other advantages and features are obtained, the subject matter briefly described above will be described in more detail with reference to the specific embodiments shown in the accompanying drawings. It is understood that these drawings are only illustrative of typical embodiments and are not to be considered limiting, and the embodiments will be described with additional specificity and detail using the accompanying drawings.

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

[0011] To explain various features of the present invention, exemplary embodiments of the present invention will be described. The embodiments described herein are not intended to limit the scope of the present invention, but rather are intended to provide examples of components, uses, and operations of the present invention.

[0012] The present disclosure relates to a smart device application program that can record a photo or video feed by a camera of a smart device and display various virtual objects (e.g., animations) on the photo and / or video feed. Depending on the objects that appeared in the photo and / or video before overlaying the virtual objects, the application can determine the selection of the objects and the functions of the objects to be displayed on the video feed. In one exemplary embodiment, the user can take a snapshot of a chalk drawing on the floor and display an animation selected based on the object depicted by the chalk drawing. With this application, the user can interact with the animation. Also, with this application, the user can record the interaction with the application and post the interaction to a social media platform.

[0013] FIG. 1 shows an exemplary account login page 110 of an augmented reality application (the "application"). The account login page 110 enables a user to access the application. The application may be configured to display a user interface 101 on a display of the smart device 100. The account login page 110 may include a field 111 for receiving a username, a field 112 for receiving a password, and a login button 114 for logging in to the account. Additionally, the page 110 may include a button 113 for accessing the augmented reality features of the application without entering a username or password. In one exemplary embodiment, when accessing the application using a username and password, the application can display customized features and settings for the user by accessing the user's profile. However, in another exemplary embodiment, a user can access the application without a username or password.

[0014] Figure 2 shows an exemplary home page 210 of the application. The home page 210 enables the user to access the augmented reality function of the application and navigate to a page that allows editing of the user's profile. The home page 110 may include a field 211 for receiving the user's name, a field 212 for receiving a theme, a button 213 for editing the user's profile, and a button 214 for activating the augmented reality function of the application. When the user enters their name, the application can display and / or pronounce the user's name, enabling the user to enjoy a more personalized experience when virtual objects are displayed to the user. For example, the animated figure displayed on the user interface 101 can announce the user's name before talking to the user or receiving commands from the user. The user can enter a theme in the field 212. When a theme is entered, the user can identify a subset of virtual objects (and / or functions) they want to interact with. For example, in one embodiment, the theme may be science or entertainment. In response to selecting science as the theme, the application can search for and display science-related virtual objects, such as compounds, and in response to selecting an entertainment theme, the application can search for and display anime characters. In one example, the search function is triggered by scanning a photo with the camera of the smart device.

[0015] In one exemplary embodiment, the application may store a series of themes and define a series of objects and virtual reality responses for each theme. For example, in the case of an entertainment theme, the application may store toys, and the virtual reality response may include animations with objects and / or customized accessories added to the objects. The application may be configured to receive a photo or video and detect one or more objects in the photo or video. The application can trigger a virtual reality response based on the detected objects. The virtual reality response may include displays such as animations, accessory objects, etc. In one exemplary embodiment, the application can change the animation and / or accessory objects based on the user's profile, geographical location information data received from the user, the temperature or humidity of the environment where the smart device is located, the speed, acceleration, and / or GPS position detected by the smart device, the face or fingerprint detected by the smart device, the strength of the cellular signal detected by the smart device, the background music, audible or inaudible sounds detected by the smart device, etc. In one example, when detecting a picture of a rabbit, the application can trigger a virtual reality response, such as an animation showing the rabbit walking. When detecting that the speed of the smart device is faster than, for example, 20 miles per hour, the application can display a rabbit "driving" a car. Similarly, if the detected humidity exceeds a threshold or data from the National Weather Service indicates that it is raining, the application can represent raindrops in the animation. As another example, when the application detects that the weather is rainy, it can prompt the user to obtain an umbrella. For example, the application can guide the user to an online app store to obtain an umbrella as an accessory.

[0016] In one example, when a barcode or object (e.g., a shopping cart) is detected, the application can trigger a virtual reality response and display an animated character, for example, running and / or walking inside the store. In one exemplary embodiment, the user can scan a barcode (or object or brand) inside the store, and the application can guide the user to the area where a specific product of the store is displayed. For example, when the barcode is detected, the application can receive information about the store, such as name, address, seller ID, etc. The application can send a signal to a central server that stores information about the store. The signal can include information about the store and / or specific products or manufacturers. The central server can store information such as an indoor map of the store, a database of where each product is located, etc. The central server can send this information, such as the indoor map and / or the indoor location of the product or the products of the manufacturer, to the smartphone. The application can devise a navigation strategy and guide the user to the area containing the product. For example, the application can display an animated character walking towards this area of the store. Also, the application can display one or more products to the user for purchase. For example, when the user arrives at the area containing the products of the manufacturer, the application can recommend one or more products based on the user's profile. In one example, the application can use triangulation technology to determine the position of the smart device inside the store.

[0017] In one example, the application can recommend products or discounts based on inaudible sounds detected at the store. The application can receive the inaudible sound and detect the message contained in the sound. The application can recommend a product based on this message. Alternatively, the application can send the message (and / or the user's profile) to a central server and determine the product based on the response from the central server. The central server can search for one or more products based on the message (e.g., 10% discount) or the user's profile (e.g., 5-year-old girl) and identify the product to recommend to the user.

[0018] In one exemplary embodiment, a user may be associated with a profile. The application can use the information included in the user's profile to select virtual objects (and / or functions or virtual reality responses) that match the user's expected preferences. For example, for a younger user, such as a 3-year-old user, the application can recommend simple science experiments, while for an older user, such as a 10-year-old user, the application can recommend complex science experiments. As another example, the application can select the language and accent for an animated figure to talk to the user based on the user's profile (and / or geographical location information). In one example, the profile may include historical information, biometric information, and / or financial information such as name, gender, age, weight, height, language, address, credit card information, etc. As another example, the profile may include information regarding the user's interaction with the application, such as the user's previous selections, snapped photos, ordered searches, purchases, etc. In one exemplary embodiment, the user can edit or update the information included in the user's profile by pressing button 213. The user's profile may be stored locally on the smart device 100 or on a server. In one exemplary embodiment, the user can access the user's profile only after logging in (and the application can customize the user's experience based on the profile). In another example, the application can maintain or obtain the user's profile without requiring the user to log in.

[0019] In one exemplary embodiment, the application can use a machine learning model to predict the types of characters, virtual reality objects, virtual reality responses, and virtual reality accessories that a user may be interested in. For example, the central server can receive data from multiple applications or smart devices. The data may include user profile information and user interaction information with each application. Based on this information, the central server can train a classification model that can predict the characters, virtual reality objects, virtual reality responses, and virtual reality accessories that a user may be interested in. The central server can send the classification model to the smartphone, and the application can use the classification model to recommend characters, virtual reality objects, virtual reality responses, and virtual reality accessories to the user.

[0020] In one exemplary embodiment, by pressing button 214, the application can activate the augmented reality function. FIG. 3 shows an exemplary augmented reality page 310 of the application. When the user accesses this page, the camera of the smart device 100 is activated, and on the augmented reality page 310, the user interface can display the video feed 311 received from the camera of the smart device or a series of images. The user interface can display the video feed 311 or a series of images received by the camera. In one exemplary embodiment, the application can detect a picture 314 or an object in the video feed, and the user interface can overlay virtual objects (e.g., pictures, shapes, or animations) on the video feed or a series of images. In one exemplary embodiment, with this page, the user can also take a photo. For example, while displaying the video feed or a series of images, the page may include a button 313 for taking a video segment or a photo. When taking a video segment or a photo, the application can identify the picture 314 or an object in the photo and display virtual objects based on the identification of the picture or object.

[0021] In one exemplary embodiment, the augmented reality page 310 may include a target 312. The target 312 may be a pointer that is displayed on the augmented reality page 310 to guide the user to point the smartphone at a specific position on the ground plane or in space, thereby facilitating the task of detecting an object or picture 314. For example, the target 312 may have a circular shape, and the application can suggest that the user point the smartphone at the picture so that the object or picture 314 fits within the target 312. By thus requesting the user to point the smartphone at the picture, the object detection task of the application can be simplified.

[0022] In one exemplary embodiment, if the user captures a photo but there is no picture within the target 312, or only a part of the picture is within the target 312, the application can display an error (e.g., "Object not detected", "Only part of the object is detected"). In this example, when the user captures a photo, the application can attempt to detect the object. If the object is not detected, or if the object is detected but the detection confidence rate is below the threshold, the application can display an error signal.

[0023] In one exemplary embodiment, the application can detect the ground plane in a video feed (e.g., streaming image data) displayed on the user interface of the smart device. Based on this detection, the application can place augmented reality objects on the ground plane. Various techniques can be considered to detect the ground plane. In one technique, the application can detect some points on a plane in the video, and if those points have the same characteristics, it can recognize that plane as the ground plane. Specifically, using feature extraction by visual processing, the application can extract specific features in the image and use the extracted features to establish specific points on the same plane. When the application has established that the extracted points are on the same plane, it can establish that the plane where these points are located is the ground plane. Grouping of feature points and the orientation of the smartphone are helpful for determining the ground plane. Some of the sensors of the smart device can also be used in this process. For example, the accelerometer and gyroscope can provide data to the application to facilitate the detection of the ground plane. In one exemplary embodiment, the application can display virtual reality objects and virtual reality responses in the visual representation of the ground plane.

[0024] In another technique, the application can identify the horizontal plane in the image being captured by the smart device. The application can further determine the size of the horizontal plane and determine that the size of the horizontal plane meets a threshold. Based on the size of the horizontal plane, the smart device can designate the horizontal plane as the ground plane and overlay an augmented reality object representation on the image for the horizontal plane designated as the ground plane.

[0025] In some embodiments, the user may need to establish a ground plane before pointing the smart device at an object. Specifically, in these embodiments, the user can point the smart device at the ground before pointing the smart device at an object. If the user does not first point the smart device at the ground, the user interface can display a message requesting the user to establish a ground plane.

[0026] Additional ground plane detection techniques are described in U.S. Patent Application No. 15 / 850234 (currently U.S. Patent No. 10,026,209), which is hereby incorporated by reference herein.

[0027] In one exemplary embodiment, the application can determine the distance between a smart device (e.g., a smartphone) and a point on the ground plane. Various techniques can be used to determine the distance. For example, when the user points the smartphone towards the ground, the application can determine the point on the ground where an assumed vertical line or plane emerging from the smartphone hits or intersects the ground plane. This vertical line or plane can correspond to the center point of the camera's field of view or the center point of the image data. This point (e.g., the point where the ground plane intersects the assumed ground plane) may be referred to as the intersection point. The application can determine the distance between the two points by determining the intersection point and the coordinates of the smart device. The intersection point is defined as described above, but those skilled in the art will recognize that there are many different ways to define the intersection point. For example, the aforementioned assumed line does not need to be perpendicular to the smartphone. In one example, the intersection point can refer to a position within the surrounding environment of the smartphone that corresponds to the center point of the target 312.

[0028] In one exemplary embodiment, after determining the ground plane, the user can point the smartphone towards an object or a picture. The application can determine the distance between the smart device and the intersection point on the object or picture. In one example, the application can identify the object and / or picture and display a virtual object on the display based on the detected object.

[0029] In one exemplary embodiment, the application can use the distance between the smart device and the intersection point (or object or picture) to determine the size for virtual reality objects and / or responses. For example, when the intersection point is close, e.g., within 5 feet, the application can display the virtual reality objects and / or responses in a large size, while when the distance exceeds 5 feet, the application can display the virtual reality objects in a small size.

[0030] In this disclosure, various embodiments have been described in which an application (or user) takes a photo of an object or the application receives image data, a video feed, or a video stream. One skilled in the art will recognize that a photo may be image data and that a video feed or video stream may also include image data. Additionally, a video feed or video stream may include multiple photos (or photo frames). Further, a photo or image data may include a single still image capture or a capture frame of streaming image data (video feed or video stream).

[0031] (Machine Learning Model and Object Recognition) In one exemplary embodiment, after the user takes a snapshot photo or captures an image, the application can feed the image data of the photo into a machine learning or predictive model. In one embodiment, the application can trim a portion of the photo before feeding the photo into the model. For example, the application can trim the central portion of the image to include only the central one-third of the image. As another example, the application may exclude pixels outside the target.

[0032] In one exemplary embodiment, the machine learning model may be designed to identify pictures and / or objects. In this exemplary embodiment, the model can be trained using many pictures of objects. For example, the machine learning model can be trained to identify pictures or objects within a home. When a picture is provided to the model, the model can return one or more identification names and a probability value associated with that identification for each identification name. For example, the model can determine that an anime character named Armin is displayed with an 80% probability in the picture. Further, if the probability exceeds a threshold, the application may display the name of the detected picture. As another example, the model can determine that the object is a toy gun with a 68% probability. In this example, since the probability is below the threshold, the application does not display the name of the detected object.

[0033] In one example, if the model is unable to identify an object or the probability that the model identifies the object is too low, an error is displayed to the user. In another example, if the model is unable to identify an object or the probability of identifying the object is too low, the application can present a predetermined virtual object, such as an animation, to the user. In one example, the machine learning model of the present disclosure may be stored in a smart device. However, in some embodiments, the model may be stored in another device, and the smart device may transmit a picture or video to the other device for identification. The smart device is configured to receive identification information from the other device.

[0034] In one exemplary embodiment, before feeding the image data of a photo (or video) into a machine learning model, the application can indicate whether the image quality is satisfactory. For example, if there is too much light or glare in the video or streaming image that the camera is capturing, the application can indicate to the user that the user needs to move the camera or point it in a different direction. As another example, if the camera is too close to an object such that the object does not fit within the display of the target or smart device, the application can display a notification.

[0035] FIG. 4 shows an augmented reality page 310 of an application that displays a notification according to an exemplary embodiment. In this exemplary embodiment, first, the user points the smart device 100 at a picture on the ground. The application determines the ground plane and detects the coordinates of the intersection points on the ground. The application also determines the distance between the smart device and the intersection points. The application detects the picture or object at the target 312. The application can determine that only a part of the picture 314 is displayed within the target 312. Accordingly, the application can display a notification 411 including a warning message on the user interface 101. In one example, when the user moves the smart device so that the entire picture is displayed within the target, the application can execute an object recognition routine.

[0036] In one exemplary embodiment, the application may perform an object recognition function. If the application detects that the entire object is not displayed in the user interface (or target), or that the object is facing in a direction that is not optimal for identification, it can notify the user to switch the angle. For example, the application may capture an image of a child's toy and be implemented to identify at least the toy's manufacturer and model. Before sending the image of the toy, the application determines whether the toy is within the frame. If the application does not detect the toy within the frame, it can display an error message to the user. If the application detects the toy within the frame, it can determine which side of the toy is within the frame. In some embodiments, the model can most accurately and efficiently identify the captured toy based on the image data of the front cover of the toy box. If the application determines that the user is pointing the smart device from the side of the toy box, it can notify the user to move the smart device around the toy box to capture a better shot of the toy box. When the user moves to the desired side of the toy box, such as the front cover, the application can proceed to the next step.

[0037] (Selection of Virtual Object) In one exemplary embodiment, after determining the identity of a picture or object, the application can perform a search to determine actions (e.g., virtual reality responses) to execute in response to the identified picture or object. The application can search the memory of the smart device and obtain animations tagged with words that match the identified picture or object. For example, in response to identifying a dog in a picture, the application can obtain an animation of a dog nodding its head up and down. The animation can be overlaid on the video feed. As another example, the application can send a search query to the server and receive from the server an animation tagged with words that match the identified picture or object. The application can display this animation to the user.

[0038] In one example, when selecting a virtual reality object and / or animation, the application may consider other elements in addition to the identification information of the object. For example, when searching for a virtual reality object and / or animation, the application may receive data points such as speed, acceleration, face identification information, fingerprint identification information, geographical location information, time, weather information, profile information, etc. In one example, the application is configured to display only animations that match these elements (e.g., speed and humidity exceed a threshold), or to obtain only such animations from the central server. In this example, when the speed and humidity exceed the threshold, the application can display, for example, a dog riding in a car when it is raining. In another example, the application is configured to modify the virtual reality object and / or animation to match the elements. For example, the application can store an animation of a moving dog. However, when the speed exceeds the threshold, the application can display that the dog is riding in a car.

[0039] FIG. 5 shows an augmented reality page 310 of an application that displays virtual objects according to an exemplary embodiment. In this exemplary embodiment, the user can direct the smart device 100 towards the picture 314. Thereafter, the user can press the capture button 313, and in response, the application can record a video segment. The machine learning model of the application can analyze the captured video segment and determine that a dog is displayed on the target 312. In response, the application can obtain a virtual object 511 depicting an anime dog nodding its head. The application can overlay the virtual object on the video feed 311.

[0040] In one exemplary embodiment, the application of the smart device can determine the virtual object based on the picture scanned on the augmented reality page 310 (and / or the user's profile or a theme defined by the user). In another exemplary embodiment, a server communicating with the smart device can determine the virtual object. For example, the application can send a signal to the server, and in response, the server can send the virtual object to the smart device. In yet another exemplary embodiment, the smart device and the server can jointly determine the virtual object. For example, the application can send a video snippet of the video stream to the server. The server can determine the theme of the virtual object based on the video stream. The server can send the theme to the smart device, and the smart device can determine the virtual object based on the theme (and / or the user's profile). In one exemplary embodiment, the application can automatically determine the theme based on data points detected by the smart device, such as speed or humidity. In yet another example, the application can determine the theme based on data received from a central server, such as a weather forecast.

[0041] In one exemplary embodiment, the application or server of the smart device may include an object recognition module. For example, the module can receive a video feed from the camera of the smart device and determine one or more objects in the video feed. The application or server of the smart device can determine a virtual object based on the identity of the object displayed on the target and other objects detected in the video feed. For example, the object recognition module can detect a dog within the target 312. The object recognition module can also identify some indoor objects such as doors, carpets, tables, chairs, etc. in the user's environment. The application or server can determine to display augmented reality objects related to the dog and these indoor objects, such as indoor dog toys. As another example, the object recognition module can detect a tree, and the application or server of the smart device can determine to display an outdoor toy for the dog.

[0042] In one exemplary embodiment, an application or server of a smart device may include a scene recognition module. For example, the module can receive a video feed from a camera of the smart device. In one embodiment, the camera provides the video feed to the module in the background so that the user is unaware of providing the video feed to the module. The module can analyze this video and determine the scene presented in this video feed. For example, in one embodiment, the module can determine that the scene depicts a beach. The module can provide this information to an application of the smart device, and the application can determine to display augmented reality objects related to the beach, such as a boat or a flag. As another example, the module can determine that the scene is an indoor shopping area. In response to this determination by the module, the application can display a shopping cart as an augmented reality object.

[0043] In one exemplary embodiment, an application of a smart device may include a module for determining the location of the smart device, and the application or server can determine virtual objects based on the location of the smart device (and / or other criteria). For example, the application can determine the location of the smart device based on GPS signals received by the smart device. As another example, the application can determine the location of the smart device based on Wi-Fi signals received at an indoor location. As yet another example, the application can determine the location of the smart device based on inaudible signals detected by the smart device. The application or server can use the location information to determine virtual objects. For example, if the application determines that the user is at a restaurant, the application can choose to display augmented reality objects related to the restaurant, such as various types of food.

[0044] In one exemplary embodiment, when determining a virtual object, in addition to using location-based data (and / or other criteria), the application can use sensory information provided by the smart device. For example, the smart device can utilize a compass, gyroscope, and accelerometer to provide data for determining components of a human challenge. Subsequently, based on this data, for example, it is determined which augmented reality object is to be presented to the user according to which side of the road the user is facing.

[0045] (Function of Virtual Object) In one exemplary embodiment, the user can interact with the virtual object. Various techniques can be considered for the user to interact with the virtual object. For example, the user can tap or click on the virtual object. As another example, the user can draw a circle around the virtual object. As yet another example, the user can draw a connection between the virtual object and another object. For example, the user can connect a virtual object to another object by moving a finger from the first virtual object to the other virtual object.

[0046] For example, as described above, in response to receiving a command, the application can display a page for selecting a type of interaction with the virtual object. FIG. 6 shows an exemplary interaction page 610 for specifying the type of interaction with the virtual object. FIG. 7 shows an exemplary implementation of the interaction requested on the interaction page 610. In this exemplary embodiment, the interaction page 610 may include a button 611 for adding an accessory to the virtual object, a button 612 for downloading a character, a button 613 for adding an object from the environment, and a button 614 for training the virtual character.

[0047] In response to pressing a button 611 for adding an accessory to a virtual object, one or more options for customizing the virtual object can be prompted to the user. For example, by pressing the button 611, the application can prompt the user to select a color (e.g., gray) of the virtual object 511 or add an item or clothing item (e.g., a dog collar) to the virtual object. FIG. 7 shows a customized object 711 added to a virtual object 711. In this example, the virtual object 711 is a dog collar. In one exemplary embodiment, the application can provide a library for storing these virtual objects. The user can collect these virtual objects by downloading them from a central server or creating these virtual objects in the user interface of the smart device, for example, by creating a picture. When attempting to customize a virtual object, the user can select one of the customized objects stored in the library and add it to the virtual object.

[0048] In response to pressing button 612 to download a character, one or more virtual objects can be selected and the user can be prompted to download from an application store. Each downloaded virtual object can be associated with one or more tags. This facilitates the selection of downloaded virtual objects in response to matching items in a photo or video. For example, by pressing button 612, the application can display the application store and select a puppy character for download. When a virtual object is downloaded, the application can display the downloaded virtual object each time it is associated with virtual object 511 or in response to a scanned photo or video. FIG. 7 shows a downloaded virtual object 712 added to the augmented reality page 310. In this example, the downloaded virtual object 712 is a puppy. In one exemplary embodiment, a predetermined set of reactions and results can be defined for the virtual object and / or the downloaded virtual object such that when the downloaded virtual object is added next to or on the same page as the virtual object, the virtual object and / or the downloaded virtual object can generate a predetermined set of reactions and results. For example, in this case, when the puppy 712 is added to the augmented reality page 310, both the dog 511 and the puppy 712 can start barking. In one example, the downloaded virtual object can be stored in a library and accessed from the library.

[0049] In response to pressing button 613 to add an object from the environment, the user can be prompted to provide a live feed from the camera. In this mode, the user can select an item, and the application can add a photo of this item (or a picture associated with the item determined by a machine learning model) to the user's profile (or library). For example, by pressing button 613, the application can prompt the user to select an object in the video feed. In one exemplary embodiment, the user can select a bone (e.g., by drawing a circle around the object or by clicking on the object). The application can then add the bone to the user's profile (or library) and display the bone associated with the virtual object. FIG. 7 shows a selected item 713 associated with virtual object 711. In this example, the selected item 713 is a bone. In one exemplary embodiment, a predetermined set of reactions and results can be defined for the virtual object and / or the selected item, such that when the selected item is added next to the virtual object or on the same page as the virtual object, the virtual object and / or the selected item can generate a predetermined set of reactions and results. For example, in this case, if bone 713 is added to the augmented reality page 310, dog 511 can grab bone 713.

[0050] In response to pressing button 614 for training a virtual character, the virtual character can be trained to perform a specific function in response to a trigger event. For example, during the training mode, the user can select an object from the library and move the virtual object and the selected object in a specific pattern. The application can learn this movement pattern and, when the virtual object and the selected object are displayed together, can reproduce that movement. In one example, the virtual object can be a unicorn and the selected object can be a rainbow. During the training mode, the user can move the unicorn towards the rainbow. Therefore, when the unicorn and the rainbow are displayed together, the unicorn can move towards the rainbow. In one example, a dinosaur can be trained to jump over an object. In another example, a dolphin can be trained to balance a ball. In yet another example, an alien can be trained to make a funny expression. For example, during training, the user can move the components of its face in a funny way.

[0051] In one exemplary embodiment, the user can collect a series of characters that can interact with each other and add them to the library. Each character can have a predetermined set of strengths and weaknesses, and the user can activate the characters to compete with each other. For example, the user can add a series of robots to the user's library. Each robot can have construction capabilities. When two or more robots are activated in the user interface, the application can display the activated robots. The application can also display that the robots can participate in a construction contest and the robot with the highest construction ability can win. In one example, the user can collect a series of princesses. In this example, these princesses can participate in a beauty contest and the princess with the highest beauty score can win.

[0052] In one exemplary embodiment, the application may include a feature detection module. For example, the user can upload a photo before editing and a photo after editing. The application can determine the features added to the photo and add similar features to the virtual reality object. For example, the user can upload a photo of the user before applying makeup and a photo of the user after applying makeup. The application can find changes in one or more makeup features, such as the addition of lipstick, and add the same makeup features to the virtual reality character. In one example, the user can add makeup to a painting. The application can determine the added makeup and provide similar makeup to the virtual reality object.

[0053] In one example, the application allows the user to customize the virtual representation of a real object. For example, the user can scan a friendship bracelet with alphabet beads added to specify a person's name. The application can recognize this object and recognize the parts of the object that can be changed or customized. In response to the user's instructions, the application can customize the friendship bracelet. For example, the application can change the color of the beads or add glitter.

[0054] (Social media aspect) In one exemplary embodiment, the application can display a virtual object that performs a task and prompt the user to perform the same task. The application can record the user performing the task and overlay the virtual object on the user's recording. For example, the application can display an anime character dancing and prompt the user to imitate the same dance. The application can record the user dancing and display a virtual character next to the user who is making the dance movements.

[0055] In one exemplary embodiment, the application can record a user who is dancing. The application can use object-oriented technology to determine the user's movements. The application can display a virtual character that makes the same dance movements next to the user.

[0056] In one exemplary embodiment, the application can post the user's record to a social media platform. For example, the application may include a page configured to receive the user's social media credentials. When creating a video, the application can prompt the user whether they are interested in posting the video to a social media platform. If the user responds affirmatively, the application can post the video to the social media platform. In one example, the user can post information from the user's profile to the social media platform. For example, the user can post a list of items they have collected to their social media account.

[0057] (Construction of 3D space) In one exemplary embodiment, the application can detect the ground plane (or horizontal plane) in a video stream displayed on the user interface of a smart device. Based on this detection, the application can place augmented reality objects on the ground plane (or horizontal plane), for example, place a dog on the floor and a cat on the bed.

[0058] In one exemplary embodiment, the application can construct a 3D model of the space in front of the user. For example, the user can first scan the space, and the application can construct a 3D model of the scanned space. The application may include a database of attributes regarding augmented reality objects. Based on the 3D model and the information provided in the memory (or server database), the application can place the augmented reality objects in their logical locations. For example, the 3D model may include information regarding horizontal and vertical planes. The application can place the augmented reality objects that are normally placed on a horizontal plane on the horizontal plane and the augmented reality objects that are normally placed on a vertical plane on the vertical plane. In one example, since the application can determine that the user is in an indoor space, it can determine to display indoor items to the user. The application can determine to display a frame on the wall (i.e., a vertical plane) of the indoor space and can determine to display a dog on the bed (i.e., a horizontal plane).

[0059] (Technical implementation of the server) FIG. 8 shows exemplary hardware components of a server. A computer system 800 or other similarly configured computer systems may include and execute one or more subsystem components for performing the functions described herein, including the steps of the various flow processes described above. Similarly, mobile devices, cell phones, smartphones, laptops, desktops, notebooks, tablets, wearable devices, servers, etc. that include some of the same components of the computer system 800 execute an application (or software) and perform the steps and functions described above. The computer system 800 may be connected to a network 814, such as the Internet or other network, to receive queries, obtain data, and transmit information and incentives as described above.

[0060] Computer system 800 typically includes a memory 802, a secondary storage device 804, and a processor 806. Also, computer system 800 may include multiple processors 806 and may be configured as multiple, for example, blade servers or other known server configurations. Additionally, computer system 800 may include a network connection device 808, a display device 810, and an input device 812.

[0061] Memory 802 may include RAM or a similar type of memory and may store one or more applications to be executed by processor 806. Secondary storage device 804 may include a hard disk drive, a floppy disk drive, a CD-ROM drive, or other types of non-volatile data storage. Processor 806 executes applications such as those described herein, which may be stored in memory 802 or secondary storage device 804 or received from the Internet or other network 814. The processing by processor 806 may be implemented in software, such as a software module, for execution on a computer or other machine. These applications preferably include executable instructions for performing the functions and methods of the system and subsystem components described above and shown in the figures of this specification. The applications preferably provide a graphical user interface (GUI) through which a user can browse and interact with the subsystem components.

[0062] Computer system 800 may store, for example, in secondary storage device 804, one or more database structures for storing and maintaining information necessary to perform the functions described above. Alternatively, such information may reside in a storage device separate from these components.

[0063] Also, as described above, the processor 806 may execute one or more software applications to provide the functions described herein, and specifically, may execute the steps and functions of the process flow described above. Such a process may be implemented in software, such as a software module, for execution on a computer or other machine. The GUI may be formatted, for example, as a web page in hypertext markup language (HTML), extensible markup language (XML), or any other format suitable for display on a display device, depending on the application used by the user to interact with the computer system 800.

[0064] The input device 812 may include any device for inputting information into the computer system 800, such as a touch screen, keyboard, mouse, cursor control device, microphone, digital camera, video recorder, or camcorder. The input device 812 may be used to input information into the GUI during the execution of the method described above. The display device 810 may include any type of device for presenting visual information, such as a computer monitor or flat screen display (or the screen of a mobile device). The display device 810 may display the output from the GUI and / or subsystem components (or software).

[0065] Examples of the computer system 800 include dedicated server computers such as blade servers, personal computers, laptop computers, notebook computers, palmtop computers, network computers, mobile devices, or any processor control device capable of executing other types of applications for interacting with a web browser or system.

[0066] Although only one computer system 800 is shown in detail, system 800 can support users using multiple computer systems or servers as needed or desired, and can also use backup or redundant servers to prevent network downtime in the event of a failure of a particular server. Also, although computer system 800 is shown with various components, one of ordinary skill in the art will understand that the system may include additional or different components. Also, although implementations consistent with the above are described as being stored in memory, one of ordinary skill in the art will understand that these implementations can also be stored on or read from other types of computer program products or computer-readable media, such as secondary storage devices like hard disks, floppy disks, CD-ROMs, or other forms of RAM or ROM. The computer-readable media may include instructions for controlling computer system 800 to perform a particular method, such as the methods described above.

[0067] This disclosure is not limited to the specific embodiments described in this application, which are intended as examples of various aspects. Clearly, many modifications and variations can be made without departing from the spirit and scope of this disclosure. In addition to those listed herein, functionally equivalent methods and apparatuses within the scope of this disclosure will become apparent from the foregoing representative description. Such modifications and variations are intended to be included within the scope of the appended representative claims. This disclosure is limited only by the terms of the appended representative claims and the equivalents of all the scopes that such representative claims grant rights to. Also, it should be understood that the terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting.

Claims

1. Displaying a user interface on a smart device, the user interface including a page for displaying a video feed received from a camera of the smart device; Detecting a ground plane in the video feed; Displaying a target at a point in the video feed, the target corresponding to an intersection point where a plane the smart device is oriented towards intersects the ground plane; Determining a distance between the smart device and the intersection point; Detecting an object in the video feed; Selecting a virtual reality object based on the object detected in the video feed; Displaying the virtual reality object in a display mode; Displaying a function of the virtual reality object; Recording a user's reaction to the function of the virtual reality object; Posting the record to a social media account; A method comprising the above.

2. The method according to claim 1, wherein the step of detecting the ground plane includes feature extraction by visual processing.

3. The smart device is configured to determine coordinates of the smart device and coordinates of the intersection point in a 3D space, and the distance is calculated based on the coordinates of the smart device and the coordinates of the intersection point. The method according to claim 1.

4. The smart device stores a series of virtual objects each associated with a tag and the function. The method according to claim 1.

5. The method according to claim 4, wherein the step of selecting the virtual reality object is based on a matching between the name of the detected object and the tag.

6. The smart device stores a profile including name, address, and history information about the user. The method according to claim 1.

7. The method according to claim 6, wherein the step of selecting the virtual reality object is based on the history information included in the profile.

8. The method according to claim 1, wherein the display mode is determined based on data detected by the smart device.

9. The method according to claim 8, wherein the data includes speed.

10. The method according to claim 8, wherein the data includes acceleration.

11. The method according to claim 8, wherein the data includes a humidity indicator.

12. The method according to claim 8, wherein the data includes a fingerprint.

13. The method according to claim 8, wherein the data includes facial features.

14. The method according to claim 1, wherein the function of the virtual reality object is to display an animation.

15. The method according to claim 14, further comprising the step of downloading the animation.

16. The method according to claim 15, further comprising the step of changing the animation to reflect data, wherein the data includes speed or humidity.

17. The method according to claim 15, further comprising the step of changing the animation to reflect high-speed movement or rain.

18. The method according to claim 1, further comprising the step of sending the name of the store to a central server.

19. The method according to claim 18, further comprising the step of receiving the indoor location of the product.

20. The method according to claim 19, further comprising the step of displaying the indoor direction to the product.

Images