Interaction system for augmented reality object

Abstract

The invention relates to an interactive system for augmented reality objects. In systems and methods for providing interactions with and between virtual objects in an augmented reality or mixed reality or virtual reality environment, a detected condition may trigger an animation response from a virtual object placed in a physical environment view. The detected conditions may include detecting that a user is within a set threshold placement distance or proximity, detecting that another virtual object is within a set threshold placement distance or proximity, detecting a particular environmental condition in a physical environment view, and other such factors. A particular behavioral animation of a virtual object may be triggered in response to detecting a particular virtual object and / or other conditions.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201980021740.4, filed on January 29, 2019. Technical Field

[0003] The description generally relates to use of a computing device in an augmented reality (AR) environment and / or a virtual reality (VR) environment. Background Art

[0004] Augmented reality systems, or mixed reality systems, or virtual reality systems can allow users to view scenes, for example, scenes corresponding to their physical environment, and enhance the scenes of their physical environment with virtual objects or virtual characters. Augmented reality (AR) systems and / or virtual reality (VR) systems can generate three-dimensional (3D) immersive augmented and / or virtual reality environments. Users can experience this augmented reality or mixed reality or virtual reality environment by interacting with various electronic devices. Users can move through elements and interact with them in augmented reality, mixed reality, or virtual reality environments by, for example, gestures, manipulation of electronic devices, and / or external devices operably coupled to electronic devices. The user experience in this type of augmented reality or mixed reality or virtual reality experience will be enhanced by close, relevant, and contextual interactions with virtual objects and / or characters placed in a scene view of the physical environment. Such interactions will enhance the reality of the user's augmented reality or mixed reality experience and enhance the user's presence in the augmented reality or mixed reality experience. Summary of the invention

[0005] In one general aspect, a computer-implemented method may include displaying a mixed reality scene by an electronic device, detecting a selection of a first virtual object to be placed in the mixed reality scene, displaying the selected first virtual object at a placement location in the mixed reality scene in response to the detected selection, detecting an animation trigger from a plurality of animation triggers, identifying an animation of the first virtual object from a plurality of animations based on the detected animation trigger, and executing the identified animation of the first virtual object in response to the detected animation trigger.

[0006] In another general aspect, a system may include a computing device configured to generate a mixed reality environment. The computing device may include a memory storing executable instructions; and a processor configured to execute the instructions. Execution of the instructions may cause the computing device to display a mixed reality scene, detect a selection of a first virtual object to be placed in the mixed reality scene, display the selected first virtual object at a placement location in the mixed reality scene in response to the detected selection, detect an animation trigger from a plurality of animation triggers, identify an animation of the first virtual object from the plurality of animations based on the detected animation trigger, and execute the identified animation of the first virtual object in response to the detected animation trigger.

[0007] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figures 1A-1C An exemplary system for generating an augmented reality or mixed reality environment is illustrated.

[0009] FIG. 2A to FIG. 2H Illustrate animation and interaction of virtual objects in an augmented reality or mixed reality environment according to implementations described herein.

[0010] Figures 3A-3D Illustrate animation and interaction of virtual objects in an augmented reality or mixed reality environment according to implementations described herein.

[0011] FIG. 4A to FIG. 4J Illustrate animation and interaction of virtual objects in an augmented reality or mixed reality environment according to implementations described herein.

[0012] Figure 5 is a block diagram of an exemplary system for generating an augmented reality or mixed reality environment.

[0013] Figure 6 is a flowchart of a method for providing interaction of virtual objects in an augmented reality or mixed reality environment according to an embodiment described herein.

[0014] Figure 7 Examples of computer devices and mobile computer devices that can be used to implement the techniques described herein are shown. DETAILED DESCRIPTION

[0015] An augmented reality (AR) system or a mixed reality system may allow a user to place a three-dimensional (3D) virtual object or an augmented reality object or a mixed reality object in a physical environment, and view the 3D virtual object / augmented reality object / mixed reality object, etc., on a display portion of a head-mounted display device, for example, within a camera view. In some embodiments, the 3D virtual / augmented reality / mixed reality objects may interact with the user and with each other in a relatively realistic manner, so that the 3D virtual / augmented reality / mixed reality objects feel as if they physically exist in the user's physical environment, rather than just virtually. In some embodiments, the interaction of the 3D virtual / augmented reality / mixed reality objects with the user and with each other may be contextual, contextually appropriate, or consistent, interactive, and responsive to the surrounding environment and changes in the surrounding environment, so as to further enhance the reality of the placement of the 3D virtual / augmented reality / mixed reality object in the physical environment. Hereinafter, for the purpose of simplifying the description only, the implementation described herein is referred to as a mixed reality system that allows a user to place a 3D virtual object in a physical environment to create a mixed reality environment or mixed reality experience.

[0016] Figures 1A-1C The illustration shows a user having one or more electronic devices that can generate an augmented reality, or a mixed reality, or a virtual environment. Figure 1A As illustrated in the example shown in , in some embodiments, a user can view and experience a mixed reality environment on a display portion 12 of a handheld device 10 that displays a camera view of a physical environment captured by an imaging device 14 of the handheld device 10 and virtual objects placed by the user in the camera view of the physical environment displayed on the display portion 12. Figure 1B As illustrated in the example shown in , in some embodiments, a user can view and experience a mixed reality environment via a display portion of a head mounted device 16. In this arrangement, the physical environment is visible to the user through the head mounted device 16, and virtual objects can be placed by the user and visible on the display portion of the head mounted device 16, which are superimposed on the user's view of the physical environment. Figure 1C As illustrated in the example shown in , in some embodiments, a user can view and experience a mixed reality environment via a display portion of a head mounted device 18 that obscures the user's direct visibility of the physical environment. In this arrangement, a pass-through image of the physical environment can be displayed on the display portion of the head mounted device 18, and virtual objects placed by the user are superimposed on the pass-through image of the physical environment. Figures 1A-1CEach of the handheld device 10 and the head mounted devices 16 and 18 illustrated in the figure is an electronic device including a display device capable of displaying a virtual reality object in an augmented reality or mixed reality environment. In the following, only for the convenience of discussion and explanation, an example of a system for providing object interaction in a mixed reality environment according to the implementation described herein will be based on the example of the system for providing object interaction in a mixed reality environment according to the implementation described herein. Figure 1A However, the principles to be described herein may be applied to other electronic devices and / or systems capable of generating and presenting a mixed reality environment in which a user may place 3D virtual objects, interact with the virtual objects, and / or interact among the virtual objects.

[0017] Figure 2A and 2B is the third-person perspective of the user in the physical environment 1000. Figure 2A and Figure 2B In FIG. 1 , a user is at position 1000A in physical environment 1000. Position 1000A of the user in physical environment 1000 may correspond in this example to the position of an imaging device or camera of an electronic device held by the user. Mixed reality scene 100 (e.g., corresponding to a camera view of physical environment 1000) may be captured by an imaging device of the electronic device. Mixed reality scene 100 may be displayed, for example, on a display device of the electronic device for viewing by the user. Figure 2B , for ease of discussion and illustration only, a mixed reality scene 100 of a physical environment 1000 is illustrated in an enlarged state separated from an electronic device. The mixed reality 100 visible to a user via an electronic device may represent a portion of the physical environment 1000 captured within the field of view of an imaging device of the electronic device.

[0018] The user can place the first virtual object A in the mixed reality scene 100, such as Figure 2C As shown in Figure 2C In the example shown in , the first virtual object A is a first virtual character A, which is positioned at a position 110 on the ground, which corresponds to a first distance D1 from the user at the position 1000A. In this example, the distance D1 may represent the distance between the user at the position 1000A and the placement position 110 of the first character A when the user places the first character A. In some cases, the user may move in the physical environment 1000. As the user approaches the placement position 110 of the character A, the distance between the user and the placement position 110 of the character A is reduced, and the system may detect that the user is approaching the character A. For example, the system may detect that the user is within a threshold distance D2 of the placement position 110 of the character A. In other words, the system may detect that the distance between the user and the placement position 110 of the character A is less than or equal to the threshold distance D2.

[0019] The system may treat the detection of the user being within a threshold distance of character A as an animation trigger. In response to detecting that the user is within a threshold distance D2 of the placement location 110 of character A (i.e., in response to the detected animation trigger), the system may cause character A to react to the user's approach in an appropriate, natural, or expected manner. For example, in some embodiments, in response to detecting that the user is within a threshold distance D2 of the placement location 110 of character A (i.e., in response to the detected animation trigger), the system may animate character A by, for example, causing character A to react by turning to face the user, such as Figure 2D as shown in .

[0020] Similarly, in some embodiments, the user may place the first virtual character A at the first placement position 110, such as Figure 2E As shown in and as described above, and then a second virtual object B in the form of, for example, a second virtual character B is placed at a second placement position 120, for example, near the first virtual character A, such as Figure 2F As shown in . The system can detect the placement of the second character B within a threshold placement distance D3 of the first character A. The system can use the detection of the first character A and the second character B being within a threshold placement distance D3 of each other as an animation trigger, and can animate the first character A and / or the second character B, or cause the first character A and / or the second character B to react in an appropriate manner. That is, the animation trigger of detecting that the first character A and the second character B are within a threshold placement distance D3 of each other can cause the system to perform an animation of at least one of the first character A or the second character B. For example, in response to detecting that the second character B is within the threshold placement distance D3 of the first character A, the system can cause the first character A to turn toward the second character B, as in Figure 2G , and / or may cause the first character A and the second character B to turn to face each other. In some embodiments, the animation of the first character A and / or the second character B may continue beyond this type of recognition of the second character B within a threshold distance of the first character A. For example, the first character A and the second character B may continue to interact in an appropriate, natural, or expected manner by, for example, talking, moving to a new location 110A, 120A, etc., as in Figure 2H In some implementations, in response to a detected animation trigger / detection of a user being within a set threshold distance of the first character A's location 110 and / or a set threshold distance of the second character B's location 120, the system may cause the first character A and / or the second character B to react to the detection of the user within the threshold distance in an appropriate manner, as described above.

[0021] Figures 2A-2HEach sequential illustration of the mixed reality scene 100 shown in FIG. 1 (in some embodiments, corresponding to a camera view 100 captured by an imaging device of the electronic device) is a still image representing a snapshot at a point in a particular animation and / or interaction of the first and second virtual characters A and B. However, it is understood that the animation or interaction displayed by the first and second virtual characters A and B in the mixed reality scene 100 can be a substantially continuous dynamic animation of a 3D virtual object, or an intermittent dynamic animation of a 3D virtual object to provide a relatively realistic representation of the movement of the virtual characters.

[0022] The location of the electronic device (and the imaging device or camera of the electronic device) in the physical environment may be known and tracked by the system. The known / tracked location of the electronic device / camera in the physical environment may substantially correspond to the location of the user in the physical environment. The system may also know the placement of virtual objects (as described above with respect to Figures 2A-2H ). In this example, the placement position of the virtual object can be a placement position in the mixed reality scene that corresponds to a physical position in the physical environment (in some implementations, corresponds to a camera view of the physical environment). This correspondence between the placement position of each virtual object in the mixed reality scene and the physical position in the physical environment can allow the system to detect the distance of the electronic device (i.e., the user) relative to the virtual object placed in the mixed reality scene including the physical environment and / or its position and / or orientation. This correspondence between the placement position of each virtual object and the physical position in the physical environment can also allow the system to detect the distance between different virtual objects placed in the mixed reality scene including the physical environment and / or the relative positioning and / or relative orientation of the different virtual objects.

[0023] In some embodiments, such detection and / or tracking of the position of each virtual object in the mixed reality scene of the physical environment, and the detection and / or tracking of the position of the electronic device / user can be based on the corresponding individual three-dimensional coordinate positions of the virtual object and the electronic device / user. For example, each virtual object in the mixed reality scene of the physical environment can have an associated three-dimensional coordinate position, for example, an associated (x, y, z) coordinate position. The (x, y, z) coordinate position of each virtual object in the mixed reality scene can correspond to a physical three-dimensional (x, y, z) coordinate position in the physical environment. Similarly, the electronic device / user can have an associated three-dimensional (x, y, z) coordinate position in the physical environment. When the mixed reality scene is updated to reflect the movement of the electronic device / user, the movement / animation of the virtual object, etc., the corresponding three-dimensional (x, y, z) coordinate positions of the virtual object and the electronic device / user can be updated intermittently, or substantially continuously. When the detected three-dimensional (x, y, z) coordinate position of the virtual object and the detected three-dimensional (x, y, z) coordinate position of the electronic device / user are updated, the corresponding detected three-dimensional (x, y, z) coordinate position can be used to calculate the distance between the electronic device / user and the virtual object and / or between the virtual objects, and update the calculated distance. The system can use the distance calculated in this manner to determine when the electronic device / user has moved to a position within a set threshold distance of the virtual object or has moved to a position outside the set threshold distance of the virtual object, when a second virtual object is within a set threshold placement distance of the first virtual object or has moved out of the set threshold placement distance of the second virtual object, etc.

[0024] When virtual objects, such as, for example, virtual characters, are placed in a mixed reality scene including a physical environment as described above (particularly in a camera view of the physical environment as described above), interaction with the user and / or each other can make the virtual objects feel more present and more realistic. Virtual objects can follow an event-based interaction system, including a set of animations, triggers, and effects determined based on, for example, a set of previously defined contextual rules. In some embodiments, a set of properties can be associated with each virtual object placed in a mixed reality scene. A detected animation trigger / trigger condition may cause the system to execute a selected or identified animation based on the type of trigger condition encountered.

[0025] In some embodiments, the property set may include, for example, an animation property set. The animation property set may include, for example, a plurality of different animations that a particular object may support or may perform in response to a detected animation trigger, condition, input, etc. In some embodiments, a plurality of different animations may be associated with each virtual object placed in a physical environment. The system may select or identify one of the animations from the plurality of animations based on a detected animation trigger or triggering condition for the virtual object to perform in a particular situation or environment.

[0026] For example, a placement animation may be performed by a virtual object when placed in a mixed reality scene that includes a physical environment to demonstrate the feeling of arriving at the placement location in the mixed reality scene. Figure 3A In the example illustrated in , a virtual object C is placed on a real table represented in a mixed reality scene 100 (or a camera view 100) including a physical environment 1000. When the virtual object C is placed on the table, for example, the virtual object C may open its eyes, look around to provide an effect of being familiar with its surroundings, and so on.

[0027] When a virtual object is displayed in a mixed reality scene, the virtual object can perform idle animations, but without any input, conditions, etc. (e.g., detected user approach) that would trigger a specific reaction or behavior. Figure 3B In the example illustrated in , without detecting the presence of, for example, an electronic device / camera / user and / or another virtual object and / or other conditions that would trigger a specific response from the virtual object C, the virtual object C moves back and forth along the table surface, consistent with the idle behavior characteristics of the particular virtual object C (i.e., the idle behavior that might be expected of a crab placed on a tabletop). Executing this idle animation of the virtual object C without the presence of a specific condition that triggers a specific interaction can provide a more realistic effect, thereby causing the virtual object C to appear more in the physical environment 1000.

[0028] A virtual object may be displayed when, for example, the system detects the proximity of a user and / or another virtual object (i.e., the electronic device / camera / user and / or other virtual object is detected to be within a set threshold distance of the placement location of the virtual object, such as Figure 2D , Figure 2G as well as Figure 2H In some embodiments, a specific or customized, affected / interactive animation may be performed by a virtual object in a specific situation / in response to a specific animation trigger. A specific situation may include, for example, a situation where a first virtual object and a second virtual object are in some related manner and a certain reaction is expected in the presence of the first and second virtual objects. Figure 3CIn the example illustrated in , virtual object C animates in response to detecting that the electronic device / camera / user is within a set threshold distance D2. In this example, virtual object C turns to face the electronic device / camera / user and snaps its claws (which may be what the crab wants in this case). Similarly, in Figure 3D In the example illustrated in , both virtual object C and virtual object D animate in response to detecting that each other is within a set threshold placement distance D3. In this example, the animations of virtual object C and virtual object D involve fighting or grabbing each other (which may happen in this case).

[0029] In some implementations, the set of properties may include behavioral properties, such as, for example, logic that triggers a particular action or animation performed by the virtual character in response to a detected condition or input. The set of behavioral properties may be contextual, in that the actions and / or animations triggered are appropriate and / or expected and / or convincing given the type of virtual objects involved, the relationship between the virtual objects and the user, the mixed reality scene or camera view in which the virtual objects are placed, and so on, e.g., Figure 3C and Figure 3D These contextual behavior attributes or characteristics can further enhance the lifelike nature of virtual objects in the mixed reality scene.

[0030] For example, in some embodiments, detecting an electronic device or camera position (substantially corresponding to the user's position) within a set threshold distance or a set threshold proximity of a virtual object's placement position may trigger a set of scripted behaviors or scripted animations that cause the virtual object to interact with the user in a contextually appropriate manner as described above. This may include, for example, looking in the direction of the detected electronic device / camera / user (as described above with respect to Figure 2D and 3C ), waving, jumping and other forms of confirmation and interaction with the user. Similarly, the detected movement of the electronic device / camera position / user position from within a set threshold distance to outside a set threshold distance can trigger different sets of scripted behaviors or animations of virtual objects.

[0031] Likewise, in some embodiments, detecting that a second virtual object is within a set threshold placement distance or a set threshold placement proximity of a placement location of a first virtual object can trigger a set of scripted behaviors or scripted animations, thereby causing the first virtual object and the second virtual object to appear to interact with each other in a contextually appropriate manner. This can include, for example, turning to look at each other (as described above with respect to Figure 2G described), moving towards each other / gesturing to each other (as described above about Figure 2Gdescribed above), the second virtual object follows the gaze of the first object when the first object moves, the second virtual object's action on the first virtual object (as described above with respect to Figure 3D The first virtual object may be configured to perform appropriate / contextual reactions, laugh, fight, dance, and other forms of confirmation and interaction that would be convincing and / or expected interactions between the first and second virtual objects. Similarly, detecting that the second virtual object moves from within a set threshold distance to outside the set threshold distance may trigger different sets of scripted behaviors or animations for the first virtual object.

[0032] As mentioned above, Figure 3A-3D Each sequential illustration of the mixed reality scene 100 shown in the figure or the camera view 100 in some embodiments is a still or static two-dimensional image representing a snapshot of a certain point in a particular animation and / or interaction of the first and second virtual characters C and D. However, it is understood that the animation, or the interaction displayed by the first and second virtual characters C and D in the mixed reality scene 100 can be a substantially continuous dynamic animation of a 3D virtual object, or can be an intermittent dynamic animation of a 3D virtual object, thereby providing a relatively realistic representation of the movement of the virtual characters.

[0033] A collection of scripted behaviors or animations may more simply be randomly executed by virtual objects in a mixed reality scene or camera view of a physical environment, without regard to camera / user position and / or orientation, and / or virtual object position and / or orientation. However, this approach does not result in realistic or convincing effects. Combining these inputs related to the relative position / orientation of the electronic device / camera / user and virtual objects with these sets of properties into a system may make virtual objects appear more convincingly in a mixed reality scene including physical objects when virtual objects interact with users and / or each other, similar to how real people, real animals, and other such objects interact in similar environments and when encountering similar situations. Reactions, responses, behaviors, interactions, etc. triggered in this manner can provide a realistic, convincing mixed reality environment, thereby enhancing the user experience.

[0034] Another example of placement of virtual objects in the camera view 310 of the physical environment and interaction with a virtual character selected from the gallery 330 of virtual characters is shown in FIG. Figures 4A-4J 3. In this particular example, the electronic device 300 may include a display 320, and an imaging device 340 or camera 340. A mixed reality scene 310 or camera view 310 including a physical environment may be displayed on the display 320 based on images captured by the camera 340. The gallery 330 may include a plurality of virtual objects or virtual characters for the user to select and place in the mixed reality scene 310 or camera view 310 including a physical environment displayed on the display 320.

[0035] The first virtual character 330A may be selected by the user, such as Figure 4B , and is placed at a first placement position 310A in the mixed reality scene 310 or camera view 310, as shown in Figure 4C and Figure 4D Then, the user can select the second virtual character 330B, as shown in Figure 4E , for placement at a second placement position 310B in the mixed reality scene 310 or the camera view 310, such as Figure 4F The system can detect the placement of the second virtual character 310B within a threshold placement distance of the first virtual character 310A and interpret it as an animation trigger. That is, Figure 4F As shown in , the first virtual character 330A can react to the placement of the second virtual character 330B, for example, by turning to the second virtual character 330B, smiling, etc. Thus, the detected animation trigger can cause identification of an animation for at least one of the first virtual character 310A or the second virtual character 310B. Other virtual characters can be added to the mixed reality scene 310 or the camera view 310 in a manner similar to that described above. Likewise, virtual characters can be removed from the mixed reality scene 310 or the camera view 310 in a similar manner.

[0036] One or both of the first virtual character 330A and / or the second virtual character 330B may be animated in response to the detected placement distance between the first virtual character 330A and the second virtual character 330B being less than the threshold placement distance D3 (i.e., in response to detecting this animation trigger). The system may select or identify the type of animation to be performed by the first and / or second virtual characters 310A and 310B from a plurality of animations based on, for example, scripted behavior, trigger conditions, etc. In some embodiments, the distance between the first virtual character 330A and the second virtual character 330B may be less than or equal to the set threshold placement distance at the time of initial placement in the camera view 310 of the physical environment. In some embodiments, the user may move one or both of the first virtual character 330A and / or the second virtual character 330B to move the first virtual character and the second virtual character within the set threshold distance D3 so that one of the first virtual character 330A or the second virtual character 330B is animated.

[0037] exist Figure 4G-4JIn the example shown in , in response to detecting that the first and second virtual characters 330A and 330B are at a distance less than or equal to the set threshold placement distance D3 (e.g., a first animation trigger), the second virtual character 330B is animated. In this example, the animation of the second virtual character 330B includes dancing. In response to the animation (dancing) of the second virtual character 330B (e.g., a second animation trigger), the first virtual character 330A reacts to the scene animation in the form of clapping in this example. In some embodiments, the orientation and / or size or scaling of the virtual character in the camera view 310 can be adjusted. In some embodiments, the adjustment of the orientation and / or size / scaling ratio can be incorporated into the animation of the virtual character. In some embodiments, the adjustment of the orientation and / or size / scaling ratio can be implemented by user input. Figure 4H An example is illustrated in which the orientation of the second virtual character 330B is adjusted. Figure 4H and Fig. 4I An example of adjusting the size or scale of the second virtual character 330B is shown.

[0038] As mentioned above, Figures 4A-4J Each sequential illustration of the mixed reality scene 310 or camera view 310 shown in the figure is a still image representing a snapshot at a certain point in a particular animation and / or interaction of the first and second virtual characters 330A and 330B. However, it is understood that the animation or interaction displayed by the first and second virtual characters 330A and 330B in the mixed reality scene 310 or camera view 310 can be a substantially continuous dynamic animation of a 3D virtual object, or an intermittent dynamic animation of a 3D virtual object to provide a relatively realistic representation of the movement of the virtual characters.

[0039] Figure 5 is a block diagram of an electronic device 500 according to an embodiment described herein, which can generate an augmented reality or mixed reality environment and provide interaction with a virtual character. Figure 5As shown in , the electronic device may include a user interface system 520, which includes an output device and an input device. The output device may include, for example, a display for visual output, a speaker for audio output, etc. The input device may include, for example, a touch input device that can receive tactile user input, a microphone that can receive audible user input, etc. The electronic device 500 may also include a sensing system 540. The sensing system 540 may include, for example, a light sensor, an audio sensor, an image sensor / imaging device or a camera, a distance / proximity sensor, a position sensor, and / or a different combination of other sensors and / or sensors. Some sensors included in the sensing system 540 may provide position detection and tracking of the electronic device 500. Some sensors of the sensing system 540 may provide capture of images of the physical environment to be displayed on components of the user interface system 520. The electronic device 500 may also include a control system 580. The control system 580 may include, for example, a power control device, an audio and video control device, an optical control device, and / or other such devices, and / or a different combination of devices. The user interface system 520 and / or the sensing system 540 and / or the control system 580 may include more or fewer devices, depending on the particular implementation, and may have a different physical arrangement than shown. The electronic device 500 may also include a processor 590, a memory 585, and a communication module 595 that communicate with the user interface system 520, the sensing system 540, and the control system 580. The communication module 595 may provide communication between the electronic device 500 and other external devices.

[0040] exist Figure 6 6 shows a method 600 for providing interaction of virtual objects in an augmented or mixed reality environment according to an embodiment described herein. A user may initiate an augmented reality, mixed reality, or virtual reality experience using, for example, an electronic device to display a mixed reality scene including a view of a physical environment (block 610). The mixed reality scene including a view of the physical environment may be, for example, a camera view of the physical environment captured by an imaging device of the electronic device and displayed on a display device of the electronic device. One or more virtual objects may be placed in the mixed reality scene including the physical environment (block 620). One or more virtual objects may be placed in the mixed reality scene or camera view including the physical environment based on selections made by the user and displayed on the display device of the electronic device, superimposed on the mixed reality scene, or on the camera view.

[0041] In response to detecting a trigger for interaction (block 630), a virtual object displayed in a mixed reality scene or camera view may be animated for interaction based on the type of the detected trigger (block 640). In some embodiments, detecting that an electronic device is within a set threshold distance or proximity of a placement location of a virtual object may trigger an animation to interact with a user. In some embodiments, detecting that a new virtual object is within a set threshold distance or proximity of a placement location of an existing virtual object may trigger an animation for interaction between the new virtual object and the existing virtual object. In some embodiments, characteristics associated with the new virtual object may trigger a specific animation for a specific type of interaction between the new virtual object and the existing virtual object. In some embodiments, conditions in the environment where the virtual object is placed may trigger an animation of the virtual object. In some embodiments, the animation of the virtual object may be implemented according to a set of rules. The process may continue until it is determined that a trigger is no longer detected for interaction (block 650) and / or the augmented reality / mixed reality experience has terminated (block 660).

[0042] Figure 7 An example of a computer device 700 and a mobile computer device 750 that can be used with the technology described herein is shown. The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 connected to the memory 704 and a high-speed expansion port 710, and a low-speed interface 712 connected to a low-speed bus 714 and the storage device 706. Each of the components 702, 704, 706, 708, 710, and 712 is interconnected using various buses and can be installed on a general motherboard or installed in other ways as appropriate. The processor 702 can process instructions for execution within the computing device 700, including instructions stored in the memory 704 or the storage device 706 to display graphical information for a GUI on an external input / output device (such as a display 716 coupled to the high-speed interface 708). In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, can be used appropriately. Furthermore, multiple computing devices 700 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0043] The memory 704 stores information within the computing device 700. In one embodiment, the memory 704 is one or more volatile memory units. In another embodiment, the memory 704 is one or more non-volatile memory units. The memory 704 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0044] The storage device 706 can provide mass storage for the computing device 700. In one embodiment, the storage device 706 can be or include a computer readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state storage device, or a series of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that perform one or more methods as described above when executed. The information carrier is a computer or machine readable medium, such as a memory 704, a storage device 706, or a memory on a processor 702.

[0045] The high-speed controller 708 manages bandwidth-intensive operations of the computing device 700, while the low-speed controller 712 manages less bandwidth-intensive operations. This allocation of functions is exemplary only. In one embodiment, the high-speed controller 708 is coupled to the memory 704, the display 716 (e.g., through a graphics processor or accelerator), and is coupled to the high-speed expansion port 710 that can accept various expansion cards (not shown). In an embodiment, the low-speed controller 712 is coupled to the storage device 706 and the low-speed expansion port 714. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router through a network adapter.

[0046] As shown, computing device 700 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 720, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 724. In addition, it can also be implemented in a personal computer such as a laptop computer 722. Alternatively, components from computing device 700 can be combined with other components in a mobile device (not shown), such as device 750. Each such device can contain one or more computing devices 700, 750, and the entire system can be composed of multiple computing devices 700, 750 that communicate with each other.

[0047] Computing device 750 includes, among other components, a processor 752, a memory 764, an input / output device such as a display 754, a communication interface 766, and a transceiver 768. A storage device, such as a microdrive or other device, may also be provided to device 750 to provide additional storage. Each of components 750, 752, 764, 754, 766, and 768 are interconnected using various buses, and multiple components may be mounted on a common motherboard or otherwise as appropriate.

[0048] Processor 752 can execute instructions within computing device 750, including instructions stored in memory 764. The processor can be implemented as a chipset including separate and multiple analog and digital processor chips. For example, the processor can provide coordination of other components of device 750, such as control of a user interface, applications running on device 750, and wireless communications of device 750.

[0049] The processor 752 can communicate with the user through the control interface 758 and the display interface 756 coupled to the display 754. For example, the display 754 can be a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other appropriate display technology. The display interface 756 may include appropriate circuits for driving the display 754 to present graphics and other information to the user. The control interface 758 can receive commands from the user and convert them to submit to the processor 752. In addition, an external interface 762 in communication with the processor 752 can be provided to enable near-field communication of the device 750 with other devices. For example, the external interface 762 can provide wired communication in some embodiments, or wireless communication in other embodiments, and multiple interfaces can also be used.

[0050] The memory 764 stores information within the computing device 750. The memory 764 may be implemented as one or more computer-readable media, one or more volatile storage units, or one or more non-volatile storage units. An expansion memory 774 may also be provided and connected to the device 750 via an expansion interface 772, for example, the expansion interface 772 may include a SIMM (single in-line memory module) card interface. Such an expansion memory 774 may provide additional storage space for the device 750, or may also store applications or other information for the device 750. Specifically, the expansion memory 774 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Thus, for example, the expansion memory 774 may be provided as a security module for the device 750, and may be programmed with instructions that allow the secure use of the device 750. In addition, a security application may be provided via a SIMM card and additional information, such as placing identification information on the SIMM card in an inviolable manner.

[0051] For example, the memory may include flash memory and / or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more of the methods described above. The information carrier is a computer or machine readable medium such as memory 764, expansion memory 774, or memory on processor 752, which may be received, for example, via transceiver 768 or external interface 762.

[0052] The device 750 can communicate wirelessly via a communication interface 766, which may include digital signal processing circuitry when necessary. The communication interface 766 can provide communications under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA200 or GPRS, etc. For example, such communications can be performed via a radio frequency transceiver 768. In addition, short-range communications can be performed, such as using Bluetooth, Wi-Fi or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module 770 can provide additional navigation and location-related wireless data to the device 750, which can be used appropriately by applications running on the device 750.

[0053] Device 750 may also communicate auditorily using an audio codec 760 that may receive voice information from a user and convert it into usable digital information. Audio codec 760 may also generate audible sounds for the user, such as through a speaker, for example, in an earpiece of device 750. Such sounds may include sounds from a voice phone call, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 750.

[0054] As shown, computing device 750 can be implemented in many different forms. For example, it can be implemented as a cellular phone 780. It can also be implemented as part of a smart phone 782, a personal digital assistant, or other similar mobile devices.

[0055] Various implementations of the systems and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor, which can be special purpose or general purpose, coupled to a storage system, at least one input device, and at least one output device to receive data and instructions, and coupled to a storage system, at least one input device, and at least one output device to send data and instructions.

[0056] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine language. As used herein, the terms "machine-readable medium," "computer-readable medium," refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0057] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) to display information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including sound, voice, or tactile input.

[0058] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0059] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between a client and a server is generated by computer programs running on the respective computers and having a client-server relationship to each other.

[0060] In some embodiments, Figure 7 The computing device depicted in FIG. 7 may include sensors that interface with a virtual reality (VR headset / HMD device 790). For example, including Figure 7One or more sensors on the computing device 750 or other computing device depicted in the VR headset 790 can provide input to the VR headset 790, or generally, can provide input to the VR space. Sensors can include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 750 can use sensors to determine the absolute position and / or detected rotation of the computing device in the VR space, which can then be used as input to the VR space. For example, the computing device 750 can be incorporated into the VR space as a virtual object such as a controller, laser pointer, keyboard, weapon, etc. When a user incorporates the computing device / virtual object into the VR space, its position can allow the user to position the computing device so as to view the virtual object in the VR space in a certain manner. For example, if the virtual object represents a laser pointer, the user can manipulate the computing device like an actual laser pointer. The user can move the computing device left and right, up and down, in a circle, etc. and use the device in a similar manner to using a laser pointer.

[0061] In some implementations, one or more input devices included in or connected to the computing device 750 can be used as input to the VR space. The input devices can include, but are not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, headphones or earbuds with input capabilities, a game controller, or other connectable input devices. When the computing device is incorporated into the VR space, a user interacting with the input devices included on the computing device 750 causes specific actions to occur in the VR space.

[0062] In some implementations, the touch screen of computing device 750 may be rendered as a touch pad in the VR space. A user may interact with the touch screen of computing device 750. For example, in VR headset 790, the interactions are rendered as movements on the touch pad rendered in the VR space. The rendered movements may control virtual objects in the VR space.

[0063] In some embodiments, one or more output devices included on the computing device 750 can provide output and / or feedback to a user of the VR headset 790 in the VR space. The output and feedback can be visual, tactile, or audible. The output and / or feedback can include, but are not limited to, vibration, turning on, off, or flashing and / or flashing one or more lights or strobe lights, sounding an alarm, playing a ringtone, playing a song, and playing an audio file. The output device can include, but is not limited to, a vibration motor, a vibration coil, a piezoelectric device, an electrostatic device, a light emitting diode (LED), a strobe light, and a speaker.

[0064] In some embodiments, computing device 750 may appear as another object in a computer-generated 3D environment. User interactions with computing device 750 (e.g., rotating, shaking, touching a touch screen, sliding a finger on a touch screen) may be interpreted as interactions with objects in the VR space. These actions may change the content display for a single user or multiple users (e.g., actions by a teacher or tour guide may change the content display for all users participating in a virtual tour or field trip). In the example of a laser pointer in VR space, computing device 750 appears as a virtual laser pointer in a computer-generated 3D environment. As the user manipulates computing device 750, the user in the VR space sees the movement of the laser pointer. The user receives feedback from the interaction with computing device 750 in the VR environment on computing device 750 or VR headset 790.

[0065] In some embodiments, the computing device 750 may include a touch screen. For example, a user may interact with the touch screen in a particular way that may mimic what is happening on the touch screen and what is happening in the VR space. For example, a user may use a pinch-type motion to zoom content displayed on the touch screen. This pinch-type motion on the touch screen may cause the information provided in the VR space to be zoomed. In another example, the computing device may be rendered as a virtual book in a computer-generated 3D environment. In the VR space, pages of a book may be displayed in the VR space, and a swipe of a user's finger across the touch screen may be interpreted as turning / flipping pages of the virtual book. As each page is turned / flipped, in addition to seeing the page content change, audio feedback may also be provided to the user, such as the sound of turning pages in a book.

[0066] In some implementations, in addition to the computing device, one or more input devices (e.g., a mouse, a keyboard) may be rendered in the computer-generated 3D environment. The rendered input devices (e.g., a rendered mouse, a rendered keyboard) may be used when rendered in the VR space to control objects in the VR space.

[0067] Computing device 700 is intended to represent various forms of digital computers and devices, including but not limited to laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 750 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are exemplary only and are not meant to limit the embodiments of the inventions described and / or claimed herein.

[0068] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the specification.

[0069] In addition, the logic flows depicted in the accompanying drawings do not require the particular order shown, or the sequential order to achieve the desired results. In addition, other steps can be provided or steps can be removed from the described flow, and other components can be added to or removed from the described system. Therefore, other embodiments are within the scope of the appended claims.

[0070] Although certain features of the described embodiments have been described as described herein, those skilled in the art will now expect many modifications, substitutions, changes and equivalents. Therefore, it should be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. It should be understood that they are given only in an exemplary and non-limiting manner, and various changes in form and detail may be made. Except for mutually exclusive combinations, any part of the apparatus and / or method described herein may be combined in any combination. The embodiments described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different embodiments described.

Claims

1. A computer-implemented method comprising: Displaying a first virtual object at a first location in the mixed reality scene; Detect animation triggers, including: detecting a second virtual object at a second location in the mixed reality scene; detecting a first position in a physical environment corresponding to the first position of the first virtual object in the mixed reality scene; detecting a second position in the physical environment corresponding to the second position of the second virtual object in the mixed reality scene; and responsive to detecting that a distance between the first position and the second position is less than or equal to a threshold distance, detecting the animation trigger; In response to detecting the animation trigger, identifying an animation for at least one of the first virtual object or the second virtual object; and In response to detecting the animation trigger and identifying the animation, executing the animation.

2. The computer-implemented method of claim 1 , wherein: Identifying the animation and executing the animation include: In response to determining that the distance is within the threshold distance, identifying an interactive animation; and The interactive animation is performed so that the at least one of the first virtual object or the second virtual object performs a context interaction in response to determining that the distance is within the threshold distance.

3. The computer-implemented method of claim 1 , wherein: Detecting the animation trigger, identifying the animation, and executing the animation include: In response to determining that the distance is within the threshold distance, identifying an affected animation; and The affected animation is executed so that at least one of the first virtual object or the second virtual object performs a context interaction with the other of the first virtual object or the second virtual object in response to detecting that the first virtual object and the second virtual object are within the threshold distance of each other.

4. The computer-implemented method of claim 3, wherein: The affected animations include: Execute an affected animation of at least one of the first virtual object or the second virtual object, the affected animation comprising a plurality of contextual behavior attributes associated with a relationship between the first virtual object and the second virtual object and associated with an environment associated with the mixed reality scene.

5. The computer-implemented method of claim 1 , wherein: Detecting a distance between the first position of the first virtual object and the second position of the second virtual object includes: detecting a three-dimensional coordinate position of the first virtual object in the physical environment corresponding to the first position of the first virtual object in the mixed reality scene; detecting a three-dimensional coordinate position of the second virtual object in the physical environment corresponding to the second position of the second virtual object in the mixed reality scene; and Based on the three-dimensional coordinate position of the first virtual object and the three-dimensional coordinate position of the second virtual object, a distance between the first virtual object and the second virtual object is detected.

6. The computer-implemented method of claim 1 , wherein: Detecting the animation trigger, identifying the animation, and executing the animation include: Responsive to displaying the first virtual object in the mixed reality scene, identifying an idle animation; and The idle animation of the first virtual object is performed, the idle animation of the first virtual object corresponding to an idle behavior characteristic associated with the first virtual object.

7. The computer-implemented method of claim 1 , further comprising displaying, by a computing device, a mixed reality scene, comprising: capturing, by an imaging device of the computing device, an image of a physical environment; displaying an image of the physical environment on a display device of the computing device; as well as The first virtual object is displayed superimposed on the image of the physical environment at a location in the mixed reality scene that corresponds to a physical location in the physical environment.

8. A system comprising: A computing device configured to generate a mixed reality environment, the computing device comprising: a memory storing executable instructions; and a processor configured to execute the instructions to cause the computing device to: Display mixed reality scenes; detecting placement of a first virtual object at a first location in the mixed reality scene; detecting placement of a second virtual object at a second location in the mixed reality scene; Detect animation triggers, including: detecting a three-dimensional coordinate position of the first virtual object in a physical environment corresponding to the first position of the first virtual object in the mixed reality scene; detecting a three-dimensional coordinate position of the second virtual object in the physical environment corresponding to the second position of the second virtual object in the mixed reality scene; Detecting a distance between the first virtual object and the second virtual object based on the three-dimensional coordinate position of the first virtual object and the three-dimensional coordinate position of the second virtual object; responsive to determining that a distance between the first virtual object and the second virtual object is within a threshold distance, detecting the animation trigger; In response to detecting the animation trigger, identifying an animation for the first virtual object; and In response to detecting the virtual trigger, an animation of the first virtual object is performed.

9. The system according to claim 8, wherein: The processor is configured to execute the instructions to cause the computing device to: In response to determining that the distance is within the threshold distance, identifying an interactive animation; as well as The interactive animation is executed to enable at least one of the first virtual object or the second virtual object to perform context interaction.

10. The system according to claim 8, wherein: The processor is further configured to execute the instructions to cause the computing device to: In response to detecting placement of the second virtual object, identifying an affected animation; as well as The affected animation is executed so that at least one of the first virtual object or the second virtual object performs a contextual interaction with the other of the first virtual object or the second virtual object in response to detecting that the first virtual object and the second virtual object are within a threshold placement distance of each other.

11. The system according to claim 10, wherein: The processor is further configured to execute the instructions to cause the computing device to execute the affected animation, wherein the affected animation includes multiple contextual behavior attributes, and the multiple contextual behavior attributes are associated with the relationship between the first virtual object and the second virtual object and are associated with an environment associated with the mixed reality scene.

12. The system according to claim 8, wherein: The processor is further configured to execute the instructions to cause the computing device to: activating an imaging device of the computing device and capturing an image of the physical environment; displaying, on a display device of the computing device, an image of the physical environment captured by the imaging device; as well as The first virtual object is displayed superimposed on the image of the physical environment on the display device at a location in the mixed reality scene corresponding to a physical location in the physical environment.

Images