Touch-based augmented reality experience

Abstract

The present disclosure relates to methods and systems for providing a touch-based augmented reality (AR) experience. During the capture phase, the first user may grasp the object. The strength of the force exerted on the object during the grip and / or the duration of the grip may be recorded. A volumetric representation of the first user gripping object may also be captured. During the experience phase, the second user may touch the object, which may provide haptic feedback (e.g., vibration) to the second user at an intensity and duration corresponding to the intensity of force exerted on the object and the duration of gripping by the object. If a volumetric representation of the object gripped by the first user is captured, touching the object may also cause presentation of a volumetric volume of the first user gripping the object.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. application serial number 18 / 056,142, filed on November 16, 2022, which claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 402,905, entitled “TOUCH-BASED AUGMENTED REALITY EXPERIENCE,” filed on August 31, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to mobile and wearable computing technologies. In particular, example embodiments of the present disclosure relate to systems, methods, devices, and user interfaces for providing a touch-based augmented reality (AR) experience that enables users to interact with smart objects. Background Art

[0004] An augmented reality (AR) experience includes applying virtual content to a real-world environment, whether by presenting the virtual content through a transparent display through which the real-world environment is visible, or by enhancing image data to include virtual content superimposed on the real-world environment depicted therein. The virtual content may include one or more AR content items. AR content items may include audio content, visual content, or visual effects. Examples of audio content and visual content include pictures, text, logos, animations, and sound effects. Devices that support AR experiences in any of these ways are referred to herein as "AR devices."

[0005] For some example AR devices, audio and visual content or visual effects are applied to media data such as live image streams. Other example AR devices include head-mounted devices that can be implemented with transparent displays or translucent displays, through which users of the head-mounted devices can view the surrounding environment. Such devices enable users to view the surrounding environment through the transparent or translucent display, and can also see objects (e.g., virtual objects such as 3D renderings, images, videos, text, etc.) generated for display as part of the surrounding environment and / or superimposed on the surrounding environment. Users of head-mounted devices can access and use computer software applications to perform various tasks or participate in entertainment activities. In order to use the computer software application, the user interacts with the 3D user interface provided by the head-mounted device.

[0006] The so-called "Internet of Things" or "IoT" is a network of physical objects (referred to as "smart devices" or "IoT devices") that are embedded with sensors, software, and other technologies that enable connection and exchange of data with other devices via the Internet. For example, IoT devices are used for home automation to control lighting, heating and air conditioning, media and security systems, and camera systems. Several IoT-enabled devices have been provided that function as smart home hubs that connect different smart home products. IoT devices have also been used in several other applications. Application layer protocols and supporting frameworks for implementing such IoT applications have been provided. Artificial intelligence is also combined with the IoT infrastructure to enable more efficient IoT operation, improve human-computer interaction, and enhance data management and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] To easily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0008] Figure 1 is a diagrammatic representation of a networked environment in which a volumetric content rendering system may be deployed, according to some example implementations.

[0009] Figure 2A is a perspective view of a head mounted device according to some example embodiments.

[0010] Figure 2B According to some example embodiments, Figure 2A Additional views of the head-mounted device.

[0011] Figure 3 is a diagram showing a method according to some example embodiments including Figure 1 A block diagram of the networking system with details of the head mounted device.

[0012] Figure 4 is a diagrammatic representation of a volumetric content rendering system having both client-side and server-side functionality, according to some example implementations.

[0013] Figure 5 is a diagrammatic representation of data structures maintained in a database according to some example implementations.

[0014] Figure 6 is a block diagram illustrating a hardware system of a smart object according to some example embodiments.

[0015] FIG. 7A to FIG. 7B is a conceptual diagram illustrating an example grasping of a smart object according to some example embodiments.

[0016] FIG. 8A to FIG. 8Bis a conceptual diagram illustrating example touches of a smart object according to some example embodiments.

[0017] Fig. 9A is a conceptual diagram illustrating an example presentation of a visual representation of a grasp superimposed on a smart object, according to some example embodiments.

[0018] Fig. 9B is a conceptual diagram illustrating an example presentation of a volumetric representation of a grasp superimposed on a smart object, according to some example embodiments.

[0019] FIG. 10A to FIG. 10B is a conceptual diagram illustrating example touches of a smart object according to some example embodiments.

[0020] Figures 11 to 13 is a flow chart illustrating an example method for providing a touch-based experience according to some example embodiments.

[0021] Fig.14 is a block diagram illustrating a representative software architecture that may be used in conjunction with the various hardware architectures described herein, according to an example implementation.

[0022] Fig.15 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) and performing any one or more of the methodologies discussed herein according to example embodiments. DETAILED DESCRIPTION

[0023] The following description includes systems, methods, techniques, instruction sequences and computing machine program products that embody the illustrative embodiments of the present disclosure. In the following description, for the purpose of explanation, many specific details are set forth to provide an understanding of the various embodiments of the subject of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the subject of the present invention can be implemented without these specific details. Typically, known instruction examples, protocols, structures and techniques are not necessarily shown in detail.

[0024] Volumetric content is an example of an augmented reality (AR) experience. Volumetric content may include volumetric video and images of a three-dimensional space captured in three dimensions (as well as audio signals recorded with the volumetric video and images). Recording of volumetric content includes using a combination of a camera and a sensor to volumetrically capture elements of the three-dimensional space, such as objects and humans. Volumetric content includes a volumetric representation of one or more three-dimensional elements (e.g., objects or people) of the three-dimensional space. The volumetric representation of an element (e.g., an AR content item) refers to a visual representation of the three-dimensional element in three dimensions. The presentation of volumetric content may include displaying one or more AR content items superimposed on a real-world space, which may be the same or different space as the three-dimensional space in which the volumetric video is captured. The presentation of volumetric content may include displaying one or more content items in motion, displaying one or more content items performing a move or other action, displaying one or more content items in a static position, or a combination thereof. The content items may be displayed for the duration of the volumetric content presentation or for a portion of the time.

[0025] The presentation of volumetric content may include tracking the user's position and movement within their physical real-world environment, and using the tracked position and movement of the user to allow the user to move around and interact with the presentation of volumetric content. Thus, the presentation of volumetric content may include displaying content items from multiple perspectives based on changes in the user's movement and position. In this manner, the presentation of volumetric content provides an immersive AR experience to the user.

[0026] Conventional volumetric content systems lack mechanisms to support fully immersive AR experiences beyond simply displaying volumetric content. That is, conventional volumetric content systems typically only provide limited engagement with the user's vision and sound, while failing to provide interaction with the user's other senses, thereby reducing the sense of realism that the AR experience aims to provide by presenting the moment captured in a volumetric manner.

[0027] Aspects of the present disclosure include systems, methods, techniques, instruction sequences, and computer program products for providing a touch-based (e.g., tactile feedback-based) AR experience. A touch-based AR experience can be provided as part of or independently of the presentation of volumetric content. A touch-based AR experience utilizes an object that is equipped with a sensor for detecting a grip of a first user and a component (e.g., a tactile component) capable of providing a touch-based interaction to a second user. In some examples, a touch-based AR experience can correspond to a pre-recorded grip of the same or similar object. Specifically, the intensity and duration of the tactile feedback can correspond to the intensity and duration of the grip. The presentation of volumetric content may include a visual representation on one or more areas of an object corresponding to the grip area and / or a presentation of a volumetric representation of a person gripping the object.

[0028] In the aforementioned example, the volumetric content presentation system registers a touch of a first user on an object and allows a second user to experience a memory associated with the touch. The system may include a smart object, such as a customized smart mug equipped with a sensor for detecting touch and a component for providing touch-based interaction. During the initial capture phase, the first user holds the smart mug. The touch points and hand pressure of the first user are determined and stored by the system. Volumetric capture technology can be used to generate a volume representation of the first user holding the smart mug. During a subsequent experience phase, a second user wearing smart glasses can look at the smart mug and see a visual representation on the smart mug (e.g., a visualization of fingerprint luminescence). When the second user touches the mug, the intensity of the mug vibration matches the intensity of the first user holding the mug. That is, the stronger the first user's grip, the stronger the vibration provided by the second user. In this way, the volumetric content presentation system allows the second user to experience the feeling of touch even if the second user is not around the first user at the moment the first user holds the mug. The second user touching the mug can also trigger the presentation of the volume representation of the first user holding the mug. In addition, the conversation of the first user can be captured and presented with the volumetric content. In this way, the volumetric content presentation system not only enables users to visualize and feel the touch, but also brings back memories associated with the objects, as if the events and conversations were happening again.

[0029] Figure 1 1 is a block diagram illustrating an example volumetric content rendering system 100 for rendering volumetric content. The volumetric content rendering system 100 includes a client device 102. The client device 102 hosts several applications including a rendering client 104. Each rendering client 104 is communicatively coupled to a rendering server system 108 via a network 106 (e.g., the Internet). In an example, the client device 102 is a wearable device (e.g., smart glasses) worn by a user 103, the wearable device including a camera and an optical element including a transparent display through which the user 103 can see a real-world environment.

[0030] The presentation client 104 is able to communicate and exchange data with another presentation client 104 and with a presentation server system 108 via the network 106. The data exchanged between presentation clients 104 and between another presentation client 104 and the presentation server system 108 includes functions (e.g., commands for invoking functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0031] The presentation server system 108 provides server-side functionality to a particular presentation client 104 via the network 106. Although certain functions of the volumetric content presentation system 100 are described herein as being performed by the presentation client 104 or by the presentation server system 108, the location of certain functions within the presentation client 104 or within the presentation server system 108 is a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the presentation server system 108, but later migrate the technologies and functions to the presentation client 104 where the client device 102 has sufficient processing power.

[0032] The presence server system 108 supports various services and operations provided to the presence clients 104. Such operations include sending data to the presence clients 104, receiving data from the presence clients 104, and processing data generated by the presence clients 104. As examples, the data may include volumetric content (e.g., volumetric video), message content, device information, geolocation information, media annotations and overlays, message content persistence conditions, social network information, and live event information. The exchange of data within the volumetric content presentation system 100 is invoked and controlled by functions available via a user interface (UI) of the presence clients 104.

[0033] Turning now specifically to the presentation server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 that stores data associated with messages processed by the application server 112.

[0034] An application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between a client device 102 and an application server 112. Specifically, the API server 110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by a presence client 104 to activate functionality of the application server 112. The API server 110 exposes various functions supported by the application server 112, including: account registration, login functionality, sending messages from a particular presence client 104 to another presence client 104 via the application server 112, sending media files (e.g., volumetric video) to a presence client 104, setting up a collection of media data (e.g., a story), retrieving a friend list of a user of the client device 102, retrieving such a collection, retrieving messages and content, adding friends to and removing friends from a social graph, locating friends in a social graph, and opening application events (e.g., related to a presence client 104).

[0035] The application server 112 hosts several applications and subsystems, including a rendering server 114, an image processing server 116, and a social network server 122. The rendering server 114 is generally responsible for managing volumetric content and facilitating the rendering of volumetric content by the client device 102. The image processing server 116 is dedicated to performing various image processing operations, typically with respect to images or videos generated and displayed by the client device 102. The rendering server 114 and the image processing server 116 can work together to provide one or more AR experiences to the user 103. For example, the rendering server 114 and the image processing server 116 can work together to support the rendering of volumetric content by the client device 102. Additional details regarding the rendering of volumetric content are discussed below.

[0036] The social network server 122 supports various social networking functions and services and makes these functions and services available to the presence server 114. To this end, the social network server 122 maintains and accesses an entity graph within the database 120. Examples of functions and services supported by the social network server 122 include identifying other users of the volumetric content presentation system 100 who have a relationship with a particular user or who the particular user is "following," and also include identifying interests and other entities of a particular user.

[0037] The application server 112 is communicatively coupled to a database server 118 , which facilitates access to a database 120 in which data associated with content rendered by the rendering server 114 and the image processing server 116 is stored.

[0038] Figure 2A is a perspective view of a head mounted display device (eg, glasses 200) according to some examples. Glasses 200 are Figure 1 The glasses 200 are examples of client devices 102. The glasses 200 are capable of displaying content and are therefore examples of display devices (described below). In addition, the display capabilities of the glasses 200 support AR experiences, and therefore the glasses 200 are examples of AR devices. As described above, AR experiences include applying virtual content to a real-world environment, whether by presenting the virtual content through a transparent display (through which the real-world environment is visible), or by augmenting image data to include virtual content superimposed on the real-world environment depicted therein.

[0039] The glasses 200 may include a frame 202 made of any suitable material, such as plastic or metal, including any suitable shape memory alloy. In one or more examples, the frame 202 includes a first or left optical element holder 204 (e.g., a display or lens holder) and a second or right optical element holder 206 connected by a bridge 212. A first or left optical element 208 and a second or right optical element 210 may be disposed within the respective left and right optical element holders 204 and 206. The right and left optical elements 210 and 208 may be lenses, displays, display components, or a combination of the foregoing. Any suitable display component may be disposed in the glasses 200.

[0040] Frame 202 additionally includes a left arm or temple piece 222 and a right arm or temple piece 224. In some examples, frame 202 can be formed from a single piece of material to have a unitary or one-piece construction.

[0041] The glasses 200 may include a computing device such as a computer 220, which may be of any suitable type to be carried by the frame 202, and in one or more examples, the computing device may be of a suitable size and shape to be at least partially disposed in one of the temple pieces 222 or temple pieces 224. The computer 220 may include one or more processors with memory, wireless communication circuitry, and a power source. As discussed below, the computer 220 includes a low-power circuitry, a high-speed circuitry, and a display processor. Various other examples may include these elements in different configurations or integrated together in different ways. As shown in the data processor 302 discussed below, additional details of various aspects of the computer 220 may be implemented.

[0042] The computer 220 additionally includes a battery 218 or other suitable portable power supply. In some examples, the battery 218 is disposed in the left temple piece 222 and is electrically coupled to the computer 220 disposed in the right temple piece 224. The glasses 200 may include a connector or port (not shown) suitable for charging the battery 218, a wireless receiver, transmitter or transceiver (not shown), or a combination of such devices.

[0043] The glasses 200 include a first or left camera 214 and a second or right camera 216. Although two cameras are depicted, other examples contemplate the use of a single or additional (i.e., more than two) cameras. In one or more examples, the glasses 200 include any number of input sensors or other input / output devices in addition to the left camera 214 and the right camera 216. Such sensors or input / output devices may additionally include biometric sensors, position sensors, motion sensors, etc.

[0044] In some examples, left camera 214 and right camera 216 provide video frame data for use by glasses 200 to extract 3D information from a real-world scene.

[0045] The glasses 200 may also include a touchpad 226 mounted to or integrated with one or both of the left temple piece 222 and the right temple piece 224. In some examples, the touchpad 226 is arranged generally vertically, approximately parallel to the temple of the user. As used herein, generally vertical alignment means that the touchpad is vertically higher than horizontal, although potentially more vertical than this. Additional user input can be provided by one or more buttons 228, which, in the example shown, are disposed on the outer upper edges of the left optical element holder 204 and the right optical element holder 206. One or more touchpads 226 and buttons 228 provide a means by which the glasses 200 can receive input from the user of the glasses 200.

[0046] Figure 2B The glasses 200 are shown from the user's perspective. Figure 2A Several elements shown in FIG. Figure 2A As described, Figure 2B The illustrated eyeglasses 200 include a left optical element 208 and a right optical element 210 secured in a left optical element holder 204 and a right optical element holder 206, respectively.

[0047] The glasses 200 include a forward optical assembly 230 including a right projector 232 and a right near-eye display 234 , and a forward optical assembly 238 including a left projector 240 and a left near-eye display 244 .

[0048] In some examples, the near eye display is a waveguide. The waveguide includes a reflective structure or a diffractive structure (e.g., a grating and / or an optical element such as a mirror, lens, or prism). Light 236 emitted by projector 232 encounters the diffractive structure of the waveguide of near eye display 234, which directs the light toward the right eye of the user to provide an image on or in the right optical element 210 that is superimposed on the view of the real world seen by the user. Similarly, light 242 emitted by projector 240 encounters the diffractive structure of the waveguide of near eye display 244, which directs the light toward the left eye of the user to provide an image on or in the left optical element 208 that is superimposed on the view of the real world seen by the user. The combination of the GPU, front optical assembly 230, left optical element 208, and right optical element 210 provides the optical engine of glasses 200. The glasses 200 use an optical engine to generate an overlay to the user's real-world view, including displaying a 3D user interface to the user of the glasses 200 .

[0049] However, it will be appreciated that other display technologies or configurations may be utilized within the optical engine to display images to the user in the user's field of view. For example, instead of the projector 232 and waveguide, an LCD, LED or other display panel or surface may be provided.

[0050] In use, the user of the glasses 200 will be presented with information, content, and various 3D user interfaces on the near-eye display. As described in more detail herein, the user can then use the touch pad 226 and / or buttons 228, associated devices (e.g., Figure 3 The user may interact with the glasses 200 through voice input or touch input on a client device 328 shown in FIG. 1 , and / or hand movements, positioning, and location detected by the glasses 200 .

[0051] Figure 3 is a block diagram showing a networked system 300 including details of the glasses 200 according to some examples. The networked system 300 includes the glasses 200, a client device 328, and a server system 332. The client device 328 may be a smart phone, a tablet computer, a tablet phone, a laptop computer, an access point, or any other such device capable of connecting to the glasses 200 using a low power wireless connection 336 and / or a high speed wireless connection 334. The client device 328 is connected to the server system 332 via a network 330. The network 330 may include any combination of wired and wireless connections. The server system 332 may be one or more computing devices that are part of a service or network computing system. The client device 328 and any elements of the server system 332 and the network 330 may be connected using a wireless network, such as a wireless network or a wireless network. Fig.14 and Fig.15The details of the software architecture 1406 or machine 1500 described in are implemented.

[0052] The glasses 200 include a data processor 302, a display 310, one or more cameras 308, and additional input / output elements 316. The input / output elements 316 may include a microphone, an audio speaker, a biometric sensor, an additional sensor, or an additional display element integrated with the data processor 302. Fig.14 and Fig.15 Examples of input / output elements 316 are further discussed. For example, input / output elements 316 may include any I / O components 1518, including output components 1526, motion components 1534, etc. Figure 2B Examples of display 310 are discussed in . In the particular examples described herein, display 310 includes displays for the user's left eye and right eye.

[0053] Data processor 302 includes image processor 306 (eg, video processor), GPU & display driver 338, tracking module 340, interface 312, low power circuitry 304, and high speed circuitry 320. Components of data processor 302 are interconnected by bus 342.

[0054] Interface 312 refers to any source of user commands provided to data processor 302. In one or more examples, interface 312 is a physical button that, when pressed, sends a user input signal from interface 312 to low-power processor 314. Low-power processor 314 can process pressing such a button and then immediately releasing it as a request to capture a single image, and vice versa. Low-power processor 314 can process pressing such a button for a first period of time as a request to capture video data when the button is pressed and stop video capture when the button is released, wherein the video captured when the button is pressed is stored as a single video file. Alternatively, pressing the button for an extended period of time can capture a still image. In some examples, interface 312 can be any mechanical switch or physical interface capable of accepting user input associated with requesting data from camera 308. In other examples, interface 312 can have a software component, or can be associated with a command received wirelessly from another source, such as from client device 328.

[0055] Image processor 306 includes circuitry for receiving signals from camera 308 and processing those signals from camera 308 into a format suitable for storage in memory 324 or for transmission to client device 328. In one or more examples, image processor 306 (e.g., a video processor) includes a microprocessor integrated circuit (IC) customized for processing sensor data from camera 308, and volatile memory used by the microprocessor in operation.

[0056] The low-power circuit system 304 includes a low-power processor 314 and a low-power wireless circuit system 318. These elements of the low-power circuit system 304 can be implemented as separate elements or can be implemented as part of a single system on a chip on a single IC. The low-power processor 314 includes logic for managing other elements of the glasses 200. As described above, for example, the low-power processor 314 can accept user input signals from the interface 312. The low-power processor 314 can also be configured to receive input signals or command communications from the client device 328 via the low-power wireless connection 336. The low-power wireless circuit system 318 includes circuit elements for implementing a low-power wireless communication system. Bluetooth Smart, also known as Bluetooth low energy consumption, is a standard implementation of a low-power wireless communication system that can be used to implement the low-power wireless circuit system 318. In other examples, other low-power communication systems can be used.

[0057] High-speed circuit system 320 includes a high-speed processor 322, a memory 324, and a high-speed wireless circuit system 326. High-speed processor 322 can be any processor capable of managing high-speed communications and operations for any general-purpose computing system for data processor 302. High-speed processor 322 includes processing resources for managing high-speed data transmission on high-speed wireless connection 334 using high-speed wireless circuit system 326. In some examples, high-speed processor 322 executes an operating system, such as a LINUX operating system or other such operating system. In addition to any other duties, high-speed processor 322, which executes the software architecture of data processor 302, is also used to manage data transmission with high-speed wireless circuit system 326. In some examples, high-speed wireless circuit system 326 is configured to implement the Institute of Electrical and Electronics Engineers (IEEE) 802.11 communication standard, which is also referred to as Wi-Fi herein. In other examples, high-speed wireless circuit system 326 can implement other high-speed communication standards.

[0058] Memory 324 includes any storage device capable of storing camera data generated by camera 308 and image processor 306. Although memory 324 is shown as being integrated with high-speed circuitry 320, in other examples, memory 324 may be a separate, stand-alone element of data processor 302. In some such examples, electrical wiring lines may provide a connection from image processor 306 or low-power processor 314 to memory 324 through a chip including high-speed processor 322. In other examples, high-speed processor 322 may manage addressing of memory 324 so that low-power processor 314 will initiate high-speed processor 322 any time a read or write operation involving memory 324 is needed.

[0059] The tracking module 340 estimates the pose of the glasses 200. For example, the tracking module 340 uses image data and corresponding inertial data from the camera 308 and the position component 1538, as well as GPS data, to track the position and determine the pose of the glasses 200 relative to a reference frame (e.g., a real-world environment). The tracking module 340 continuously collects and uses updated sensor data describing the movement of the glasses 200 to determine an updated three-dimensional pose of the glasses 200, which indicates changes in relative position and orientation relative to physical objects in the real-world environment. The tracking module 340 allows the glasses 200 to visually place virtual objects relative to physical objects within the user's field of view via the display 310.

[0060] The GPU and display driver 338 may use the pose of the glasses 200 to generate frames of virtual content or other content to be presented on the display 310 when the glasses 200 are operating in a conventional augmented reality mode. In this mode, the GPU and display driver 338 generate updated frames of virtual content based on the updated three-dimensional pose of the glasses 200 that reflects changes in the position and orientation of the user relative to physical objects in the user's real-world environment.

[0061] One or more functions or operations described herein may also be performed in an application resident on the glasses 200 or on the client device 328 or on a remote server. The glasses 200 may be a stand-alone client device capable of independent operation, or may be a companion device that works with a host device to offload intensive processing and / or exchange data with the presentation server system 108 over a network. The glasses 200 may also be communicatively coupled to a companion device such as a smart watch and may be configured to exchange data with the companion device. The glasses 200 may also include various components common to mobile electronic devices such as smart glasses or smart phones (e.g., including a display controller for controlling the display of visual media on a display mechanism incorporated into the device).

[0062] Figure 41 is a block diagram showing additional details about a volumetric content rendering system 100 according to some examples. Specifically, the volumetric content rendering system 100 is shown to include a rendering client 104 and an application server 112. The volumetric content rendering system 100 contains several subsystems that are supported on the client side by the rendering client 104 and on the server side by the application server 112. These subsystems include, for example, a collection management system 402, a rendering control system 404, an enhancement system 406, and a touch-based experience generation system 408.

[0063] The collection management system 402 is responsible for managing sets or collections of content (e.g., collections of text, images, video, and audio data). Collections of content may be organized into "event galleries" or "event stories." Such collections may be made available for a specified period of time, such as for the duration of an event to which the content relates. For example, content related to a concert may be available as a "story" for the duration of the concert. The collection management system 402 may also be responsible for publishing an icon to the user interface of the presentation client 104 that provides notification of the existence of a particular collection.

[0064] In addition, the collection management system 402 includes a curation interface 412 that allows collection managers to manage and curate specific content collections. For example, the curation interface 412 enables event organizers to curate a collection of content related to a specific event (e.g., to remove inappropriate content or redundant messages). In addition, the collection management system 402 uses machine vision (or image recognition technology) and content rules to automatically curate content collections.

[0065] The presentation control system 404 is responsible for facilitating and controlling the volumetric content presentation. Thus, the presentation control system 404 provides a mechanism that allows a user to specify control operations for controlling the volumetric content presentation. The control operations may include, for example: a stop operation to stop the presentation; a pause operation to pause the presentation; a fast forward operation to advance the presentation at a higher speed; a rewind operation to rewind the presentation; a zoom in operation to increase the zoom level of the presentation; a zoom out operation to decrease the zoom level of the presentation; and a playback speed modification operation to change the speed of the presentation (e.g., to produce a slow motion presentation of the volumetric video).

[0066] For some embodiments, a user may specify inputs indicative of control operations for controlling presentation of volumetric content by providing one or more inputs via one or more I / O components (examples of which are described below with reference to Fig.15For some embodiments, the presentation control system 404 may provide an interactive control interface including one or more interactive elements (e.g., virtual buttons) to trigger a control operation, and the presentation control system 404 monitors interactions with the interactive interface to detect input indicating a control operation. For some embodiments, a user may trigger a control operation using a gesture (such as a hand or head gesture) that may be associated with a particular control operation.

[0067] The enhancement system 406 provides various functions that enable a user to enhance (e.g., annotate or otherwise modify or edit) media content. For example, the enhancement system 406 provides functions related to the generation, publication, and application of enhancement data, such as media overlays (e.g., image filters) to volumetric content. Media overlays can include audio and visual content and visual effects. Examples of audio content and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio content and visual content or visual effects can be applied to media content items (e.g., photos) at the client device 102. For example, a media overlay can include text or images that can be superimposed on a photo taken by the client device 102. The enhancement system 406 is operable to supply one or more media overlays to the presentation client 104 based on the geographic location of the client device 102 or based on other information (such as social network information of a user of the client device 102). The media overlays can be stored in the database 120 and accessed through the database server 118.

[0068] A filter is an example of a media overlay that is displayed as an overlay on an image or video during presentation to a user. Filters can be of various types, including filters selected by a user from a set of filters presented to the user by presentation client 104. Other types of filters include geolocation filters (also referred to as geofilters), which can be presented to a user based on a geographic location. For example, a geolocation filter specific to a nearby or special location can be presented by presentation client 104 within a user interface based on geolocation information determined by a global positioning system (GPS) unit of client device 102.

[0069] Another type of filter is a data filter, which can be selectively presented to the user by the presentation client 104 based on other input or information collected by the client device 102. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, the battery life of the client device 102, or the current time.

[0070] AR content items are another example of a media overlay. AR content items can be real-time special effects and / or sounds that can be added to images or videos, including volumetric images and videos.

[0071] In general, AR content items, overlays, image transformations, images, and similar terms refer to modifications that can be applied to image data (e.g., videos or images) that include volumetric content. This includes real-time modifications that modify images as they are captured using a device sensor (e.g., one or more cameras) of the client device 102 and then display the modified image by a display device (e.g., an embedded display of the client device) of the client device 102. This also includes modifications to stored content (e.g., volumetric videos in a library or collection that can be modified). For example, in a client device 102 that has access to multiple AR content items, a user can use a single volumetric video with multiple AR content items to see how different AR content items will modify the stored content. For example, by selecting different AR content items for the content, multiple augmented reality content items that apply different pseudo-random movement models can be applied to the same content. Similarly, real-time video capture can be used with illustrated modifications to show how the video image currently being captured by the sensor of the client device 102 will modify the captured data. Such data may be displayed only on the screen and not stored in memory, or content captured by the device sensors may be recorded and stored in memory with or without modification (or both). In some systems, a preview function may show how different augmented reality content items will be displayed simultaneously in different windows of the display. For example, this may enable viewing multiple windows with different pseudo-random animations on the display at the same time.

[0072] Thus, data and various systems using augmented reality content items or other such augmented systems that modify content using augmented data may involve detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.) in video frames, tracking of such objects when these objects leave the field of view, enter the field of view, and move around the field of view, and modification or transformation of such objects when tracking these objects. In various examples, different methods for implementing such transformations may be used. Some examples may involve generating a three-dimensional mesh model of one or more objects, and implementing the transformation using transformations and animated textures of the models within the video. In other examples, tracking of points on the object may be used to place an image or texture (which may be two-dimensional or three-dimensional) at the tracked location. In yet another example, a neural network analysis of a video frame may be used to place an image, model, or texture in content (e.g., an image or video frame). Thus, an AR content item refers to both images, models, and textures used to create transformations in the content, and additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0073] Real-time video processing can be performed with any kind of video data (e.g., video streams, video files, volumetric video, etc.) stored in the memory of any kind of computerized system. For example, a user can load a video file and save it in the memory of the device, or a video stream can be generated using the device's sensors. In addition, any object, such as human faces and parts of the human body, animals, or non-living objects (e.g., chairs, cars, or other objects) can be processed using computer animation models.

[0074] In some examples, when a specific modification is selected together with the content to be enhanced (e.g., edited), the element to be transformed is identified by the computing device, and then if the element to be transformed is present in the frame of the video, the element to be transformed is detected and tracked. The elements of the object are modified according to the request for modification, so the frame of the video stream is transformed. For different types of transformations, the frame transformation of the video stream can be performed by different methods. For example, for frame transformations that mainly involve changing the form of the elements of the object, feature points for each element of the object are calculated (e.g., using active shape models (ASM) or other known methods). Then, for each element of at least one element of the object, a grid based on feature points is generated. The grid can be used for subsequent stages of tracking the elements of the object in the video stream. In the tracking process, the grid mentioned for each element is aligned with the position of each element. Then, additional points are generated on the grid. A set of first points is generated for each element based on the modification request, and a set of second points is generated for each element based on the set of first points and the modification request. Then, the frame of the video stream can be transformed by modifying the elements of the object based on the set of first points and the set of second points and the grid. In such methods, the background of the modified object may also be changed or distorted by tracking and modifying the background.

[0075] In some examples, a transformation that changes certain areas of an object using elements of an object can be performed by calculating feature points of each element of the object and generating a grid based on the calculated feature points. Points are generated on the grid, and then various areas are generated based on the points. The elements of the object are then tracked by aligning the area of ​​each element with the position of each element of at least one element, and the properties of the area can be modified based on the modification request, thereby transforming the frame of the video stream. Depending on the specific modification request, the properties of the mentioned area can be transformed in different ways. Such modifications may involve: changing the color of the area; removing at least some parts of the area from the frame of the video stream; including one or more new objects into the area based on the modification request; and modifying or distorting the elements of the area or object. In various examples, any combination of such modifications or other similar modifications may be used. For certain models to be animated, some feature points can be selected as control points for determining the entire state space of model animation options.

[0076] In some examples of computer animation models that use face detection to transform image data, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). An active shape model (ASM) algorithm is then applied to the face region of the image to detect facial feature reference points.

[0077] Other methods and algorithms suitable for face detection can be used. For example, in some examples, landmarks are used to locate features, and the landmarks represent distinguishable points that are present in most of the images considered. For example, for facial landmark points, the location of the left pupil can be used. If the initial landmarks are not recognizable (for example, if the person has an eye mask), secondary landmarks can be used. Such a boundary recognition process can be used for any such object. In some examples, a set of landmarks forms a shape. The coordinates of the points in the shape can be used to represent the shape as a vector. A shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The mean shape is the average of the aligned training shapes.

[0078] In some examples, the landmark search begins with an average shape that aligns with the position and size of the face determined by a global face detector. Such a search then repeats the steps of suggesting tentative shapes by adjusting the positions of shape points by template matching of the image texture around each point, and then conforming the tentative shapes to a global shape model until convergence occurs. In some systems, template matching alone is unreliable, and the shape model pools the results of weak template matches to form a stronger overall classifier. The entire search is repeated at each level of the image pyramid from coarse to fine resolution.

[0079] The augmentation system 406 can capture an image or video stream on a client device (e.g., client device 102) and perform complex image manipulations locally on the client device 102 while maintaining an appropriate user experience, computation time, and power consumption. Complex image manipulations can include size and shape changes, emotion conversions (e.g., changing a face from a frown to a smile), state transitions (e.g., aging an object, reducing apparent age, changing gender), style transfer, application of graphical elements, and any other suitable image or video manipulations enabled by a convolutional neural network that has been configured to execute efficiently on the client device 102.

[0080] In some examples, a computer animation model for transforming video and image content can be used by an augmentation system 406, where a neural network operates as part of a rendering client 104 operating on a client device 102. The augmentation system 406 determines the presence of a face within an image or video stream and provides an interactive modification element (e.g., an icon) associated with the computer animation model for transforming image data, or the computer animation model can exist in association with the interface described herein. The interactive modification element includes a change that can be the basis for modifying the user's face within the image or video content as part of the modification operation. Once the modification icon is selected, the transformation system initiates a process of transforming the user's image to reflect the selected modification icon (e.g., generating a smiley face on the user). Once the image or video content is captured and the specified modification is selected, the modified image or video stream can be presented in a graphical user interface displayed on the client device 102. The augmentation system 406 can implement a complex convolutional neural network on a portion of the image or video content to generate and apply the selected modification. That is, the modified content can be presented to the user in real time or near real time. In addition, the modification can be persistent when the content is presented. Neural networks for machine learning can be used to implement such modifications.

[0081] The touch-based experience generation system 408 is responsible for providing a touch-based experience to the user. In some examples, the touch-based experience generation system 408 can access data related to the grip of the object. The data may include a first area of ​​the grip of the object. The data may also include the strength of the force applied to the object in the grip and / or the duration of the grip. In some examples, the touch-based experience generation system 408 may cause the display device (e.g., glasses 200) to display a visual representation on the first area. The touch-based experience generation system 408 may detect a user's touch on a second area of ​​the object. The second area may overlap at least partially with the first area. In response to the detection, the touch-based experience generation system 408 may generate tactile feedback (e.g., vibration) to the user. The strength of the tactile feedback may correspond to the strength of the force applied to the object in the grip. The duration of the tactile feedback may correspond to the duration of the grip. If the data includes a volume representation of the user's grip of the object, the touch-based experience generation system 408 may display the volume representation simultaneously with the tactile feedback or independently of the tactile feedback. If the data includes audio signals recorded during grasping of the object, touch-based experience generation system 408 may also cause an audio device to present the audio signals simultaneously with or independently of the volumetric representation and / or haptic feedback.

[0082] Figure 5is a diagrammatic representation of a data structure 500 maintained in database 120, according to some examples. Although the contents of database 120 are shown as including a plurality of tables, it will be appreciated that data may be stored in other types of data structures (eg, as an object-oriented database).

[0083] Entity table 504 stores entity data and is linked (e.g., by reference) to entity graph 506 and profile data 508. Entities whose records are maintained within entity table 504 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the entity type, any entity about which presence server system 108 stores data may be an identified entity. A unique identifier is provided for each entity as well as an entity type identifier (not shown). Entity table 504 may associate various enhancements from enhancement table 510 with various images and videos stored in image table 516 and video table 514.

[0084] Entity graph 506 stores information about relationships and associations between entities. Such relationships may be social, professional (e.g., working in a common company or organization), interest-based, or activity-based, by way of example only.

[0085] The profile data 508 stores various types of profile data about a particular entity. The profile data 508 can be selectively used and presented to other users of the volumetric content presentation system 100 based on the privacy settings specified by the particular entity. In the case where the entity is a person, the profile data 508 includes, for example, a user name, phone number, address, settings (e.g., notification and privacy settings), and an avatar representation (or a collection of such avatar representations) selected by the user.

[0086] The database 120 also stores augmentation data, such as overlays including AR content items and filters, in augmentation table 510. Augmentation data, including volumetric video and images, is associated with and applied to videos (data for which is stored in video table 514) and images (data for which is stored in image table 516).

[0087] The story table 512 stores data about collections of content including associated images, video, or audio data that are compiled into collections (e.g., stories or libraries). The creation of a particular collection can be initiated by a particular user (e.g., each user whose record is saved in the entity table 504). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface of the presentation client 104 may include an icon that is user-selectable to enable the sending user to add specific content to his or her personal story.

[0088] As mentioned above, the video table 514 stores video data including volumetric videos. Similarly, the image table 516 stores image data including volumetric images.

[0089] Figure 6 6 is a block diagram illustrating a hardware system of a smart object 600 according to some example implementations. The smart object 600 may include a processor 602, one or more sensors 604, a display 606, a communication interface 608, a vibration motor, a speaker 612, and / or a storage device 614.

[0090] One or more sensors 604 can generate sensor data related to the first user's grasping of the smart object 600. One or more sensors 604 can include a pressure sensor or a tactile sensor that detects the location and strength of the grasping. The sensor data related to the detected grasping can be sent to the processor 602, or stored in the storage device 614. The sensor data can specify the strength of the grasping (e.g., as a measurement of the pressure or force applied to the object), the duration of the grasping, and the location of the grasping on the object. For example, the sensor data can identify the first area on the object where the grasping occurs. For some embodiments, the smart object 600 (e.g., the processor 602) can determine that the grasping occurs on the first area of ​​the object based on the sensor data. In some embodiments, the data related to the grasping can be pre-stored in the storage device 614. The processor 602 can access the pre-stored data from the storage device 614.

[0091] The one or more sensors 604 may also generate sensor data related to the second user's touch on the smart object 600. The smart object 600 (e.g., the processor 602) may monitor the sensor data generated by the one or more sensors 604 to detect the second user's touch. For some embodiments, the sensor data may include a measurement of pressure applied to at least one area of ​​the object, and detecting the touch may include detecting the satisfaction of a threshold condition based on the measurement of pressure indicated by the sensor data. For example, the smart object 600 (e.g., the processor 602) may determine that the second user is touching the object based on a pressure measurement value indicated by the sensor data that exceeds a threshold pressure measurement value.

[0092] For some embodiments, the one or more sensors 604 include one or more biometric sensors or other sensors capable of providing data to identify the second user. The smart object 600 (e.g., the processor 602) can identify that the second user is touching the object from or based on data from such sensors. For example, the smart object 600 can provide biometric data to the volumetric content presentation system 100, and the volumetric content presentation system 100 can match the biometric data with stored biometric data associated with a user profile to identify the second user.

[0093] For some embodiments, one or more sensors 604 can detect that the second user touches the second area of ​​the object. Consistent with these embodiments, detecting the second user's touch also includes determining that the second area at least partially overlaps the first area. For some embodiments, an overlap constraint defining a threshold overlap can be predefined, and as part of detecting the second user's touch, the smart object 600 (e.g., processor 602) can determine whether an amount of overlap between the first area and the second area satisfies the overlap constraint.

[0094] Display 606 can generate a presentation of a visual representation (e.g., a fingerprint-like representation) of the first user's grip on the surface of smart object 600. The visual representation of the grip can be displayed on the first area where the grip occurs to provide an indication to the second user of the location where the first user's grip occurs. In some embodiments, display 606 can be omitted and the visual representation of the grip is displayed by a display device (e.g., glasses 200) worn by the user using AR technology.

[0095] The communication interface 608 may send and / or receive data. For example, the communication interface 608 may send sensor data related to the first user's grasping of the smart object 600 to a display device (e.g., glasses 200). The sensor data related to the grasping may include the location of the first area of ​​the grasping, and the display device may use AR technology to present a visual representation of the grasping (e.g., a fingerprint-like representation) superimposed on the smart object 600. The visual representation of the grasping may be displayed on the first area where the grasping occurs so as to provide an indication to the second user of the location where the grasping of the first user occurs. For some embodiments, the communication interface 608 may receive control instructions from the volumetric content presentation system 100. The communication interface 608 may send control instructions to other components of the smart object 600. For example, the communication interface 608 may send control instructions related to providing tactile feedback to the vibration motor 610, and cause the vibration motor 610 to provide tactile feedback to the second user. The tactile feedback may be provided at an intensity and / or duration based on the intensity and / or duration of the grasping of the first user. As another example, the communication interface 608 can send control instructions related to generating a sound signal to the speaker 612. The sound signal can be recorded by a recorder (not shown) during the first user's grasping of the object. As part of providing interaction with the second user, the speaker 612 can present the audio signal. The speaker 612 can be embedded in the smart object 600, or can be a different audio device that is communicatively coupled to the smart object 600.

[0096] The storage device 614 can store data related to the first user's grasping of the object. The data can specify the intensity of the grasp (e.g., as a measure of pressure or force applied to the object), the duration of the grasp, and the location of the grasp on the object. For example, the data can identify the first area on the object where the grasp occurred. The data can also include a volumetric representation of one or more elements of the real-world environment in which the first user's grasp was performed. For example, the volumetric content can include a volumetric video of the first user grasping the smart object 600.

[0097] FIG. 7A to FIG. 7B 702 is a conceptual diagram illustrating an example grasping of a smart object 702 according to some example embodiments. Smart object 702 is an example of smart object 600. FIG. 7A to FIG. 7B As shown, the first user can grasp the smart object 702 with one of their hands. The grasping of the smart object 702 by the finger 704 can be detected by one or more sensors (e.g., one or more sensors 604) of the smart object 702 on one or more first areas 706 of the smart object 702.

[0098] FIG. 8A to FIG. 8B802 is a conceptual diagram illustrating an example touch of a smart object 802 according to some example embodiments. Smart object 802 is an example of smart object 600. Figures 8A to 8B As shown, the second user may touch smart object 802 with one of their hands. Smart object 802 may be the same as or different from smart object 702. In the event that smart object 702 is different from smart object 802, smart object 802 may communicate with smart object 702 to obtain data related to the grasp of smart object 702, or smart object 802 accesses data from a data storage component (e.g., an internal data storage component or a network accessible data storage component of smart object 802). The second user's finger 804 may touch smart object 802 on one or more second regions 806. Volumetric content presentation system 100 may determine that one or more second regions 806 do not at least partially overlap with one or more first regions 706, and therefore, smart object 802 may not provide any kind of interaction with the second user.

[0099] For some embodiments, the volumetric content presentation system 100 may provide a visual representation of the first user's grasp on the smart object 802 as an indication to the second user, an example of which is shown in FIG. Fig. 9A See Fig. 9A , showing an example presentation of a visual representation 902 of a grasp superimposed on a smart object. The volumetric content presentation system 100 can provide a presentation of the visual representation 902 superimposed on the smart object over a first area of ​​the grasp. The presentation of the visual representation 902 can be specifically provided by a display device (e.g., glasses 200) using AR technology or by a display (e.g., 606) on the smart object (600). As a non-limiting example, the visual representation can be a fingerprint-like representation. It should be understood that a variety of different types of visual representations (e.g., a solid shape, an empty shape, a shape with a dashed border) can be used to represent the grasp of the first user.

[0100] In some embodiments, visual representation 902 may be presented after the second user's touch is detected. The second area touched by the second user may not at least partially overlap with the first area gripped by the first user, and the visual representation may serve to indicate to the second user where the first user's grip occurs. Alternatively, visual representation 902 may be presented to the second user when the second area at least partially overlaps with the first area. The visual representation may make the second user feel that he / she is in the company of the first user.

[0101] Reference Fig. 9B, an additional example visual representation of a first user's grasp is shown in the form of a volumetric representation of the first user grasping the object. The volumetric representation can be presented by an AR display device (e.g., glasses 200). The presentation of the volumetric representation can include displaying one or more AR content items 904 related to the first user's grasp superimposed on the smart object. In this example, the AR content item 904 corresponds to a volumetric representation of the first user's hand, but in other examples, the volumetric representation of the first user may include a representation of another part of the first user or the entire body. The presentation of the volumetric representation of the first user may also include presenting a sound signal generated by the first user when grasping the smart object.

[0102] Fig. 10A and Fig. 10B 1 is a conceptual diagram illustrating an example touch of a third user on a smart object 1002 according to some example embodiments. The smart object 1002 is an example of the smart object 600. Fig. 10A and Fig. 10B As shown, a third user may touch smart object 1002 with one of their hands. The third user may be the same as the second user. Smart object 1002 may be the same as smart object 802. Smart object 1002 may be the same as or different from smart object 702. A finger 1004 of the third user may touch smart object 1002 over one or more third regions 1006. The volumetric content presentation system 100 may determine that the one or more third regions 1006 at least partially overlap with the one or more first regions 706, and provide tactile and / or audio interaction with the second user.

[0103] Figures 11 to 13 1 is a flow chart illustrating an example method for providing a touch-based experience according to some example implementations. The method 1100 may be implemented in computer-readable instructions for execution by one or more processors, such that the operations of the method 1100 may be performed in part or in whole by functional components of the volumetric content rendering system 100; therefore, the method 1100 is described below by way of example with reference thereto. However, it should be understood that at least some of the operations of the method 1100 may be deployed on various other hardware configurations besides the volumetric content rendering system 100.

[0104] In operation 1102, the volumetric content rendering system 100 may access data related to a first user's grasping of an object. The object includes one or more sensors to detect the user's grasping, including detecting the location and strength of the grasping. Thus, the data may be or include sensor data (e.g., pressure sensor data or tactile sensor data) obtained by the volumetric content rendering system 100. For some embodiments, the object may also include a memory component capable of storing historical sensor data.

[0105] For some embodiments, the volumetric content rendering system 100 may communicate directly with the object to obtain sensor data (e.g., via a network or physical interface). Consistent with these embodiments, the object may be configured to include any one or more of a network interface or a physical interface to facilitate communication with the volumetric content rendering system 100. For some embodiments, the volumetric content rendering system 100 may communicate with a third-party service to obtain sensor data.

[0106] The data may specify the intensity of the grasp (e.g., as a measure of pressure or force applied to the object), the duration of the grasp, and the location of the grasp on the object. For example, the data may identify a first area on the object where the grasp occurred. For some embodiments, the volumetric content presentation system 100 may determine that the grasp occurred at the first area of ​​the object based on the sensor data.

[0107] In the example, the object is a coffee cup. However, it should be understood that the object may be any object that can be grasped by a user and is equipped with one or more sensors for detecting a user's grasping.

[0108] In operation 1104, the volumetric content presentation system 100 detects that the second user is touching the object. The volumetric content presentation system 100 may detect that the second user is touching the object based on sensor data generated by one or more sensors of the object. Thus, the volumetric content presentation system 100 may monitor the sensor data generated by the one or more sensors to detect the touch of the second user. For some embodiments, the sensor data may include a measurement of pressure applied to at least one area of ​​the object, and detecting the touch may include detecting satisfaction of a threshold condition based on the measurement of pressure indicated by the sensor data. For example, the volumetric content presentation system 100 may determine that the second user is touching the object based on a pressure measurement indicated by the sensor data exceeding a threshold pressure measurement.

[0109] For some embodiments, the object includes one or more biometric sensors or other sensors capable of providing data to identify the second user. The volumetric content rendering system 100 can identify that the second user is touching the object from or based on data from such sensors. For example, the object can provide biometric data to the volumetric content rendering system 100, and the volumetric content rendering system 100 can match the biometric data with stored biometric data associated with a user profile to identify the second user.

[0110] For some embodiments, the volumetric content presentation system 100 can detect that the second user touches the second area of ​​the object. Consistent with these embodiments, detecting the touch of the second user also includes determining that the second area at least partially overlaps the first area. For some embodiments, an overlap constraint defining a threshold overlap can be predefined, and as part of detecting the touch of the second user, the volumetric content presentation system 100 can determine whether an amount of overlap between the first area and the second area satisfies the overlap constraint.

[0111] At operation 1106, the volumetric content presentation system 100 causes the object to provide interaction with the second user in response to detecting the touch of the second user. In some examples, the interaction with the second user includes providing tactile feedback (e.g., vibration) to the second user. Thus, in causing the object to provide interaction, the volumetric content presentation system 100 causes the object to provide tactile feedback. The tactile feedback may be provided by one or more tactile components of the object (e.g., a tactile sensor and / or actuator). As described above, the data related to the object's grasp may also include the intensity and / or duration of the grasp and / or the duration. For some embodiments, the volumetric content presentation system 100 may cause the object to provide tactile feedback at an intensity and / or duration based on the intensity and / or duration of the first user's grasp.

[0112] In some examples, the data related to the grasping of the object may also include audio signals recorded during the first user's grasping of the object. In these examples, as part of providing interaction with the second user, the volumetric content presentation system 100 may also cause an audio device to present the audio signals. The audio device may be embedded in the object, or may be a different audio device communicatively coupled to the object.

[0113] In some examples, a third user may touch the second object. The second object may be different from the first object, but may be in direct communication with the first object, or may have accessed or downloaded data related to the grasp of the first object. The third user's touch on the second object may be on a third area of ​​the second object that at least partially corresponds to the first area of ​​the first object. In response to detecting the touch on the second object, the volumetric content presentation system 100 causes the second object to provide interaction with the third user.

[0114] In some examples, in response to detecting a third user's touch to the second object, the volumetric content presentation system 100 causes the first object to provide a second interaction with the second user. In this way, the second user is notified when the third user touches the second object.

[0115] In operation 1108 (optional in some implementations), the volumetric content presentation system 100, in response to detecting the second user's touch on the object, causes a display device (e.g., glasses 200) worn by the second user to present a visual representation (e.g., a fingerprint-like representation) of the first user's grip superimposed on the object. The visual representation of the grip may be displayed on the first area where the grip occurs to provide the second user with an indication of where the first user's grip occurs.

[0116] In operation 1110 (optional in some embodiments), the volumetric content presentation system 100 causes a display device to present volumetric content associated with a first user's grasp of an object. The volumetric content may include a volumetric representation of one or more elements of a real-world environment in which the first user performs the grasp. For example, the volumetric content may include a volumetric representation of the first user's grasp of the object.

[0117] For some embodiments, the display device is an AR device (e.g., glasses 200), and the presentation of volumetric content may include displaying one or more AR content items superimposed on a real-world space, which may be the same or a different space than the three-dimensional space in which the volumetric content is captured. The presentation of volumetric content may include displaying one or more content items in motion, displaying one or more content items performing movement or other actions, displaying one or more content items statically positioned, or a combination thereof. The content items may be displayed for the duration of the presentation of the volumetric content or a portion thereof.

[0118] The presentation of the volumetric content may also include tracking the position and movement of the second user within their physical real-world environment, and using the tracked position and movement of the second user to allow the second user to move within and interact with the presentation of the volumetric content. Thus, the presentation of the volumetric content may include displaying content items from multiple perspectives based on changes in the movement and position of the second user. The position and movement of the second user may be based on any one or more motion and position components from the display device, a host device of which the display device is a complementary device, or a complementary device (e.g., a smartwatch) that communicates with the display device (examples of which are described below with reference to Fig.15 Detailed description is provided below.

[0119] Depending on the implementation, the operations of method 1100 may be repeated in different ways or involve intermediate operations that are not shown. Although the operations of method 1100 may be depicted and described in a particular order, the order in which the operations are performed may vary between implementations, including performing certain operations in parallel. For example, for some implementations, volumetric content presentation system 100 may cause a visual representation of a first user's grasp to be presented before a second user is detected touching an object.

[0120] like Fig.12 As shown, in some embodiments, method 1100 may include operations 1202 and 1204. Consistent with these embodiments, operation 1202 may be performed as part of operation 1104 (the volumetric content rendering system 100 detects that the second user has touched the object). In operation 1202, the volumetric content rendering system 100 determines that the area of ​​the object touched by the second user at least partially overlaps with the grasped area. For some embodiments, an overlap constraint defining a threshold overlap may be predefined, and as part of detecting the touch of the second user, the volumetric content rendering system 100 may determine whether an amount of overlap between the first area and the second area satisfies the overlap constraint. The overlap constraint may specify a threshold overlap amount, and the volumetric content rendering system 100 may determine that the amount of overlap satisfies the overlap constraint based on the amount of overlap exceeding the threshold overlap amount.

[0121] Consistent with these embodiments, operation 1204 may be performed as part of operation 1106, in which the volumetric content presentation system 100 causes the object to provide interaction with the second user in response to detecting the touch of the second user. In operation 1204, the volumetric content presentation system 100 causes the object to provide tactile feedback to the second user. The tactile feedback may be provided by one or more tactile components (e.g., tactile sensors and / or actuators) of the object. As described above, the data related to the object's grasp may also include the intensity and / or duration of the grasp and / or the duration. For some embodiments, the volumetric content presentation system 100 may cause the object to provide tactile feedback at an intensity and / or duration based on the intensity and / or duration of the first user's grasp.

[0122] like Fig.13 As shown, in some embodiments, method 1100 may include operations 1302 and 1304. Consistent with these embodiments, operations 1302 and 1304 may be performed as part of operation 1110, in which volumetric content presentation system 100 causes presentation of volumetric content. In operation 1302, volumetric content presentation system 100 presents a volumetric representation of a first user's grasped object superimposed on a real-world environment. The volumetric representation of the first user's grasped object may be superimposed on an object of a second user. The volumetric representation of the first user may include displaying one or more content items in motion, displaying one or more content items performing a movement or other action, displaying one or more content items statically positioned, or a combination thereof. Content items may be displayed for the duration of the volumetric content presentation or for a portion thereof.

[0123] In operation 1304, the volumetric content presentation system 100 causes an audio device to present an audio signal. The audio signal may be recorded during the first user's grasping of the object. The audio signal may include a voice signal of the first user or music played by the first user while grasping the object. The audio signal may be presented simultaneously with or independently of the volume representation in 1302 and / or the haptic feedback in 1106.

[0124] Software Architecture

[0125] Fig.14 is a block diagram illustrating an example software architecture 1406 that may be used in conjunction with the various hardware architectures described herein. Fig.14 is a non-limiting example of a software architecture, and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 1406 may be implemented in a system such as Fig.15 1500, which includes a processor 1504, a memory / storage device 1506, and an I / O component 1518, among others. A representative hardware layer 1452 is shown and may represent, for example, Fig.15 The representative hardware layer 1452 includes a processing unit 1454 having associated executable instructions 1404. The executable instructions 1404 represent executable instructions of the software architecture 1406, including implementations of the methods, components, etc. described herein. The hardware layer 1452 also includes a memory and / or storage device module 1456 also having executable instructions 1404. The hardware layer 1452 may also include other hardware 1458.

[0126] exist Fig.14 In the example architecture shown, software architecture 1406 can be conceptualized as a layer stack, wherein each layer provides specific functionality. For example, software architecture 1416 may include layers such as operating system 1402, library 1420, framework / middleware 1418, application 1416, and presentation layer 1414. In operation, application 1416 and / or other components within a layer may call API call 1408 through a software stack and receive a response to API call 1408 in the form of message 1412. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 1418, while other operating systems may provide such layers. Other software architectures may include additional layers or different layers.

[0127] The operating system 1402 can manage hardware resources and provide public services. For example, the operating system 1402 may include a core 1422, a service 1424, and a driver 1426. The core 1422 may serve as an abstraction layer between the hardware and other software layers. For example, the core 1422 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The service 1424 may provide other public services for other software layers. The driver 1426 is responsible for controlling the underlying hardware or connecting to the underlying hardware. For example, depending on the hardware configuration, the driver 1426 includes a display driver, a camera driver, a Bluetooth driver, a flash memory driver, a serial communication driver (e.g., a universal serial bus (USB) driver), a Wi-Fi driver, an audio driver, a power management driver, etc.

[0128] The library 1420 provides a common infrastructure used by the application 1416 and / or other components and / or layers. Compared with directly interacting with the underlying operating system 1402 functions (e.g., the kernel 1422, the service 1424 and / or the driver 1426), the library 1420 provides functions that allow tasks to be performed in an easier manner. The library 1420 may include a system library 1444 (e.g., a C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 1420 may include an API library 1446, such as a media library (e.g., a library that supports the presentation and manipulation of various media formats such as MPEG4, H.294, MP3, AAC, AMR, JPG, and PNG), a graphics library (e.g., an OpenGL framework that can be used to render 2D and 3D graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a web library (e.g., WebKit that can provide web browsing functions), etc. The library 1420 may also include various other libraries 1448 to provide numerous other APIs to the applications 1416 and other software components / modules.

[0129] The framework / middleware 1418 provides a higher level common infrastructure that can be used by the applications 1416 and / or other software components / modules. For example, the framework / middleware 1418 can provide various GUI functions, advanced resource management, advanced location services, etc. The framework / middleware 1418 can provide a wide range of other APIs that can be utilized by the applications 1416 and / or other software components / modules, some of which can be specific to a particular operating system 1402 or platform.

[0130] Applications 1416 include built-in applications 1438 and / or third-party applications 1440. Examples of representative built-in applications 1438 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 1440 may include applications developed by entities other than the vendor of a particular platform using an ANDROID or IOS software development kit (SDK), and may be mobile software running on a mobile operating system such as IOS, ANDROID, WINDOWSPhone, or other mobile operating system. Third-party applications 1440 may call API calls 1408 provided by a mobile operating system (e.g., operating system 1402) to facilitate the functionality described herein.

[0131] Applications 1416 may create user interfaces using built-in operating system functionality (e.g., kernels 1422, services 1424, and / or drivers 1426), libraries 1420, and frameworks / middleware 1418 to interact with users of the system. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer such as presentation layer 1414. In these systems, the application / component "logic" may be separated from aspects of the application / component that interact with the user.

[0132] Fig.15 is a block diagram illustrating components of a machine 1500 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methods discussed herein, according to some example embodiments. Specifically, Fig.15A diagrammatic representation of a machine 1500 in the example form of a computer system is shown, within which instructions 1510 (e.g., software, programs, applications, applet, app, or other executable code) for causing the machine 1500 to perform any one or more of the methods discussed herein may be executed. Thus, the instructions 1510 may be used to implement the modules or components described herein. The instructions 1510 convert a general-purpose, unprogrammed machine 1500 into a specific machine 1500 that is programmed to perform the functions described and shown in the described manner. In alternative embodiments, the machine 1500 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 1500 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 1500 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a PDA, an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web device, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1510 that specify actions to be taken by the machine 1500, either sequentially or otherwise. In addition, while only a single machine 1500 is shown, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute instructions 1510 to perform any one or more of the methodologies discussed herein.

[0133] The machine 1500 may include a processor 1504, a memory / storage device 1506, and an I / O component 1518, which may be configured to communicate with each other, for example, via a bus 1502. In an example embodiment, the processor 1504 (e.g., a CPU, a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a GPU, a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 1508 and a processor 1509 that may execute instructions 1510. Although Fig.15 Multiple processors 1504 are shown, but the machine 1500 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0134] The memory / storage 1506 may include a memory 1512, such as a main memory or other storage device, and a storage unit 1514, both of which are accessible to the processor 1504, for example, via the bus 1502. The storage unit 1514 and the memory 1512 store instructions 1510 that embody any one or more of the methods or functions described herein. During execution of the instructions 1510 by the machine 1500, the instructions 1510 may also reside, in whole or in part, within the memory 1512, within the storage unit 1514, within at least one of the processors 1504 (e.g., within a cache memory of the processor), or within any suitable combination thereof. Thus, the memory 1512, the storage unit 1514, and the memory of the processor 1504 are examples of machine-readable media.

[0135] I / O components 1518 may include a variety of components that receive input, provide output, generate output, send information, exchange information, capture measurements, etc. The specific I / O components 1518 included in a particular machine 1500 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It will be appreciated that I / O components 1518 may include Fig.15 15. The I / O components 1518 are grouped according to function only to simplify the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 1518 may include output components 1526 and input components 1528. The output components 1526 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 1528 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), touch input components (e.g., physical buttons, touch screen displays or other touch input components that provide the location and / or force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0136] In another example embodiment, the I / O component 1518 may include a biometric component 1530, a motion component 1534, an environmental component 1536, or a positioning component 1538, as well as various other components. For example, the biometric component 1530 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 1534 may include: an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental component 1536 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 1538 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure that can provide altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0137] A variety of technologies may be used to implement communications. I / O components 1518 may include communications components 1540 operable to couple machine 1500 to network 1532 or device 1520 via coupling 1524 and coupling 1522, respectively. For example, communications components 1540 may include a network interface component or other suitable device to interface with network 1532. In further examples, communications components 1540 may include a wired communications component, a wireless communications component, a cellular communications component, a near field communications (NFC) component, a Bluetooth component (e.g., Bluetooth low energy), Device 1520 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0138] In addition, the communication component 1540 can detect the identifier or can include a component that can be operated to detect the identifier. For example, the communication component 1540 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following items: a one-dimensional barcode, such as a universal product code (UPC) barcode; a multi-dimensional barcode, such as a quick response (QR) code, an Aztec code, a data matrix, a data glyph, a MaxiCode, a PDF4114, a super code, a UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 1540, such as a location obtained via Internet Protocol (IP) geolocation, a location obtained via Wi-Fi signal triangulation, a location obtained via detecting an NFC beacon signal that can indicate a specific location, etc.

[0139] Glossary

[0140] "Carrier signal" in this context refers to any intangible medium capable of storing, encoding or carrying instructions for execution by a machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device and using any of a number of well-known transmission protocols.

[0141] A "client device" in this context refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a PDA, a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics system, a game console, a set-top box, or any other communication device that a user may use to access a network.

[0142] A "communication network" in this context refers to one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless or cellular network, and the coupling to the network may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, including the third generation partnership project (3GPP) of 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other standards defined by various standard setting organizations, other remote protocols or other data transmission technologies.

[0143] "Machine-readable medium" in this context refers to a component, device, or other tangible medium capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be deemed to include a single medium or multiple media capable of storing instructions (e.g., centralized or distributed databases or associated caches and servers). The term "machine-readable medium" should also be deemed to include any medium or combination of multiple media capable of storing instructions (e.g., code) for machine execution, such that the instructions, when executed by one or more processors of the machine, cause the machine to perform any one or more of the methods described herein. Thus, "machine-readable medium" refers to a single storage device or device as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0144] A "component" in this context refers to a logical or device, physical entity having boundaries defined by function or subroutine calls, branch points, APIs, or other techniques that provide partitioning or modularization of specific processing or control functions. Components may be combined with other components via their interfaces to perform machine processes. A component may be a packaged functional hardware unit designed for use with other components, and is part of a program that generally performs specific functions of related functions. A component may constitute a software component (e.g., code embodied on a machine-readable medium) or a hardware component.

[0145] A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a physical manner. In various example embodiments, one or more computer systems (e.g., an independent computer system, a client computer system, or a server computer system) or one or more hardware components (e.g., a processor or a processor group) of a computer system can be configured by software (e.g., an application or an application part) to operate to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include a dedicated circuit or logic that is permanently configured to perform certain operations. A hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an ASIC. A hardware component may also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor.

[0146] Once configured by such software, the hardware component becomes a specific machine (or specific component of a machine) that is uniquely customized to perform the configured function, rather than a general-purpose processor. It will be understood that the decision to implement the hardware component mechanically in a dedicated and permanently configured circuit system or in a temporarily configured (e.g., configured by software) circuit system can be made for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a certain manner or perform certain operations described herein.

[0147] Considering implementations where hardware components are temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each of the hardware components at any one time in a timely manner. For example, where the hardware components include a general-purpose processor that is configured by software to be a special-purpose processor, the general-purpose processor can be configured as a respective different special-purpose processor (e.g., including different hardware components) at different times. The software configures one or more specific processors accordingly, for example, to constitute a specific hardware component at one time and to constitute a different hardware component at a different time.

[0148] Hardware components can provide information to other hardware components and receive information from other hardware components. Accordingly, the described hardware components can be considered to be communicatively coupled. In the case of multiple hardware components simultaneously, communication can be achieved by signal transmission between two or more hardware components in the hardware components (e.g., by appropriate circuits and buses). In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessed by multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled to the communication. Then, other hardware components can access the memory device at a subsequent time to retrieve and process the stored output. The hardware component can also initiate communication with an input or output device, and can operate on resources (e.g., the collection of information).

[0149] The various operations of the example methods described herein may be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the related operations. Whether temporarily configured or permanently configured, such a processor may constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some of the operations of the method may be performed by one or more processors or processor-implemented components.

[0150] In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some operations can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interface (API)). The execution of certain operations in the operation can be distributed between processors, not just resident in a single machine but deployed across multiple machines. In some example embodiments, the processor or the component implemented by the processor can be located in a single geographic location (e.g., in a home environment, an office environment, or a server cluster). In other example embodiments, the processor or the component implemented by the processor can be distributed across multiple geographical locations.

[0151] A "processor" in this context refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a CPU, a RISC processor, a CISC processor, a GPU, a DSP, an ASIC, an RFIC, or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0152] A "timestamp" in this context refers to a sequence of characters or coded information that identifies when some event occurred, for example, thereby giving a date and time of day, sometimes accurate to a fraction of a second.

Claims

1. A method comprising: accessing data describing a first user's grasp of an object; detecting a second user touching the object; as well as In response to detecting that the second user touches the object, the object is caused to provide interaction with the second user based on the data describing the first user's grip of the object.

2. The method according to claim 1, wherein: the data indicating the user's grip on a first area of ​​the object; Detecting that the second user touched the object includes determining that a second area of ​​the object touched by the second user at least partially overlaps the first area.

3. The method according to claim 1, wherein: Causing the object to provide interaction with the second user includes causing the object to provide tactile feedback to the second user.

4. The method according to claim 3, wherein: said data relating to said grasping of said object comprises a measurement of a force exerted on said object by said grasping, The intensity of the tactile feedback of the object is based on a measurement of the force exerted on the object by the grasp.

5. The method according to claim 3, wherein: The data related to the grasping of the object also includes a duration of the grasping; as well as The duration of the tactile feedback provided by the object is based on the duration of the grasping.

6. The method according to claim 1, wherein: The data related to the grasping of the object further includes an audio signal recorded during the first user's grasping of the object; as well as Enabling the object to provide interaction with the second user includes: An audio device is caused to render the audio signal, the audio device being communicatively coupled to the object.

7. The method according to claim 1, further comprising: In response to detecting that the second user is touching the object, causing presentation by a display device of the second user of a visual representation of the grip superimposed on the object.

8. The method according to claim 7, wherein: The visual representation comprises a fingerprint-like representation.

9. The method according to claim 1, further comprising: accessing volumetric content associated with a grasp of the object, the volumetric content comprising a volumetric representation of the first user's grasp of the object; as well as In response to detection of the object, causing a display device of the second user to present the volumetric content, such that presenting the volumetric content includes causing the display device to present a volumetric representation of the first user grasping the object superimposed on a real-world environment in a field of view of the second user.

10. The method of claim 1, wherein: The object is a first object; The method further comprises: detecting a third user touching the second object; and In response to detecting that the third user touches the second object, causing the second object to provide the interaction to the third user.

11. The method according to claim 10, further comprising: In response to detecting that the third user touches the second object, causing the first object to provide a second interaction with the second user.

12. A system comprising: one or more processors; as well as at least one memory storing instructions that, when executed by the one or more processors, configure the system to perform operations comprising: accessing data describing a first user's grasp of an object; detecting a second user touching the object; and In response to detecting that the second user touches the object, the object is caused to provide interaction with the second user based on the data describing the first user's grip of the object.

13. The system according to claim 12, wherein: the data indicating the user's grip on a first area of ​​the object; Detecting that the second user touched the object includes determining that a second area of ​​the object touched by the second user at least partially overlaps the first area.

14. The system according to claim 12, wherein: Causing the object to provide interaction with the second user includes causing the object to provide tactile feedback to the second user.

15. The system of claim 14, wherein: said data relating to said grasping of said object comprises a measurement of a force exerted on said object by said grasping, The intensity of the tactile feedback of the object is based on a measurement of the force exerted on the object by the grasp.

16. The system of claim 14, wherein: The data relating to the grasping of the object further comprises a duration of the grasping; and The duration of the tactile feedback provided by the object is based on the duration of the grasping.

17. The system of claim 12, wherein: The data related to the grasping of the object further includes an audio signal recorded during the grasping of the object by the first user; as well as Enabling the object to provide interaction with the second user includes: An audio device is caused to render the audio signal, the audio device being communicatively coupled to the object.

18. The system of claim 12, further comprising: In response to detecting that the second user is touching the object, causing presentation by a display device of the second user of a visual representation of the grip superimposed on the object.

19. The system of claim 18, wherein: The visual representation comprises a fingerprint-like representation.

20. A non-transitory computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising: accessing data describing a first user's grasp of an object; detecting a second user touching the object; as well as In response to detecting that the second user touches the object, the object is caused to provide interaction with the second user based on the data describing the first user's grip of the object.

Images