Gesture-based selection and transmission of content

Abstract

Examples of disclosure relate to systems and methods for capturing and transmitting content associated with objects displayed in a three-dimensional environment. While a computer system displays a three-dimensional environment that includes a first object and a first electronic device, the computer system detects that a user performs a first gesture directed at the first object. In response to the detection of the first gesture directed at the first object, the computer system gathers information associated with the first object. The computer system detects that the user performs a second gesture directed at a first electronic device (e.g., a laptop or other computing device).In response to the recognition of the second gesture directed at the first electronic device, the computer system transmits the collected information associated with the first object to the first electronic device.

Description

CROSS-REFERENCE TO RELATED REGISTRATIONS

[0001] This application claims the benefit of preliminary U.S. patent application no. 63 / 700,667, filed on September 28, 2024, and U.S. patent application no. 19 / 331,960, filed on September 17, 2025, the contents of which are incorporated herein by reference in their entirety for all purposes. Scope of Disclosure: This generally relates to systems and methods for gesture-based selection and transmission of content within a three-dimensional environment. BACKGROUND OF THE REVELATION

[0002] Some computer systems include cameras configured to capture images and / or videos. Using these cameras, some computer systems display three-dimensional environments that include representations of both physical objects in the real world and virtual objects. SUMMARY OF THE REVELATION

[0003] Some examples of the disclosure relate to systems and methods for capturing and transmitting content associated with objects represented in a three-dimensional environment. In one or more examples, while a computer system represents a three-dimensional environment that includes a first object and optionally a first electronic device, the computer system detects that a user is performing a first gesture directed at the first object. In one or more examples, in response to the detection of the first gesture directed at the first object, the computer system collects information associated with the first object and stores the collected information in a memory associated with the computer system.In one or more examples, and after the information associated with the first object has been stored in the memory allocated to the electronic device, the device detects that the user is performing a second gesture (optionally different from the first gesture) directed at the first electronic device (e.g., a laptop or other computer device located in the user's physical environment and visible in the displayed three-dimensional environment). In one or more examples, in response to the detection of the second gesture directed at the first electronic device, the computer system transfers the collected information associated with the first object to the first electronic device.

[0004] In one or more examples, the collected information associated with the first object includes a visual scan of the first object, captured by one or more cameras of the computer system. Additionally or alternatively, when capturing the visual scan of the first object, the computer system compares the visual scan (including any text captured as part of the visual scan or subsequently, such as using optical character recognition) against one or more database entries to determine whether the first object is relevant to one or more elements of the media content. If a match is found, information about the relevant media content is added to the collected information associated with the first object. In one or more examples, the first object may include an electronic device running one or more software applications.Optionally, in response to the detection of the first gesture directed at the electronic device, the computer system collects information about the one or more software applications running on the electronic device.

[0005] In one or more examples, the first electronic device includes a media device, such as a smart speaker, a music player, and / or a video player. In some examples, in response to the detection that the second gesture is directed at a media device, the computer system transmits information about media content relevant to the first object, enabling the media player to play the media content relevant to the first object. In one or more examples, the second gesture may be directed at the computer system itself. Optionally, in response to the detection that the second gesture is directed at the computer system, the computer system displays a visual representation of the collected information associated with the first object in the three-dimensional environment.

[0006] The complete descriptions of these examples are included in the drawings and the detailed description, and it is understood that this summary does not in any way limit the scope of the revelation. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] For a better understanding of the various examples described herein, please refer to the detailed description below, along with the accompanying drawings. The same reference numerals throughout the drawings often refer to corresponding parts. Fig. Figure 1 illustrates an electronic device that represents an augmented reality environment, according to some examples from the revelation. Fig. Figure 2 illustrates a block diagram of an exemplary architecture for a device according to some examples from the revelation. Fig. 3A to 3N illustrate an example system for collecting and transmitting content in a three-dimensional environment according to some examples from the Revelation. Fig. Figure 4 illustrates an exemplary flowchart that demonstrates a procedure for collecting and transmitting content within a three-dimensional environment according to examples from the Revelation. DETAILED DESCRIPTION

[0008] Some examples of the disclosure relate to systems and methods for capturing and transmitting content associated with objects displayed in a three-dimensional environment. In one or more examples, while a computer system represents a three-dimensional environment including a first object and a first electronic device, the computer system detects that a user performs a first gesture directed at the first object. In one or more examples, in response to the detection of the first gesture directed at the first object, the computer system collects information associated with the first object and stores the collected information in a memory associated with the electronic device.In one or more examples, and after the information associated with the first object has been stored in the memory assigned to the electronic device, the device detects that the user is performing a second gesture directed at the first electronic device (e.g., a laptop or other computer device located in the user's physical environment and visible in the displayed three-dimensional environment). In one or more examples, in response to the detection of the second gesture directed at the first electronic device, the computer system transfers the collected information associated with the first object to the first electronic device.

[0009] In one or more examples, the information associated with the first object includes a visual scan of the first object, captured by one or more cameras of the computer system. Additionally or alternatively, when capturing the visual scan of the first object, the computer system compares the visual scan (including any text captured as part of the visual scan) with one or more database entries to determine whether the first object is relevant to one or more elements of the media content. If a match is found, information about the relevant media content is added to the collected information associated with the first object. In one or more examples, the first object may include an electronic device running one or more software applications.Optionally, in response to the detection of the first gesture directed at the electronic device, the computer system collects information about the one or more software applications running on the electronic device.

[0010] In one or more examples, the first electronic device includes a media device, such as a smart speaker, a music player, and / or a video player. In some examples, in response to the detection that the second gesture is directed at a media device, the computer system transmits information about media content relevant to the first object, enabling the media player to play the media content relevant to the first object. In one or more examples, the second gesture may be directed at the computer system itself. Optionally, in response to the detection that the second gesture is directed at the computer system, the computer system displays a visual representation of the collected information associated with the first object in the three-dimensional environment.

[0011] Fig. Figure 1 illustrates a computer system 101 that presents an augmented reality (XR) environment (e.g., a computer-generated environment that optionally includes representations of physical and / or virtual objects) according to some examples from Revelation. In some examples, as in Fig. As shown in Figure 1, Computer System 101 is a head-mounted display or other head-mounted device configured to be worn on the head of a user of Computer System 101. Additionally or alternatively, Computer System 101 can be any computer system (such as a mobile phone) in which one or more cameras can generate images of the user's surroundings and overlay virtual objects onto a displayed environment. Examples of Computer System 101 are given below with reference to the architectural block diagram of [reference missing]. Fig. 2 described. As in Fig. As shown in Figure 1, the computer system 101 and the table 106 are located in a physical environment. The physical environment can include physical features such as a physical surface (e.g., floor, walls) or a physical object (e.g., table, lamp, etc.). In some embodiments, the computer system 101 can be configured to detect and / or capture images of the physical environment, including the table 106 (illustrated in the field of view of the computer system 101).

[0012] In some examples, such as in Fig. As shown in Figure 1, the computer system 101 includes one or more internal image sensors 114a that are directed towards a user's face (e.g., eye-tracking cameras, which are described below with reference to Fig. 2 (described). In some examples, internal image sensors 114a are used for eye tracking (e.g., to detect the user's gaze). Internal image sensors 114a are optionally located in the left and right areas of the display 120 to enable eye tracking of the user's left and right eyes. In some examples, the computer system 101 also includes external image sensors 114b and 114c, which are directed outwards from the user, to detect and / or capture the physical environment of the computer system 101 and / or movements of the user's hands or other body parts.

[0013] In some examples, the display 120 has a user-visible field of view (e.g., which may or may not correspond to the field of view of the external image sensors 114b and 114c). Since the display 120 is optionally part of a head-mounted device, its field of view is optionally the same as, or similar to, the user's eye field of view. In other examples, the display 120's field of view may be smaller than the user's eye field of view. In some examples, the computer system 101 may be an optically transparent device, with the display 120 being a transparent or translucent display through which parts of the physical environment can be directly viewed. In some examples, the display 120 may be contained within a transparent lens and overlap all or part of the transparent lens.In other examples, computer system 101 can be a video passthrough device where display 120 is an opaque display configured to show images of the physical environment captured by external image sensors 114b and 114c. Although only a single display 120 is shown, it is understood that display 120 can also include a stereo pair of displays.

[0014] In some examples, computer system 101 can be configured in response to a trigger to display a virtual object 104 in the XR environment, which is triggered by a Fig. The cube shown in Figure 1 is not present in the physical environment but is displayed in the XR as positioned on the top of table 106 in the real world (or a representation thereof). Optionally, the virtual object 104 can be displayed on the surface of table 106 in the XR environment via the display 120 of computer system 101 in response to the detection of the flat surface of table 106 in the physical environment 100.

[0015] In some examples, display 120 is provided as a passive component (e.g., instead of an active component) within computer system 101. For example, as mentioned above, display 120 may be a transparent or translucent display and may not be configured to display virtual content (e.g., images of the physical environment captured by external image sensors 114b and 114c and / or the virtual object 104). Alternatively, in some examples, computer system 101 does not include display 120. In some examples where display 120 is provided as a passive component or is not included in computer system 101, computer system 101 may nevertheless include sensors (e.g., internal image sensor 114a and / or external image sensors 114b and 114c) and / or other input devices, such as one or more of those listed below with reference to Fig. 2 described components.

[0016] It is understood that virtual object 104 is a representative virtual object, and one or more different virtual objects (e.g., of different dimensions, such as two-dimensional or other three-dimensional virtual objects) can be enclosed and rendered within a three-dimensional XR environment. For example, the virtual object can represent an application or a user interface displayed in the XR environment. In some examples, the virtual object can represent content corresponding to the application and / or displayed via the user interface in the XR environment. In some examples, virtual object 104 is optionally configured to be interactive and responsive to user input (e.g., air gestures such as pinch air gestures, tap air gestures, and / or touch air gestures), allowing a user to virtually touch, tap, move, rotate, or otherwise interact with virtual object 104.

[0017] In some examples, displaying an object in a three-dimensional environment may involve interaction with one or more user interface objects within that environment. For instance, initiating the display of the object in the three-dimensional environment may involve interaction with one or more virtual options / affordances displayed within that environment. In some examples, a user's gaze may be tracked as input to identify one or more virtual options / affordances available for selection when the display of an object in the three-dimensional environment is initiated. The gaze may, for example, be used to identify one or more virtual options / offers to be selected using further selection input. In some examples, a virtual option or affordance may be...A virtual offering can be selected using hand tracking input, which is captured via an input device communicating with the computer system. In some examples, objects displayed in the three-dimensional environment can be moved and / or reoriented within the three-dimensional environment in accordance with the motion input captured by the device.

[0018] The following explanation describes a computer system that communicates with a display-generating component and one or more input devices. It is understood that the computer system optionally communicates with one or more other physical user interface devices, such as a touch-sensitive surface, a physical keyboard, a mouse, a joystick, a hand-tracking device, an eye-tracking device, a stylus, etc. Furthermore, it is understood, as described above, that the described computer system, the display, and the touch-sensitive surface are optionally distributed among two or more devices.Therefore, as used in this disclosure, information displayed on or by the computer system is optionally used to describe information output by the computer system for display on a separate display device (touch-sensitive or not). Similarly, as used in this disclosure, input received by the computer system (e.g., touch input received on a touch-sensitive surface of the computer system or touch input received on the surface of an input pen) is optionally used to describe input received on a separate input device from which the computer system receives input information.

[0019] The device typically supports a variety of applications, such as one or more of the following: a drawing application, a presentation application, a word processing application, a website creation application, a disc authoring software development application, a spreadsheet application, a gaming application, a telephone application, a video conferencing application, an email application, an instant messaging application, a training support application, a photo management application, a digital camera application, a digital video camera application, an internet browsing application, a digital music playback application, a television channel browsing application, and / or a digital video playback application.

[0020] Fig. Figure 2 illustrates a block diagram of an exemplary architecture for a system 201 according to some examples of the disclosure. In some examples, device 201 includes one or more computer systems. For example, computer system 201 may be a portable device, an auxiliary device communicating with another device, a head-mounted display, etc. In some examples, computer system 201 corresponds to the above with reference to Fig. 1 described computer system 101.

[0021] As in Fig. As illustrated in Figure 2, the computer system 201 optionally includes various sensors, such as one or more hand tracking sensors 202, one or more position sensors 204, one or more image sensors 206 (optionally the internal image sensors 114a and / or external image sensors 114b and 114c in Fig. 1 accordingly), one or more touch-sensitive surfaces 209, one or more motion and / or orientation sensors 210, one or more eye-tracking sensors 212, one or more microphones 213 or other audio sensors, one or more body-tracking sensors (e.g., upper body and / or head-tracking sensors), one or more display-generating components 214, optionally the display 120 in Fig. 1 accordingly, one or more loudspeakers 216, one or more processors 218, one or more memory units 220 and / or communication switching logic 222. One or more communication buses 208 are optionally used for communication between the aforementioned components of the computer system 201.

[0022] The 222 communication switching logic optionally includes switching logic for communicating with computer systems, networks such as the Internet, intranets, wired and / or wireless networks, cellular networks, and wireless local area networks (LANs). The 222 communication switching logic optionally includes switching logic for communicating using near-field communication (NFC) and / or short-range communication, such as Bluetooth®.

[0023] The processor(s) 218 ​​include one or more general-purpose processors, one or more graphics processors, and / or one or more digital signal processors. In some examples, the memory 220 is a non-transitory, computer-readable storage medium (e.g., flash memory, random-access memory, or other volatile or non-volatile memory or storage) that stores computer-readable instructions configured to be executed by one or more processors 218 to perform the techniques, processes, and / or procedures described below. In some examples, memory 220 may include more than one non-transitory, computer-readable storage medium. A non-transitory, computer-readable storage medium can be any medium (e.g.,(with the exception of a signal), which can contain or store tangible computer-executable instructions for use by, or in connection with, the instruction execution system, instruction execution device, or instruction execution apparatus. In some examples, the storage medium is a transient computer-readable storage medium. In some examples, the storage medium is a non-transient computer-readable storage medium. The non-transient computer-readable storage medium may include, but is not limited to, magnetic, optical, and / or semiconductor storage. Examples of such storage include magnetic disks, optical disks based on Compact Disc (CD), Digital Versatile Disc (DVD), or Blu-ray technologies, and persistent semiconductor storage such as flash memory, solid-state drives, and the like.

[0024] In some examples, a display generating component(s) 214 includes a single display (e.g., a liquid crystal display (LCD), organic light-emitting diode (OLED), or other types of displays). In some examples, the display generating component(s) 214 includes multiple displays. In some examples, the display generating component(s) 214 may include a touch-enabled display (e.g., a touchscreen), a projector, a holographic projector, a retinal projector, a transparent or translucent display, etc. In some examples, the computer system 201 includes a touch-sensitive surface(s) 209 for receiving user input, such as typing, swiping, or other gestures. In some examples, the display generating component(s) 214 and the touch-sensitive surface(s) 209 form a touch-sensitive display (e.g., a touchscreen).a touchscreen that is integrated into computer system 201 or is located outside of computer system 201 and is in contact with computer system 201).

[0025] The computer system 201 optionally includes one or more image sensors 206. The one or more image sensors 206 optionally include one or more visible light image sensors, such as CCD (charge-coupled device) sensors and / or CMOS (complementary metal-oxide semiconductor) sensors, which can be operated to obtain images of physical objects from the real environment. The image sensor(s) 206 also optionally include one or more infrared (IR) sensors, such as a passive or active IR sensor, for detecting infrared light from the real environment. For example, an active IR sensor includes an IR emitter for emitting infrared light into the real environment. The image sensor(s) 206 also optionally include one or more cameras configured to detect movement of physical objects in the real environment.The image sensor(s) 206 optionally includes one or more depth sensors configured to detect the distance of physical objects from the computer system 201. In some examples, information from one or more depth sensors can enable the device to identify objects in the real-world environment and distinguish them from other objects in the real-world environment. In some examples, one or more depth sensors can enable the device to determine the texture and / or topography of objects in the real-world environment.

[0026] In some examples, the computer system 201 uses CCD sensors, event cameras, and depth sensors in combination to detect the physical environment around the computer system 201. In some examples, the image sensor(s) 206 includes a first image sensor and a second image sensor. The first and second image sensors work together and are optionally configured to capture different information about physical objects in the real-world environment. In some examples, the first image sensor is a visible light image sensor, and the second image sensor is a depth sensor. In some examples, the computer system 201 uses the image sensor(s) 206 to detect the position and orientation of the computer system 201 and / or the display generating component(s) 214 in the real-world environment.For example, the computer system 201 uses the image sensor(s) 206 to track the position and orientation of the display generating component(s) 214 relative to one or more fixed objects in the real environment.

[0027] In some examples, the computer system 201 includes microphone(s) 213 or other audio sensors. The computer system 201 optionally uses the microphone(s) 213 to detect sound from the user and / or from the user's real-world environment. In some examples, the microphone(s) 213 include an array of microphones (a plurality of microphones) that optionally work together to identify ambient noise or to localize the sound source within the real-world environment.

[0028] The computer system 201 includes location sensor(s) 204 to detect the location of the computer system 201 and / or the display generating component(s) 214. The location sensor(s) 204 may include a GPS (Global Positioning System) receiver that receives data from one or more satellites and enables the computer system 201 to determine the absolute position of the device in the physical world.

[0029] The computer system 201 includes orientation sensor(s) 210 for detecting the orientation and / or movement of the computer system 201 and / or the display-generating component(s) 214. For example, the computer system 201 uses orientation sensor(s) 210 to track changes in the position and / or orientation of the computer system 201 and / or the display-generating component(s) 214, for example, relative to physical objects in the real-world environment. The orientation sensor(s) 210 optionally includes one or more gyroscopes and / or one or more accelerometers.

[0030] In some examples, the computer system 201 includes hand tracking sensor(s) 202 and / or eye tracking sensor(s) 212 (and / or other body tracking sensor(s), such as leg, torso, and / or head tracking sensor(s)). The hand tracking sensor(s) 202 is / are configured to track the position / location of one or more sections of the user's hands and / or movements of one or more sections of the user's hands in relation to the augmented reality relative to the display generating component(s) 214 and / or relative to another defined coordinate system. The one or more eye tracking sensor(s) 212 is / are configured to track the position and movement of a user's gaze (eyes, face or generally the head) in relation to the real world or the augmented reality environment and / or relative to the one or more display generating component(s) 214.In some examples, the one or more hand tracking sensor(s) 202 and / or the one or more eye tracking sensor(s) 212 are implemented together with the one or more display generation component(s) 214. In other examples, the one or more hand tracking sensor(s) 202 and / or the one or more eye tracking sensor(s) 212 are implemented separately from the one or more display generation component(s) 214.

[0031] In some examples, one or more hand tracking sensors 202 (and / or other body tracking sensors such as leg, torso, and / or head tracking sensors) can use one or more image sensors 206 (e.g., one or more IR cameras, 3D cameras, depth cameras, etc.) that detect three-dimensional information from the real world, including one or more body parts (e.g., hands, legs, or torso of a human user). In some examples, the hands can be resolved with sufficient resolution to distinguish fingers and their respective positions. In some examples, one or more image sensors 206 are positioned relative to the user to define a field of view for the one or more image sensors 206 and an interaction space in which finger / hand position, orientation, and / or movement detected by the image sensors are used as inputs (e.g.,(to distinguish between the user's hand at rest and the hands of other people in the real-world environment). Tracking fingers / hands for input (e.g., gestures, touch, taps, etc.) can be advantageous because the user does not need to touch, hold, or wear any kind of beacon, sensor, or other device.

[0032] In some examples, the eye-tracking sensor(s) 212 include at least one eye-tracking camera (e.g., infrared cameras (IR cameras)) and / or light sources (e.g., IR light sources, such as LEDs) that emit light toward a user's eyes. The eye-tracking cameras can be directed toward a user's eyes to receive reflected IR light from the light sources directly or indirectly from the eyes. In some examples, both eyes are tracked separately by their respective eye-tracking cameras and light sources, and a gaze can be determined from tracking both eyes. In some examples, one eye (e.g., a dominant eye) is tracked by one or more of their respective eye-tracking cameras / light sources.

[0033] The computer system 201 is not based on the components and configuration from Fig. 2 is limited, but can include fewer, different, or additional components in several configurations. In some examples, Computer System 201 can be implemented between two computer systems (e.g., as a system). In some of these examples, each (or multiple) computer system can include one or more of the components discussed above, such as various sensors, one or more display-generating components, one or more loudspeakers, one or more processors, one or more memory units, and / or communication switching logic. A person or persons using Computer System 201 are optionally referred to herein as users of the device.

[0034] Attention now focuses on interactions with physical objects in the physical environment (e.g., represented in the three-dimensional environment). These interactions can also be applied to one or more virtual objects and / or a visual representation of real-world objects displayed in a three-dimensional environment rendered on a computer system (e.g., corresponding to Computer System 201).

[0035] Fig. 3A to 3N illustrate an example system for collecting and transmitting content in a three-dimensional environment according to some examples from the Revelation. Fig. Figure 3A illustrates an exemplary three-dimensional environment 302 rendered by the computer system 101. In one or more examples, the three-dimensional environment 302 rendered by the computer system 101 includes one or more representations of physical objects located in the surrounding real-world environment of the user of the computer system 101. As shown in Fig. As illustrated in Figure 3A, the three-dimensional environment 302 includes, for example, Table 312 (at least the portion of Table 312 visible in the field of view of Computer System 101 and displayed on Display 120 of Computer System 101). In one or more examples, real objects located on or near Table 312 are also represented as part of the three-dimensional environment 302. For example, and as shown in Figure 3A, the three-dimensional environment 302 includes, for example, Table 312 ... Fig. 3A illustrates the three-dimensional environment 302, book 308, a smart speaker 306, music album 310, and laptop 304.

[0036] In one or more examples, Laptop 304 is a laptop controlled by, or authorized to transmit messages to, the user of Computer System 101 from Computer System 101 (for example, because the user of both Laptop 304 and Computer System 101 has registered and / or logged in to each device using the same credential). Thus, in some examples, the user of Computer System 101 is authorized to transmit electronic data from Computer System 101 to Laptop 304 and / or to receive data from Laptop 304 to Computer System 101. In one or more examples, if an authorized user of Laptop 304 wishes to obtain a visually scanned copy of Book 308 (e.g.,(the page and / or pages visible in the three-dimensional environment 302), the user would have to manually create a scan of the book 308 by placing the book in a special scanning device that is communicatively coupled to the laptop 304, so that the scanner would capture an image and then transmit it to the laptop 304. In one or more examples, and as explained in more detail below, the user of the computer system 101 can use the computer system 101 to perform scanning and other data acquisition operations initiated by a user-executed gesture that can be transmitted to an electronic device, as in . Fig. 3B to 3G illustrated.

[0037] In one or more examples, the data capture gesture involves bringing the fingers of one hand together in a pincer grip directed at an object for a threshold period and / or pulling the hand while maintaining the pincer grip to a threshold distance away from the object toward the computer system or the user of the computer system, as in Fig. 3B to 3C illustrated. As in Fig. As illustrated in Figure 3B, the user of computer system 101, for example, initiates the execution of a gesture 316 using hand 314, which is directed towards book 308 (e.g., the gesture 316 is performed at a location in the three-dimensional environment that overlaps with and / or is near book 308 (using a raycast or other applicable method), so that computer system 101 recognizes that the gesture 316 is directed towards book 308). Optionally, and as shown in Fig. Figure 3B illustrates that gesture 316 is initiated when the device detects that the user's hand 314 is extended with all fingers, followed by a movement of hand 314 as shown in Fig. 3C illustrates this. Optionally, in one or more examples, the fingers are separated when initiating gesture 316 and are not necessarily extended (e.g., partially extended, with the back of the hand facing the user).

[0038] In one or more examples, and as in Fig. As illustrated in Figure 3C, Computer System 101 recognizes the continuation of gesture 316 and, in particular, that the outstretched hand 314, directed towards book 308, moves in such a way that one or more fingers of hand 314 come together (e.g., are squeezed together) and the hand moves away from the target (such as book 308) as if the hand were extracting the information from book 308, as in Fig. 3C illustrates this. In some examples, gesture 316 is initiated when the fingertips are in Fig. 3C come together and are not initiated by the outstretched hand, as in Fig. 3B illustrates this. In such a scenario, the state of the hand is in Fig. 3B before the start of the gesture, during the Fig. 3C illustrated the state of the hand, representing the hand performing gesture 316. In the example of Fig. 3C recognizes that the gesture 316, which is in Fig. 3B, initiated by extending hand 314 towards book 308, is continued by the user withdrawing one or more fingers of hand 314 so that the fingers come together in a pincer grip gesture. In the example of Fig. 3C shows two fingers touching, but the number of fingers shown is exemplary and could include more or fewer. In particular, in some examples, all five fingers can be used for the gesture, so that all five fingers touch and are withdrawn to perform gesture 316. [[Here or elsewhere, discuss in more detail gestures that can be used, such as physical touch and other modalities, such as using gaze in combination with hand gestures.]]

[0039] In one or more examples, and in response to the detection that the user has performed gesture 316 and is continuing to maintain the gesture (e.g., keeping their fingers pressed together and withdrawing their hand as described above), computer system 101 begins collecting information related to book 308 (e.g., the object in the three-dimensional environment 302 toward which gesture 316 is directed). In some examples, the information collected includes, but is not limited to, a visual scan of book 308 using one or more cameras 114a to c and / or text optically detected on book 308. Information collected may be limited to visually detected information or may include information obtained using visually detected information (e.g., retrieving the content of an observed book title, determining the book name based on a passage seen in the book, etc.).In one or more examples, computer system 101 displays a visual indicator 318 configured to provide a visual representation of the progress of collecting information associated with book 308 in response to the detection of the gesture 316 directed at book 308. Optionally, the visual indicator includes a progress meter that gradually fills up over time as information is collected. For example, in the non-restrictive example, visual indicator 318 includes... Fig. 3C optionally displays an icon representing data collection, with a progress ring around the icon. Alternatively and / or additionally, a visual representation of the gesture's progress can be displayed. For example, the initial pinch grip (see above) causes a progress indicator to appear, and this progress indicator can be configured to illustrate how far the user needs to retract their pinched fingers (as described above) to complete the gesture.

[0040] In one or more examples, computer system 101 continues collecting the information associated with book 308 as long as the computer system detects the user's hold gesture 316 and / or computer system 101 detects that the collection of the information associated with book 308 is complete. For example, in Fig. Visualized in 3D, Computer System 101 recognizes that the user ended gesture 316 by releasing their fingers before Computer System 101 had gathered all the information associated with Book 308. In response to recognizing that the user ended gesture 316 in Fig. Once 3D has finished, Computer System 101 terminates the data collection process without completing it and does not save any information collected before Computer System 101 determined that the gesture was complete. In this way, Computer System 101 provides the user with the option to cancel a previously initiated data collection process (e.g., an opportunity for the user to change their mind).

[0041] Alternatively, in one or more examples, Computer System 101 terminates the collection of information linked to Book 308 in response to the completion of the process, as in Fig. 3E illustrates this. In the example of Fig. 3E and in response to Computer System 101 determining that the user maintained gesture 316 (and / or completed the gesture) during the collection process, Computer System 101 completes the collection process. In one or more examples, Computer System 101 displays a visual indicator 318 indicating completion. For example, in Fig. As shown in 3E, the visual indicator 318 has its appearance from Fig. 3C to the one from in Fig. The visual indicator shown in Figure 3E is modified to provide a visual indication that the information gathering process is complete. For example, the visual indication can change from a symbol indicating data gathering to a symbol indicating completion (e.g., a check mark), or to an indication of the type of data gathered (e.g., a document). Additionally or alternatively, the visual indication can stop displaying the progress indicator (e.g., stop displaying a ring). It is understood that other changes to the visual indicator are also possible, such as a change in the color or opacity of the visual indicator. In one or more examples, in response to determining that the gathering process is complete, Computer System 101 stores the gathered information in a memory allocated to the computer system (e.g., a memory location).a memory that is physically located on computer system 101 and / or a memory that is communicatively coupled to computer system 101).

[0042] In one or more examples, the information stored in a memory allocated to computer system 101 in response to gesture 316 in Fig. 3B to 3E are stored and transferred to an electronic device. In one or more examples, the transfer of the information to another electronic device requires that the electronic device be within the field of view of the user of computer system 101 and / or within the field of view of the sensors of computer system 101, as shown in Fig. 3F to 3G illustrate. In one or more examples, the data transmission gesture is the reverse of the data collection gesture. For example, the data transmission gesture includes spreading the fingers of a hand (releasing a pincer grip) directed at an electronic device, and / or pushing the hand, upon releasing it over a threshold distance, toward the electronic device (away from the computer system or the user of the computer system), as shown in Fig. 3F to 3G illustrated. As in Fig. As illustrated in Figure 3F, Computer System 101 recognizes that the user is performing gesture 320 directed at Laptop 304 and, in response, transmits the collected and stored information associated with Book 308 to Laptop 304. In one or more examples, Computer System 101 recognizes gesture 320, which is performed when hand 314 is extended and directed at an electronic device, such as Laptop 304.

[0043] In one or more examples and in response to the recognition of gesture 320 (and if the computer system has stored 101 information, which was processed using the above in relation to Fig. (The information associated with Book 308 has been collected in the process described in 3B to 3E), Computer System 101 transfers the stored information associated with Book 308 to Laptop 304 using a pre-established communication link (e.g., a wired, wireless, and / or cloud-based communication link) between Computer System 101 and Laptop 304 (as described above). In one or more examples, and similarly to Gesture 316, which is used to initiate the collection of the information associated with Book 308, Computer System 101 terminates the process of transferring the information to Laptop 304 if Computer System 101 detects that Gesture 320 has been terminated by the user before the transfer of the information associated with Book 308 to Laptop 304 is complete.In one or more examples, during the process of transferring information to Laptop 304, Computer System 101 displays visual indicator 318 to provide a visual indication of the progress of the information transfer (similar to the example of visual indicator 318 described above). In some examples, visual indicator 318 changes to indicate that the transfer of the information associated with Book 308 is complete, as in . Fig. 3G illustrated.

[0044] In one or more examples, and as in Fig. 3G illustrates that Computer System 101 displays visual indicator 318, which now indicates that the transfer of the information associated with the book from Computer System 101 to Laptop 304 is complete. Furthermore, Laptop 304 shows how in Fig. 3G optionally illustrates the scanned image 322 from book 308, which was collected and transferred to the laptop in the examples described above. In some examples, and similar to the information gathering gesture example described above, computer system 101 can display a visual indicator showing the progress of the gesture.

[0045] In one or more examples, Computer System 101 can gather other types of information in addition to obtaining a visual scan of an object such as Book 308 in a three-dimensional environment. For example, and as described in more detail below, Computer System 101 can use the scan obtained from the processes described above to obtain textual or graphical information (such as images appearing in Book 308). In one or more examples, Computer System 101 compares the textual or graphical information against one or more databases (such as a media database) to determine whether Book 308 is related to any media content (such as music, movies, and / or television programs). In one or more examples, all relevant entries found as a result of the comparison can be recorded and stored along with the information associated with Book 308.

[0046] In one or more examples, Computer System 101 can transfer the information / data collected and linked to Book 308 to other device types in addition to computing devices such as Laptop 304. For example, Computer System 101 can transfer the information linked to an object to a multimedia device, such as a smart TV, a smart hub, or a smart speaker, as in Fig. 3H to 3J illustrated. In the example of Fig. 3H detects that Computer System 101 has initiated the collection of information related to Music Album 310 by performing gesture 324 (similar to gesture 316 described above), and in response, Computer System 101 collects information related to Music Album 310 (as indicated by visual indicator 326). Optionally, the information collected related to Music Album 310 includes a visual scan of Music Album 310 and also includes comparing the information retrieved from the visual scan (e.g., lyrics and graphics) with one or more media databases to determine if there are any media content elements relevant to Music Album 310.In the example of music album 310, the images and text obtained by scanning music album 310 can be compared with a music database to determine if there are any music albums and / or songs associated with the album, and any matches can be added to the collected information associated with album 310. In one or more examples, computer system 101 can transfer sections of the collected information associated with an object (such as music album 310) based on the type of electronic device to which computer system 101 recognizes the user wishes to transfer the collected information.For example, if it is detected that, in response to the recognition, the user is transferring the collected information about the music album 310 to a smart speaker, the computer system transmits information about the music that is part of the collected information, as in . Fig. 3I illustrates.

[0047] In the example of Fig. 3I Computer system 101 recognizes that the user is performing gesture 328 directed at the smart speaker 306. In one or more examples, computer system 101 determines that the smart speaker 306 is a device that plays music. For example, computer system 101 determines the category of electronics to which the smart speaker 306 belongs by using the scan data captured as part of the collected information associated with the smart speaker 306. In one or more examples, and in response to determining the type of electronic device to which gesture 328 is directed (e.g., smart speaker 306), computer system 101 transmits only the portion of the collected information associated with music album 310 to the smart speaker 306.Since the smart speaker 306 is a music player, the computer system 101 can, for example, transmit the portion of the collected information about the music that was found to be relevant to the music album 310 (through the search process in the media databases described above). In one or more examples, and similar to the examples described above, the computer system 101 displays a visual indicator 330, which provides the user with a visual indication of the progress of the data transfer from the computer system 101 to the smart speaker 306. In one or more examples, after the transfer is complete, the smart speaker can begin playing the music (e.g., song and / or album) that was included in the information transmitted from the computer system 101 to the smart speaker 306, as shown in [reference]. Fig. Figure 3J illustrates this. In some embodiments, the device receiving the transmission can perform an operation with the received information based on its content. In the example of a smart speaker, the smart speaker might play a song based on the information contained in the received information (e.g., a song title, artist information, etc.).

[0048] As in Fig. As illustrated in 3J, in response to receiving the portion of the collected information relating to relevant music linked to music album 310, the smart speaker 306 plays the music identified as being linked to music album 310. In one or more examples, as part of the process of transmitting the collected information to the smart speaker 306, computer system 101 can send a command to the smart speaker 306 instructing it to play the music linked to the information transmitted by computer system 101. Alternatively, the smart speaker 306 automatically begins playing the music found in the transmitted information as soon as it receives the information from computer system 101.

[0049] In some examples, the above-described process of collecting information from computer system 101 can be used to generate virtual content in the three-dimensional environment 302, as in Fig. 3K to 3N illustrated. In the example of Fig. 3K includes the three-dimensional environment 302, the same elements as in the example of Fig. 3A is represented by computer system 101 (e.g., laptop 304, smart speaker 306, book 308, and music album 310). In the example of Fig. However, Laptop 304 runs a display application 330, which is shown on the display of Laptop 304. In one or more examples, the user can gather information about the application running on Laptop 304 by using the same or similar gestures described above for gathering information about objects in the three-dimensional environment 302, as shown in Fig. 3L illustrates.

[0050] As in Fig. Figure 3L illustrates that, in response to the recognition of gesture 332 performed by the hand 314 of the user of computer system 101, computer system 101 initiates a process to collect information related to application 330. In one or more examples, computer system 101 recognizes that the object toward which gesture 332 is directed is a computing device on which an application is running and, in response to this determination, sends a request to laptop 304 to provide information about the application 330 running on it. This information includes, for example, information about the file and / or files that application 330 uses during operation.In some examples, Laptop 304, in response to a request from Computer System 101, transmits information about Application 330, enabling Computer System 101 to display and run Application 330 in the three-dimensional environment 302 shown on Display 120 (e.g., in addition to or instead of a display from Laptop 304). In one or more examples, once the information associated with Application 330 running on Laptop 304 has been collected by Computer System 101, the computer system can, in response to the detection of a gesture, display the application and / or a visual representation of the application in the three-dimensional environment 302, as shown in [reference]. Fig. 3M illustrates.

[0051] In the example of Fig. 3M recognizes that Computer System 101 recognizes that the user is directing gesture 332 at Computer System 101 itself, thus expressing a desire for Computer System 101 to display a visual representation of the information collected and associated with Application 330. In one or more examples, gesture 332 shares one or more properties with the one described above regarding Fig. 3F to 3G described gesture 316. In response to the recognition of the gesture 332 directed at computer system 101, the computer system accesses the memory in which the collected information associated with application 330 is stored and displays a visual representation of the application in the three-dimensional environment 302 based on the information stored in memory, as shown in Fig. 3N illustrates.

[0052] In the example of Fig. 3N displays computer system 101 in response to gesture 332 recognition. Fig. 3M Content Window 334, which is a visual representation of the application 330 running on laptop 304, is displayed in the three-dimensional environment 302. In one or more examples, Content Window 334 may include a graphical representation of the content that was displayed on laptop 304 while application 330 was running (as in Fig. 3M illustrated). Additionally or alternatively, content window 334 is interactive, so that the user can interact with content window 334 in the same way as they could interact with application 330 running on laptop 306. In one or more examples, computer system 101 runs its own copy of application 330 using the file(s) transferred from laptop 304 to computer system 101, and the user of the computer system can operate application 330 using computer system 101 (displayed within the three-dimensional environment 302) in essentially the same way as they would operate application 330 if it were running on laptop 304. In one or more examples, the examples of Fig. 3A to 3N are to be understood as examples and should not be regarded as a limitation of the revelation.

[0053] In some examples, Method 400 takes place on a computer system that communicates with one or more displays and one or more input devices. In one or more examples, the computer system is or includes an electronic device, such as a mobile device (e.g., a tablet, a smartphone, a media player, or a portable device) or a computer. In one or more examples, the display-generating component is a display integrated into the electronic device (optionally a touchscreen display), an external display such as a monitor, a projector, a television, or a hardware component (optionally integrated or external) for projecting a user interface or causing a user interface to be displayed to one or more users.In one or more examples, the one or more input devices include an electronic device or component capable of receiving user input (e.g., capturing or recognizing user input) and transmitting information associated with the user input to the electronic device. Examples of input devices include an image sensor (e.g., a camera), a location sensor, a hand tracking sensor, an eye tracking sensor, a motion sensor (e.g., a hand motion sensor), an orientation sensor, a microphone (and / or other audio sensors), a touchscreen (optionally integrated or external), a remote control (e.g., external), another mobile device (e.g., separate from the electronic device), a handheld device (e.g., external), and / or a controller.

[0054] In one or more examples, (404) the computer system, while displaying a three-dimensional environment that includes a first object (402), detects, via one or more displays, a first gesture made by a user of the computer system and directed at the first object. In one or more examples, the three-dimensional environment is generated, displayed, or otherwise made visible by the first computer system. For example, the three-dimensional environment is an extended reality (XR) environment, such as a virtual reality (VR) environment, a mixed reality (MR) environment, or an augmented reality (AR) environment. In one or more examples, the three-dimensional environment includes at least partially or completely the physical environment of the user of the computer system.For example, the computer system optionally includes one or more outward-facing cameras and / or passive optical components (e.g., lenses, discs or sheets of transparent material, and / or mirrors) configured to allow the user to view the physical environment and / or a representation of the physical environment (e.g., images and / or other visual reproductions of the physical environment). In one or more examples, the three-dimensional environment includes one or more virtual objects and / or representations of objects in a user's physical environment. Examples of objects include real and physical documents, pictures, and furniture that would otherwise exist in a physical environment.In one or more examples, the first gesture is performed with the user's hand to indicate to the computer system that the user wishes to collect information associated with the first object. In one or more examples, the gesture is predefined to be visibly distinct from other gestures used to perform other computational operations, so that when the device detects that the gesture is being performed, the computer system initiates the collection of information about the object toward which the gesture is directed. In one or more examples, a gesture is considered to be directed at an object if the part of the user used to perform the gesture (e.g., the user's hand) points toward the object and / or partially obscures the object (from the user's perspective), as described above.

[0055] In one or more examples, (406) the computer system, in response to recognizing the first gesture directed at the first object, collects information associated with the first object. In one or more examples, the computer system captures an image of the first object (as part of collecting information associated with the first object). In one or more examples, as part of collecting information about the first object, the computer system collects text data (e.g., text written on the object). Additionally or alternatively, collecting information about the first object includes querying one or more databases containing the collected image and / or text data to determine whether the database contains information relevant to the object.If a match is found, the matching information can be included as part of the collected information linked to the first object.

[0056] In one or more examples, (410) the computer system recognizes a second gesture performed by the user, directed at the first electronic device, while, via the display generation component, it represents the three-dimensional environment that includes a first electronic device (408), wherein the first electronic device is communicatively coupled to the computer system. Examples of the first electronic device include, but are not limited to: a computing device (e.g., a laptop and / or desktop computer), a music player, a television or other media device, a head-mounted computing system, and / or a smart speaker.

[0057] In one or more examples, in response to the detection of the second gesture directed at the first electronic device, the information collected and associated with the first object is transferred to the first electronic device. In one or more examples, the second gesture is visually distinguishable from the first gesture described above, allowing the computer system to differentiate between the first and second gestures and thus know when to collect information and when to transmit that information. In one or more examples, the computer system takes no action in response to the detection of the second gesture if it has no stored information associated with objects in the three-dimensional environment, since no information has been collected to transmit.In one or more examples, transferring the stored information associated with the first object to the first electronic device involves establishing a communication link with the electronic device (e.g., using a wireless or wired communication link such as Bluetooth, near-field radio frequency (RF) protocols, Universal Serial Bus (USB), or another known communication link). In one or more examples, the computer system establishes the communication link to the first electronic device only after verifying that the user of the computer system is authorized to transmit information to the electronic device.

[0058] In one or more examples, recognizing the first gesture directed at the first object involves recognizing the user's hand with one or more fingers extended, followed by the movement of that one or more fingers coming together. In one or more examples, the computer system does not recognize the first gesture until it has recognized both parts of the gesture (e.g., the extended hand and the bringing together of the fingers). In one or more examples, upon recognizing that both parts of the gesture have been performed, the computer system begins collecting information associated with the first object, as described above.

[0059] In one or more examples, the information associated with the first object is collected while the computer system detects that the user is performing the first gesture. In one or more examples, the computer device only collects the information while the computing device detects that the first gesture is being performed. In one or more examples, the first gesture is still being performed while the device detects that the fingers are still held together. In one or more examples, the information associated with the first object is collected while the computer system continues to detect that the first gesture is being performed. If the computer system does not detect that the first gesture is being performed while the information is being collected, the computer system optionally stops collecting the information and terminates the information gathering process.In one or more examples, the computer system no longer recognizes whether the first gesture is performed once it recognizes that the computer system has completed the process of gathering the information.

[0060] In one or more examples, the computer system displays an initial visual indicator within the three-dimensional environment while gathering information associated with the first object, indicating the progress of this information gathering. In one or more examples, the visual indicator is configured to provide the user with a visual indication of the information gathering progress (related to the first object), allowing the user to determine how long to maintain the initial gesture. In one or more examples, the visual indicator includes an animation sequence configured to demonstrate the information gathering progress.In one or more examples, the animation sequence includes a progress bar (or circle) that gradually fills as information gathering progresses, and the animation sequence optionally ends when the progress bar is completely filled, indicating that information gathering is finished. In one or more examples, the visual indicator is no longer displayed by the computer system if the information gathering process is interrupted or otherwise terminated without completion. In one or more examples, the visual indicator, and in particular the animation sequence, also provides a visual indication of when information gathering is complete.For example, the visual indicator includes a checkmark or other confirmatory visual signal configured to warn the user that information gathering is complete (and also tells the user when they can stop performing the initial gesture). In one or more examples, the visual indicator is accompanied by an audio indicator that specifies when the information gathering process is finished.

[0061] In one or more examples, the computer system detects the termination of the first gesture while collecting information associated with the first object, but before the collection process is complete, and terminates the collection of information associated with the first object in response to this detection. In one or more examples, while the computer system is collecting information associated with the first object, the user can signal the computer system to terminate the information collection process (e.g., to stop collecting information associated with the first object) by terminating the first gesture before the information collection process is complete.In the example of the first gesture, which involves bringing one or more fingers together, the device, for instance, terminates the data collection process and, in response to determining that the user's fingers are no longer pressed together (e.g., the first gesture is no longer being performed), refrains from storing the collected information in memory allocated to the computer system. Alternatively, the computer system stores the collected information in memory allocated to the computer system before the computer system detects the termination of the first gesture. In one or more examples.

[0062] In one or more examples, the recognition of the second gesture includes recognizing the user's hand, with one or more fingers extended, directed toward the first electronic device. In one or more examples, the second gesture resembles the first part of the first gesture (e.g., the extended fingers), but unlike the first gesture, where the user brings their fingers together, the second gesture only includes the user's fingers being extended and directed toward the first electronic device. In one or more examples directed toward the first electronic device (in the context of the second gesture), one or more similarities are shared with the first gesture's orientation toward the first object.Thus, the computer system determines that the second gesture is directed at the electronic device, based on the position and orientation of the hand in the three-dimensional environment, when the computer system determines that the hand is performing the second gesture.

[0063] In one or more examples, the computer system refrains from transmitting the information collected and associated with the first object if it determines that the user is performing the second gesture, but also determines that the second gesture is not directed at an electronic device (for example, because an electronic device is not within the user's field of vision). In one or more examples, in response to the detection of the second gesture, but also in response to the detection that the second gesture is not directed at an electronic device, the computer system displays a visual indicator (such as an X mark) indicating to the user that no information transmission has occurred in response to the user performing the second gesture.

[0064] In one or more examples, the stored information associated with the first object is transferred to the first electronic device while the computer system detects that the user is performing the second gesture. In one or more examples, and similarly to the first gesture example, the computer system, in response to determining that the user has completed the second gesture, stops transferring the collected information associated with the first object to the first electronic device while the data transfer is in progress. For example, if the second gesture involves detecting one or more of the user's extended fingers, the computer system determines that the second gesture has ended when it determines that the extended fingers (which initiated the second gesture) are no longer extended.

[0065] In one or more examples, while the stored information associated with the first object is being transferred to the first electronic device, the computer system displays a second visual indicator within the three-dimensional environment, indicating the progress of the transfer of the information associated with the first object to the first electronic device. In one or more examples, the second visual indicator shares one or more properties with the first visual indicator described above.

[0066] In one or more examples, the computer system detects the end of the second gesture while the stored information associated with the first object is being transferred to the first electronic device, but before the transfer of the information associated with the first object to the first device is complete, and terminates the transfer of the information associated with the first object to the first electronic device in response to the detection of the end of the second gesture. In one or more examples, the computer system terminates the transfer of the collected information in response to the detection of the end of the second gesture, even if the transfer of the information is not yet complete.Alternatively, the computer system completes the transmission of the collected information before it terminates the transmission, even if the detection of the second gesture's completion occurs before the information transfer is finished. In one or more examples, in response to determining that the second gesture has been completed before the transfer is finished, the computer system displays a visual indicator configured to warn the user that the transfer has been terminated without actually being completed (such as an X-mark similar to the one described above). In one or more examples, the visual indicator may also be accompanied by an audio indicator configured to warn the user that the transfer is complete.

[0067] In one or more examples, the collected information associated with the first object includes a visual scan of the first object. In one or more examples, the computer system collects image data associated with the first object in response to the detection of the first gesture and the fact that the first gesture is directed at the first object. In one or more examples, the image data is captured by one or more cameras connected to the computer system. In one or more examples, the computer system determines the extent and boundaries of the first object (using the one or more cameras) and generates an image of the first object within the determined extent and boundaries (such that the image data covers an area that is within or even slightly outside the determined extent and boundaries of the first object).In one or more examples, the image data is a still image of the first object. Alternatively, the image data includes video data of the first object. In one or more examples, after determining the dimensions and boundaries of the first object, the computer system generates image data of a predefined area that surrounds and includes the first object. In one or more examples, the resolution and / or other visual properties of the image data are based on a determination of the identity or nature of the first object. For example, in response to the determination that the first object is a document, the computer system generates image data of the first object with a resolution sufficient for the text on the document to be read by the user of the computer system and / or the user of the first electronic device.In one or more examples, the image data captured by the computer system is similar in its visual quality / properties to the type of image data that would be captured by a scanner if the first object were placed in a scanner and scanned. In one or more examples, the user can specify a predefined visual quality level at which the image data is captured (for example, by providing setting information in a settings menu).

[0068] In one or more examples, in response to the completion of the collection of information associated with the first object, the computer system compares the collected information associated with the first object with one or more entries in one or more media databases to determine whether the collected information matches the one or more entries.

[0069] In one or more examples, the computer system, based on a determination that the collected information associated with the first object matches one or more entries in the one or more media databases, adds information about the one or more matching entries to the collected information associated with the first object. For example, if the first object includes text that was scanned as part of the collected information associated with the first object, the scanned text is compared against one or more databases to determine whether one or more entries in the database contain information that is relevant to or related to the scanned text. The one or more media databases include databases containing music information (e.g.,The system collects information such as artist, album title, song title, film information, television show information, podcast information, and other media compilations. In the example where the collected information associated with the first object includes scanned text, the scanned text is compared against the media databases to determine if there is a relevant song, film, and / or television show that matches the text. If a particular song, film, and / or show matches the scanned text, this information (e.g., information about the match) is added to the collected information associated with the first object and is thus available for transmission to the first electronic device (in response to the computer system recognizing the second gesture directed at the first electronic device, as described above).

[0070] In one or more examples, the computer system, in response to the recognition of the second gesture, determines an identity of the first electronic device. In one or more examples, according to the determined identity of the first electronic device, which is a first type of electronic device, the computer system transmits a first section of the stored information associated with the first object to the first electronic device. In one or more examples, according to the determined identity of the first electronic device, which is a second type of electronic device different from the first type, a second section of the stored information, different from the first section and associated with the first object, is transmitted to the first electronic device.In one or more examples, the computer system adapts the information transmitted to the first electronic device based on the type of electronic device that is the first electronic device. For example, if it is determined that the first electronic device is a music player and / or a smart speaker, the computer system transmits a portion of the collected information that would be relevant to a music player and / or a smart speaker, such as any song title or artist name associated with the first device. In response to receiving the portion of the collected information relating to music content, the music player and / or the smart speaker can play a song or other music content associated with the transmitted information.If the first electronic device is determined to be a video player and / or a smart TV, the computer system transmits the portion(s) of the collected information associated with the first object and relating to links between the first object and video content (such as matching TV shows and / or movies) in a similar manner. In one or more examples, and in the example where the first electronic device is a video player and / or smart TV, the visual scan itself is not transmitted to the first electronic device because it is not relevant to the operation of the video player / smart TV, even though the collected information may include a visual scan of the first object (as described above).

[0071] In one or more examples, the three-dimensional environment includes a second object. In one or more examples, while displaying the three-dimensional environment, after collecting information associated with the first object and before recognizing the second gesture, the computer system detects a third gesture performed by the user of the computer system directed at the second object and, in response to the detection of this third gesture, collects information associated with the second object. In one or more examples, and in the case where the user performs the first gesture multiple times directed at multiple objects, the computer system stores the information associated with each instance when the computer system detects that the first gesture is performed separately (e.g., one entry is stored for each detected occurrence of the first gesture).Additionally or alternatively, and in the event that the user has directed multiple initial gestures at the same object, the computer system collects the information associated with that particular object in a single memory entry. Thus, in one or more instances relating to a single object, there can be multiple instances of collected and associated information, and / or multiple separate instances of collected information can refer to the same object.

[0072] In one or more examples, in response to the collection of information associated with the second object, the computer system displays a user interface with stored information in three-dimensional space, wherein the user interface with stored information includes a first selectable option linked to the information associated with the first object, and wherein the user interface with stored information includes a second selectable option linked to the information associated with the second object.In one or more examples, and in the case that the computer system has gathered information about multiple objects (by recognizing multiple instances of the first gesture performed), the computer system displays a stored information user interface that lists each instance of the gathered information available for transfer to an electronic device (in response to the computer system recognizing an instance of the second gesture directed at the electronic device). In one or more examples, each entry in the stored information user interface is a selectable option.

[0073] In one or more examples, the computer system recognizes an initial input at the first selectable option that is linked to the information associated with the first user interface object containing the stored information.

[0074] In one or more examples, the computer system, in response to the detection of the first input and in response to the detection of the second user gesture directed at the first electronic device, transfers the stored information associated with the first object to the electronic device. In one or more examples, in response to the detection that a selectable user interface option with stored information has been selected, the computer system ensures that the information associated with the input is transferred to an electronic device the next time the computer system detects the execution of the gesture directed at the electronic device.Thus, in one or more examples, in response to the detection of a second gesture directed at the first electronic device, the computer system transmits the collected information associated with the selectable option chosen on the user interface with the stored information.

[0075] In one or more examples, the computer system, in response to the detection of the second gesture directed at the first electronic device, without having detected the first input, transmits the stored information associated with the second object to the first electronic device. In one or more examples, if multiple sets of information are stored on the computer system (for example, in response to the detection of the first gesture performed multiple times), the computer system, in response to the detection of the second gesture, transmits the information associated with the object that was collected when the first gesture was last performed. The first gestures thus operate according to the LIFO (Last In - First Out) principle, so that the most recently stored information is the first information transmitted in response to the detection of the second gesture.In one or more examples, in response to the detection of a selection of a selectable option from the user interface with the stored information, the computer system stops LIFO operation and instead transfers the information associated with the selectable option that was detected as having been selected from the user interface with the stored information.

[0076] In one or more examples, the first electronic device is the computer system, and transferring the stored information associated with the first object to the first electronic device involves accessing the collected information associated with the first object in a memory of the computer system.In one or more examples, the computer system is configured to recognize, using one or more cameras that are part of the computer system and / or communicatively coupled to it, that the second gesture is directed at the computer system itself. In one or more examples, and as explained below, in response to recognizing that the second gesture is directed at the computer system itself, the computer system accesses the memory in which the collected information associated with the first object is stored, thus transferring the collected information associated with the first object to itself.

[0077] In one or more examples, the computer system displays a representation of the first object in the three-dimensional environment in response to the detection of the second gesture directed at the computer system. In one or more examples, the computer system displays a visual image or other graphical representation of the first object in the three-dimensional environment in response to the detection of the second gesture directed at the computer system. For example, if the first object is a document and the collected information associated with the first object includes a scan of the first object, the computer system displays the scanned image of the document in a graphical user interface and / or content window displayed in the three-dimensional environment in response to the detection of the second gesture directed at the computer system.In the example of collected information, which includes songs, videos and / or media contacts relevant to the first object, the computer system, in response to the recognition of the second gesture directed at the computer system, displays a media player and plays the media (e.g., song, film, television show) associated with the first object (this association is recorded in the collected information linked to the first object).

[0078] In one or more examples, the first object is a computing device. In one or more examples, the first object is a computing device that is visible in the displayed three-dimensional environment. For example, in the examples described above, the first object is a laptop or other computing device (tablet, desktop computer) located in the physical space displayed in the three-dimensional environment. In one or more examples, and as described in more detail below, the computer system recognizes that the first object (e.g., the object toward which the first gesture is directed) is a computing device and, in response, accesses information from the computing device itself, which it uses to display a visual representation in the three-dimensional environment.

[0079] In one or more examples, the information gathered, according to the first object being a computing device, and where the computing device is running a first software application, includes information associated with the application running on the computing device. In one or more examples, the application running on the computing device includes a content creation application, a rendering application, a photo application, a video application, a music application, and / or a media application. In one or more examples, and in one example where the application is a media application (such as a video application or a music application), the information gathered includes information about the application currently running on the computing device.In one or more examples, in response to the determination that the first object is a computing device, the computer system transmits a request to the computing device to retrieve information about the application running on the computing device, including but not limited to all files used by the application (e.g., photo files and / or other media files), operations performed on the files used by the application, settings relating to the application, user information relating to the application (provided that the user of the computer system has the appropriate permission to access the information), and any other information relating to the application currently running on the computing device.

[0080] It is understood that Process 400 is an example and that more, fewer, or different operations can be performed in the same or a different sequence. Additionally, the operations described above in Process 500 can optionally be implemented by using one or more functional modules in an information processing device, such as general-purpose processors (e.g., with reference to...). Fig. 2 described) or application-specific chips, and / or by other components of Fig. 2 will be executed.

[0081] Some examples of the disclosure relate to an electronic device comprising: one or more processors; a working memory; and one or more programs stored in the working memory and configured to be executed by the one or more processors, the one or more programs containing instructions for carrying out one of the above-mentioned procedures.

[0082] Some examples of the disclosure relate to a non-transitory, computer-readable storage medium that stores one or more programs, wherein the one or more programs comprise instructions which, when executed by one or more processors of an electronic device, cause the electronic device to perform one of the above-mentioned procedures.

[0083] Some examples of the disclosure relate to an electronic device comprising one or more processors, memory, and means for carrying out one of the foregoing methods.

[0084] Some examples of the disclosure relate to an information processing device for use in an electronic device, the information processing device comprising means for carrying out one of the methods disclosed above.

[0085] The foregoing description was provided for the purpose of explanation with reference to specific examples. However, the preceding illustrative discussions are neither intended to be exhaustive nor to limit revelation to the exact forms revealed. In light of the foregoing teachings, many modifications and variations are possible. The examples were chosen and described to best explain the principles of revelation and their practical application, thus enabling other professionals to make the best possible use of revelation and the various examples described, with different modifications, as appropriate for their intended particular use. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 700,667

[0001] US 19 / 331,960

[0001]

Claims

[1] Procedure, encompassing: on a computer system communicating with one or more displays and one or more input devices: During the display, across one or more displays, of a three-dimensional environment that includes a first object, recognition of a first gesture performed by a user of the computer system and directed at the first object: In response to recognizing the initial gesture directed at the first object, information is received that is linked to the first object. during the display, via one or more displays, the three-dimensional environment including a first electronic device, wherein the first electronic device is communicatively coupled with the computer system, recognizes a second gesture performed by the user, directed at the first electronic device; and In response to the recognition of the second gesture directed at the first electronic device, the information obtained from the first object is transferred to the first electronic device. [2] Method according to claim 1, wherein the detection of the first gesture directed towards the first object comprises the detection of a user's hand with one or more fingers of the hand extended, followed by a movement of the one or more fingers coming together. [3] Method according to one of claims 1 to 2, wherein the information associated with the first object is obtained while the computer system recognizes that the user performs the first gesture. [4] Method according to any one of claims 1 to 3, wherein the method further comprises: During the process of obtaining the information associated with the first object, a first visual indicator is displayed within the three-dimensional environment, indicating progress in obtaining the information associated with the first object. [5] Method according to any one of claims 1 to 4, wherein the method further comprises: during the acquisition of the information associated with the first object, but before the completion of the acquisition of the information associated with the first object, recognition of the completion of the first gesture; and as a reaction to the recognition of the termination of the first gesture, the cessation of the receipt of information associated with the first object. [6] Method according to any one of claims 1 to 5, wherein the recognition of the second gesture comprises the recognition of a user's hand directed towards the first electronic device with one or more outstretched fingers. [7] Method according to any one of claims 1 to 6, wherein the received information associated with the first object is received by the first electronic device according to the computer system that recognizes that the user is performing the second gesture. [8] Method according to any one of claims 1 to 7, wherein the method further comprises: While the information associated with the first object is being transferred to the first electronic device, a second visual indicator is displayed within the three-dimensional environment, indicating the progress of the transfer of the information associated with the first object to the first electronic device. [9] Method according to any one of claims 1 to 8, wherein the method further comprises: While the information associated with the first object is being transferred to the first electronic device, but before the transfer of the information associated with the first object to the first electronic device is completed, the termination of the second gesture is detected; and in response to the detection of the termination of the second gesture, the transfer of the information associated with the first object to the first electronic device is terminated. [10] Method according to any one of claims 1 to 9, wherein the method further comprises: in response to the completion of the retrieval of the information associated with the first object, comparing the retrieved information associated with the first object with one or more entries in one or more media databases to determine whether the retrieved information matches the one or more entries; and According to a stipulation that the received information associated with the first object matches one or more entries in the one or more media databases, add information about the one or more matching entries to the received information associated with the first object. [11] Method according to any one of claims 1 to 10, wherein the method further comprises: as a reaction to the recognition of the second gesture, determining an identity of the first electronic device; According to the determined identity of the first electronic device, which is a first type of electronic device, a first section of the information associated with the first object is transmitted to the first electronic device; and According to the determined identity of the first electronic device, which is a second type of electronic device that differs from the first type, a second section of information, which differs from the first section and is linked to the first object, is transmitted to the first electronic device. [12] Method according to any one of claims 1 to 11, wherein the three-dimensional environment includes a second object and wherein the method further comprises: During the rendering of the three-dimensional environment, after receiving information associated with the first object, and before recognizing the second gesture, recognition of a third gesture performed by the user of the computer system and directed at the second object; and In response to the recognition of the third gesture directed at the second object, information is received that is linked to the second object. [13] Method according to any one of claims 1 to 12, wherein the first electronic device is the computer system, and wherein the transfer of the information associated with the first object to the first electronic device comprises accessing the received information associated with the first object in a memory of the computer system. [14] Computer system that communicates with one or more displays and one or more input devices, the computer system comprising: one or more processors; a storage facility; and one or more programs, wherein the one or more programs are stored in memory and configured to be executed by the one or more processors, wherein the one or more programs include instructions for carrying out the method according to any one of claims 1 to 13. [15] Non-transitory, computer-readable storage medium that stores one or more programs comprising instructions which, when executed by one or more processors of an electronic device, cause the electronic device to perform a method according to any one of claims 1 to 13.

Images