Method and apparatus for presenting synthetic reality user interface

Abstract

The invention relates to a method and device for presenting a synthetic reality user interface. In various embodiments, a method of presenting a Synthetic Reality (SR) user interface is disclosed, including moving data items in the SR user interface, regrouping the data items in the SR user interface, selecting a group of data items in the SR user interface, and playing an audio file represented by the data items in the SR user interface.

Description

[0001] Division Statement

[0002] This application is a divisional application of the Chinese invention patent application with the application date of May 29, 2019, the invention name of which is “Method and device for presenting a synthetic reality user interface” and the application number is: 201980036736.5. Technical Field

[0003] The present disclosure relates generally to synthetic reality user interfaces, and in particular, to systems, methods, and apparatus for presenting a synthetic reality user interface that includes a group representation of a group of files. Background Art

[0004] A physical set refers to a world that an individual can perceive and / or interact with without the aid of an electronic system. A physical set (e.g., a physical forest) includes physical elements (e.g., physical trees, physical structures, and physical animals). An individual can directly interact with and / or perceive a physical set, such as by touch, sight, smell, hearing, and taste.

[0005] In contrast, a synthetic reality (SR) set refers to a set that is fully or partially computer-created that an individual can perceive and / or that the individual can interact with via an electronic system. In SR, a subset of the individual's movements is monitored, and in response thereto, one or more properties of one or more virtual objects in the SR set are changed in a manner that complies with one or more laws of physics. For example, the SR system may detect that the individual has taken a few steps forward, and in response thereto, adjust the graphics and audio presented to the individual in a manner similar to how such scenes and sounds would change in a physical environment. Modifications to one or more properties of one or more virtual objects in the SR set may also be made in response to a representation of movement (e.g., an audio command).

[0006] The individual may interact with and / or perceive SR objects using any of his senses, including touch, smell, sight, taste, and sound. For example, the individual may interact with and / or perceive auditory objects that create a multi-dimensional (e.g., three-dimensional) or spatial auditory scenery and / or implement auditory transparency. The multi-dimensional or spatial auditory scenery provides the individual with the perception of discrete auditory sources in a multi-dimensional space. Auditory transparency selectively combines sounds from the physical scenery with or without computer-created audio. In some SR scenery, the individual may interact with and / or perceive only auditory objects.

[0007] An example of SR is virtual reality (VR). A VR set refers to a simulated set designed to include only computer-created sensory input for at least one sense. The VR set includes a plurality of virtual objects that an individual can interact with and / or perceive. The individual can interact with and / or perceive virtual objects in the VR set by simulating a subset of the individual's actions within the computer-created set and / or by simulating the individual or his / her presence within the computer-created set.

[0008] Another example of SR is mixed reality (MR). An MR set refers to a simulated set designed to integrate computer-created sensory input (e.g., virtual objects) with sensory input from a physical set, or representation thereof. On the reality spectrum, a mixed reality set is between VR sets at one end and fully physical sets at the other end, and does not include these sets.

[0009] In some MR sets, computer-created sensory input can adapt to changes in sensory input from the physical set. In addition, some electronic systems used to render MR sets can monitor orientation and / or position relative to the physical set to enable virtual objects to interact with real objects (i.e., physical elements from the physical set or representations thereof). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0010] An example of mixed reality is augmented reality (AR). AR scenery refers to a simulated scenery in which at least one virtual object is superimposed on a physical scenery or a representation thereof. For example, an electronic system may have an opaque display and at least one imaging sensor, the imaging sensor being used to capture images or videos of a physical scenery, which are representations of the physical scenery. The system combines the image or video with the virtual object and displays the combination on the opaque display. An individual uses the system to indirectly view the physical scenery via an image or video of the physical scenery and observes the virtual object superimposed on the physical scenery. When the system uses one or more image sensors to capture images of the physical scenery and uses those images to present the AR scenery on an opaque display, the displayed image is referred to as video transmission. Alternatively, an electronic system for displaying an AR scenery may have a transparent or translucent display through which an individual can directly view the physical scenery. The system may display virtual objects on a transparent or translucent display so that an individual uses the system to observe virtual objects superimposed on the physical scenery. As another example, the system may include a projection system that projects virtual objects into a physical scenery. Virtual objects may be projected, for example, on a physical surface or as a hologram, such that an individual using the system observes the virtual objects superimposed on the physical scenery.

[0011] An augmented reality set may also refer to a simulated set in which a representation of a physical set is altered by computer-created sensory information. For example, a portion of the representation of a physical set may be graphically altered (e.g., enlarged) such that the altered portion may still represent one or more originally captured images but is not a faithfully reproduced version. For another example, when providing video pass-through, the system may alter at least one of the sensor images to impose a particular viewpoint that is different from the viewpoint captured by one or more image sensors. For another example, the representation of a physical set may be altered by graphically blurring or eliminating portions thereof.

[0012] Another example of mixed reality is augmented virtuality (AV). An AV set refers to a simulated set in which a computer-created set or a virtual set incorporates at least one sensory input from a physical set. The one or more sensory inputs from the physical set may be a representation of at least one feature of the physical set. For example, a virtual object may present the color of a physical element captured by one or more imaging sensors. As another example, a virtual object may present features consistent with actual weather conditions in a physical set, as identified via weather-related imaging sensors and / or online weather data. In another example, an augmented reality forest may have virtual trees and structures, but animals may have features accurately reproduced from images taken of physical animals.

[0013] Many electronic systems enable individuals to interact with and / or perceive various SR scenery. One example includes a head-mounted system. The head-mounted system may have an opaque display and one or more speakers. Alternatively, the head-mounted system may be designed to receive an external display (e.g., a smart phone). The head-mounted system may have one or more imaging sensors and / or microphones for taking images / video of the physical scenery and / or capturing audio of the physical scenery, respectively. The head-mounted system may also have a transparent or translucent display. The transparent or translucent display may be combined with a substrate through which light representing the image is directed to the individual's eyes. The display may be combined with LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The substrate that transmits light may be an optical waveguide, an optical combiner, an optical reflector, a holographic substrate, or any combination of these substrates. In one embodiment, the transparent or translucent display may be selectively switched between an opaque state and a transparent or translucent state. As another example, the electronic system may be a projection-based system. A projection-based system may project an image onto the retina of an individual using retinal projection. Alternatively, the projection system may also project the virtual object into a physical setting (e.g., onto a physical surface or as a hologram). Other examples of SR systems include a head-up display, a car windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics, headphones or earbuds, a speaker arrangement, an input mechanism (e.g., a controller with or without tactile feedback), a tablet, a smartphone, and a desktop or laptop computer.

[0014] Navigating between a large number of data items (such as audio files, video files, document files, or web pages) to locate a specific data item can be cumbersome. For example, locating a song to play in a large music library can be a tedious task. In addition, transitioning between songs or otherwise playing multiple songs simultaneously can produce sound inconsistencies. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be had with reference to aspects of some illustrative embodiments, some of which are illustrated in the accompanying drawings.

[0016] Figure 1A is a block diagram of an exemplary operating architecture according to some implementations.

[0017] Figure 1B is a block diagram of an exemplary operating architecture according to some implementations.

[0018] Figure 2 is a block diagram of an example controller according to some implementations.

[0019] Figure 3is a block diagram of an exemplary head mounted device (HMD) according to some implementations.

[0020] Figure 4 is a third person view of an SR volumetric environment according to some embodiments.

[0021] Figures 5A to 5P A first person view of an SR volumetric environment including an SR group representation of a group of document files is shown according to some embodiments.

[0022] 6A to 6H A first person view of an SR volumetric environment including an SR group representation of a group of audio files is shown according to some embodiments.

[0023] FIG. 7A to FIG. 7C An SR volumetric environment is shown in which an SR group representation is selected by pulling the SR group representation towards a user according to some embodiments.

[0024] FIG. 8A to FIG. 8E An SR volume environment is shown in which an SR group representation is selected by pulling the SR group representation apart, according to some embodiments.

[0025] FIG. 9A to FIG. 9B An SR volumetric environment is shown in which an SR group representation is selected by gazing at the SR group representation according to some embodiments.

[0026] FIG. 10A to FIG. 10B An SR volume environment is shown according to some embodiments, wherein an SR group representation is selected by moving towards the SR group representation.

[0027] FIG. 11A to FIG. 11B An SR volume environment is shown in which two audio files are played simultaneously according to some implementations.

[0028] FIG. 12A to FIG. 12C An SR volume environment is shown in which a first audio file cross-fades to a second audio file in accordance with some implementations.

[0029] Fig.13 A flowchart representation of a method of moving an object in a SR user interface according to some embodiments is shown.

[0030] Fig.14 is a flowchart representation of a method of reorganizing data items in a SR user interface according to some embodiments.

[0031] Fig.15 is a flowchart representation of a method of selecting a group of data items in a SR user interface according to some embodiments.

[0032] Fig.16A flowchart representation of a method of playing two audio files according to some implementations is shown.

[0033] Fig.17 is a flowchart representation of a method of crossfading between two audio files according to some implementations.

[0034] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0035] Various embodiments disclosed herein include devices, systems, and methods for moving objects in an SR user interface. In various embodiments, the method is performed by a device including one or more processors, a non-volatile memory, and a display. The method includes identifying a plurality of data items, each of the plurality of data items having a first metadata field. The method includes displaying an SR volumetric environment on a display, the SR volumetric environment including a plurality of first SR group representations corresponding to a plurality of first group data items, each of the plurality of first group data items including a data item having a corresponding first metadata field value among a plurality of first metadata field values ​​in the plurality of data items. The method includes detecting a first user input directed to a particular first SR group representation in the plurality of first SR group representations. The method includes, in response to detecting the first user input, moving a particular first SR group representation in the plurality of first SR group representations relative to at least another first SR group representation in the plurality of first SR group representations in the SR volumetric environment.

[0036] Various embodiments disclosed herein include devices, systems, and methods for regrouping data items in an SR user interface. In various embodiments, the method is performed by a device including one or more processors, a non-volatile memory, and a display. The method includes identifying a plurality of data items, each of the plurality of data items having a first metadata field and a second metadata field. The method includes displaying an SR volume environment on a display, the SR volume environment including a plurality of first SR group representations corresponding to a plurality of first group data items, each of the plurality of first group data items respectively including a data item having a corresponding first metadata field value among a plurality of first metadata field values ​​of the first metadata field in the plurality of data items. The method includes detecting a first user input indicating a second metadata field. The method includes, in response to detecting the first user input, replacing the plurality of first SR group representations with a plurality of second SR group representations corresponding to a plurality of second group data items, each of the plurality of second group data items respectively including a data item having a corresponding second metadata field value among a plurality of second metadata field values ​​of the second metadata field in the plurality of data items.

[0037] Various embodiments disclosed herein include devices, systems, and methods for selecting a group of data items in an SR user interface. In various embodiments, the method is performed by a device including one or more processors, a non-volatile memory, and a display. The method includes identifying a plurality of data items, each of the plurality of data items having a first metadata field and a second metadata field. The method includes displaying an SR volumetric environment on a display, the SR volumetric environment including a plurality of first SR group representations corresponding to a plurality of first group data items, each of the plurality of first group data items respectively including a data item in the plurality of data items having a first metadata field, the first metadata field including a corresponding first metadata field value in a plurality of first metadata field values. The method includes detecting a first user input directed to a particular first SR group representation in a plurality of first SR group representations corresponding to a particular first group data item in the first group data items, the particular first group data item in the first group data items including a data item in the plurality of data items having a first metadata field, the first metadata field including a particular first metadata field value in the plurality of first metadata field values. The method includes, in response to detecting a first user input, replacing a specific first SR group representation among a plurality of first SR group representations with a plurality of second SR group representations corresponding to a plurality of second group data items, respectively, each of the plurality of second group data items including a data item among the plurality of data items having a first metadata field and a second metadata field, the first metadata field including a specific first metadata field value among the plurality of first metadata field values, and the second metadata field including a corresponding second metadata field value among the plurality of second metadata field values.

[0038] Various embodiments disclosed herein include devices, systems, and methods for playing two audio files. In various embodiments, the method is performed by a device including one or more processors, a non-volatile memory, a directional speaker system, and a display. The method includes displaying a plurality of SR objects at corresponding positions in an SR volume space on the display, each of the plurality of SR objects being associated with a corresponding metadata field value of a metadata field of at least one of a plurality of audio files. The method includes determining a first distance between a user position in the SR volume space and a first position in the SR volume space of a first SR object of the plurality of SR objects, wherein the first SR object is associated with a first metadata field value. The method includes determining a second distance between a user position in the SR volume space and a second position in the SR volume space of a second SR object of the plurality of SR objects, wherein the second SR object is associated with a second metadata field value. The method includes selecting a first audio file of the plurality of audio files having a metadata field including a first metadata field value. The method includes selecting a second audio file of the plurality of audio files having a metadata field including a second metadata field value based on the first audio file. The method includes playing a first audio file at a first volume in a direction from a first position based on a first distance via a directional speaker system, and simultaneously playing a second audio file at a second volume in a direction from a second position based on a second distance via the directional speaker system.

[0039] According to some embodiments, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by the one or more processors. One or more programs include instructions for executing or causing the execution of any method described herein. According to some embodiments, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to execute or cause the execution of any of the methods described herein. According to some embodiments, a device includes: one or more processors, non-volatile memory, and a device for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0040] Many details are described in order to provide a thorough understanding of the example embodiments shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the example embodiments described herein.

[0041] As described above, navigating among a large number of data items (such as audio files, video files, document files, or web pages) to locate a particular data item can be cumbersome. Described herein is an SR user interface that can be used to open a data item from a plurality of data items. In various embodiments, presentation of the SR user interface includes display of an SR volumetric environment that includes SR group representations at various locations around a user in the SR volumetric environment. Each SR group representation corresponds to a respective group of data items, wherein each data item in the group shares attribute values. Thus, a large number of files are represented in the SR user interface as a smaller number of SR group representations.

[0042] In various embodiments, the SR group representations may be rearranged and organized by a user within the SR volume environment. In various embodiments, the data items may be reordered and / or regrouped such that a corresponding SR group representation associated with a corresponding value of a first attribute is replaced with a corresponding SR group representation associated with a corresponding value of a second attribute. In various embodiments, an SR group representation associated with a particular value of a first attribute may be selected such that it may be replaced with (1) a representation of the data items having the particular value of the first attribute or (2) a corresponding SR group representation of a group of data items having the particular value of the first attribute but corresponding different values ​​of the second attribute.

[0043] Figure 1A 1 is a block diagram of an exemplary operational architecture 100A according to some embodiments. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein. To this end, as a non-limiting example, the operational architecture 100A includes an electronic device 120A.

[0044] In some embodiments, the electronic device 120A is configured to present CGR content to the user. In some embodiments, the electronic device 120A includes a suitable combination of software, firmware, and / or hardware. According to some embodiments, the electronic device 120A presents SR content to the user via a display 122 when the user is physically present in a physical environment 103, which includes a table 107 located within a field of view 111 of the electronic device 120A. Therefore, in some embodiments, the user holds the electronic device 120A in his / her hand. In some embodiments, when providing augmented reality (AR) content, the electronic device 120A is configured to display an AR object (e.g., an AR cube 109) and implement video pass-through of the physical environment 103 (e.g., including a representation 117 of the table 107) on the display 122.

[0045] Figure 1B1 is a block diagram of an exemplary operating environment 100B according to some embodiments. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein. To this end, as a non-limiting example, the operating environment 100B includes a controller 110 and a head mounted device (HMD) 120B.

[0046] In some embodiments, the controller 110 is configured to manage and coordinate the SR presentation to the user. In some embodiments, the controller 110 includes a suitable combination of software, firmware and / or hardware. Figure 2 The controller 110 is described in more detail. In some embodiments, the controller 110 is a computing device that is located locally or remotely relative to the scene 105. For example, the controller 110 is a local server located within the scene 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 105. In some embodiments, the controller 110 is communicatively coupled to the HMD 120B via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is included in the housing of the HMD 120B.

[0047] In some embodiments, the HMD 120B is configured to present SR content to the user. In some embodiments, the HMD 120B includes a suitable combination of software, firmware, and / or hardware. Figure 3 HMD 120B is described in further detail. In some implementations, the functionality of controller 110 is provided by and / or integrated with HMD 120B.

[0048] According to some implementations, the HMD 120B presents the SR content to the user while the user is virtually and / or physically present within the scene 105 .

[0049] In some embodiments, the user wears the HMD 120B on his / her head. Thus, the HMD 120B includes one or more SR displays provided for displaying SR content. For example, in various embodiments, the HMD 120B surrounds the user's field of view. In some embodiments, such as in Figure 1AIn some embodiments, a handheld device (such as a smartphone or tablet) configured to present SR content is used in place of HMD 120B, and the user no longer wears HMD 120B but instead holds the device with the display facing the user's field of view and the camera facing the scene 105. In some embodiments, the handheld device may be placed in a housing that may be worn on the user's head. In some embodiments, HMD 120B is replaced with an SR pod, housing, or chamber configured to present SR content, wherein the user no longer wears or holds HMD 120B.

[0050] Figure 2 is a block diagram of an example of a controller 110 according to some embodiments. While some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE and / or similar type interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0051] In some embodiments, the one or more communication buses 204 include circuits for interconnecting system components and controlling communications between system components. In some embodiments, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0052] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 230 and an SR experience module 240.

[0053] The operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, the SR experience module 240 is configured to manage and coordinate single or multiple SR experiences of one or more users (e.g., a single SR experience of one or more users, or multiple SR experiences of corresponding groups of one or more users). To this end, in various embodiments, the SR experience module 240 includes a data acquisition unit 242, a tracking unit 244, a coordination unit 246, and a data transmission unit 248.

[0054] In some embodiments, the data acquisition unit 242 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the HMD 120B. To this end, in various embodiments, the data acquisition unit 242 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0055] In some embodiments, the tracking unit 244 is configured to map the scene 105 and to track at least the position / location of the HMD 120B relative to the scene 105. To this end, in various embodiments, the tracking unit 244 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0056] In some embodiments, the coordination unit 246 is configured to manage and coordinate the SR experience presented to the user by the HMD 120B. To this end, in various embodiments, the coordination unit 246 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0057] In some embodiments, the data transmission unit 248 is configured to transmit data (e.g., presentation data, position data, etc.) to at least the HMD 120B. To this end, in various embodiments, the data transmission unit 248 includes instructions and / or logic components for instructions and heuristics and metadata for the heuristics.

[0058] Although the data acquisition unit 242, tracking unit 244, coordination unit 246, and data transfer unit 248 are illustrated as residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, tracking unit 244, coordination unit 246, and data transfer unit 248 may be located in separate computing devices.

[0059] also, Figure 2 More as a functional description of various features that may be present in a particular embodiment, rather than a block diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure can be implemented in a single module, and the various functions of a single functional block can be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary from implementation to implementation, and in some embodiments, depends in part on the specific combination of hardware, software and / or firmware selected for a particular implementation.

[0060] Figure 31 is a block diagram of an example of an HMD 120B according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. To this end, as a non-limiting example, in some embodiments, the HMD 120B includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more SR displays 312, one or more optional internal-facing and / or external-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0061] In some embodiments, the one or more communication buses 304 include circuits that interconnect and control communications between system components. In some embodiments, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers (e.g., headphones or amplifiers), a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), and the like.

[0062] In some embodiments, one or more SR displays 312 are configured to provide a SR experience to the user. In some embodiments, one or more SR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some embodiments, one or more SR displays 312 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, HMD 120B includes a single SR display. In another example, HMD 120B includes an SR display for each eye of the user. In some embodiments, one or more SR displays 312 can present AR and VR content.

[0063] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face (including the user's eyes) (and thus may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 314 are configured to face forward so as to acquire image data corresponding to the scene that the user would see when the HMD 120B is not present (and thus may be referred to as a scene camera). The one or more image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0064] The memory 320 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some embodiments, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules and data structures or a subset thereof, including an optional operating system 330 and an SR presentation module 340.

[0065] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, the SR presentation module 340 is configured to present SR content to a user via one or more SR displays 312. To this end, in various embodiments, the SR presentation module 340 includes a data acquisition unit 342, an SR user interface unit 344, a sound processing unit 346, and a data transmission unit 348.

[0066] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from one or more of the controller 110 (e.g., via the communication interface 308), the I / O devices and sensors 306, or the one or more image sensors 314. To this end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0067] In some embodiments, the SR user interface unit 344 is configured to present SR content including an SR user interface via the one or more SR displays 312. To this end, in various embodiments, the SR presentation unit 344 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0068] In some embodiments, the sound processing unit 346 is configured to analyze and / or modify the sound data. To this end, in various embodiments, the plane detection unit 346 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0069] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110. To this end, in various embodiments, the data transmission unit 348 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0070] Although the data acquisition unit 342, the SR user interface unit 344, the sound processing unit 346, and the data transmission unit 348 are illustrated as residing on a single device (e.g., the HMD 120B), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the SR user interface unit 344, the sound processing unit 346, and the data transmission unit 348 may be located in separate computing devices.

[0071] also, Figure 3More as a functional description of various features that may be present in a particular embodiment, rather than a block diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules shown separately in the figure can be implemented in a single module, and the various functions of a single functional block can be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary from implementation to implementation, and in some embodiments, depends in part on the specific combination of hardware, software and / or firmware selected for a particular implementation.

[0072] Figure 4 An SR volume environment 400 based on a real environment in which a user 420 is present is shown. Figure 4 , a user 420 is wearing an HMD and investigating an SR volumetric environment 400 (as shown in the first-person view in the following figures). The SR volumetric environment 400 includes a plurality of objects, including a plurality of real objects (e.g., a table 412 and a lamp 414 corresponding to a real table and lamp of the real environment) and a plurality of virtual objects (e.g., a plurality of SR group representations 440A-440G). In various embodiments, each object is displayed at a location in the first SR volumetric environment 400 (e.g., at a location defined by three coordinates in a three-dimensional (3D) SR coordinate system). Thus, when the user 420 moves (e.g., changes position and / or orientation) in the SR volumetric environment 400, the objects are moved on the display of the HMD but maintain their locations in the SR environment 400. In various embodiments, one or more of the virtual objects (e.g., SR group representations 440A-440D) are located at locations within the field of view of the user 420 in the SR volumetric environment 400, while virtual objects (e.g., SR group representations 440E-440G) are located at locations within the SR volumetric environment 400 that are not within the field of view of the user 420, at least until the user 420 changes a position and / or orientation within the SR volumetric environment 400.

[0073] In various embodiments, the SR group representations correspond to data item groups, respectively. In various embodiments, each data item has a first metadata field (including a corresponding first metadata field value). In various embodiments, each data item has a second metadata field (including a corresponding second metadata field value). For example, in various embodiments, the data item is an audio file, and the first metadata field is an artist metadata field (including values ​​such as "Artist Name 1", "Artist Name 2", and "Artist Name 3"), and the second metadata field is an album metadata field (including values ​​such as "Album Name 1", "Album Name 2", and "Album Name 3"). In various embodiments, the data item is a document file, and the first metadata field is a document type field (including values ​​such as "text", "spreadsheet", and "slide presentation"), and the second metadata field is an author metadata field (including values ​​such as "Alice", "Bob", and "Carl"). In various embodiments, the data item is a movie file, and the first metadata field is a genre metadata field, and the second metadata field is a release year metadata field. In various embodiments, the data item is a web page of an online store, and the first metadata field is a brand metadata field, and the second metadata field is a price metadata field. In various implementations, a data item may have additional and / or other metadata fields.

[0074] In various embodiments, the SR group representations correspond to data item groups including data items having a first metadata field including a corresponding first metadata field value from among a plurality of first metadata field values. For example, in various embodiments, the SR group representations include a first SR group representation corresponding to a group of data items having a document type metadata field including "text", and a second SR group representation corresponding to a group of data items having a document type metadata field including "spreadsheet". In various embodiments, the SR group representations include a first SR group representation corresponding to a group of data items having an artist metadata field including "artist name 1", and a second SR group representation corresponding to a group of data items having an artist metadata field including "artist name 2".

[0075] Figure 5A An SR volumetric environment 500 from a user's perspective is shown according to some embodiments. The SR volumetric environment 500 includes a plurality of objects. The plurality of objects in the SR volumetric environment 500 include a plurality of real objects, such as a table 512 and a lamp 514 corresponding to a real table and a lamp of the real environment. The real objects also include a left hand 590L and a right hand 590R corresponding to the left and right hands of the user. The plurality of objects in the SR volumetric environment 500 include a plurality of virtual objects, such as a plurality of first SR group representations 541A-541D.

[0076] The first SR group representations 541A-541D correspond to a group of document files having a document type metadata field and an author metadata field. The respective first SR group representations 541A-541D correspond to respective document type metadata field values. In various embodiments, the first SR group representations 541A-541D are displayed with indicia having respective document type metadata field values, such as an icon of an application for opening a document file of the document type.

[0077] The first SR group representations 541A-541D include a spreadsheet group representation 541A corresponding to a set of document files having a document type metadata field including the value "spreadsheet". The first SR group representations 541A-541D include a presentation group representation 541B corresponding to a set of document files having a document type metadata field including the value "slide presentation". The first SR group representations 541A-541D include other group representations 541C corresponding to a set of document files having a document type metadata field including the value "other" or at least lacking a value corresponding to other first SR group representations in the first SR group representation. The first SR group representations 541A-541D include a text group representation 541D corresponding to a set of document files having a document type metadata field including the value "text".

[0078] Figure 5B A diagram showing a user input pointing to text group representation 541D is shown. Figure 5A The SR volume environment is 500. Figure 5B , the user input includes the user grabbing the text group representation 541D with his right hand and moving it toward the top of the table 512.

[0079] Figure 5C In response to detecting a user input pointing to text group representation 541D, Figure 5B The SR volume environment is 500. Figure 5C In the text group, 541D has moved from its initial position (such as Figure 5B ) to a position on top of table 512 (as shown) Figure 5C ). Thus, the text group representation 541D moves in the SR volume environment 500 relative to at least one other of the first SR group representations 541A- 541C.

[0080] Figure 5D The user is shown moving within the SR volume environment 500. Figure 5C The SR volume environment is 500. Figure 5D , the user moves from a first position in the SR volumetric environment 500 facing the table toward the right rear corner of the SR volumetric environment 500 facing the left rear corner of the SR volumetric environment 500.

[0081] Figure 5E Response to detecting user movement within the SR volume environment 500 is shown. Figure 5D Although the multiple objects are displayed at different locations in the user's field of view, for example, on a display of an HMD worn by the user, the multiple objects are displayed at the same location within the SR volumetric environment 500. Therefore, in various embodiments, Figure 5D In the embodiment of the present invention, first SR group representations 541A-541D are displayed at multiple locations in the SR volumetric environment 500 at a first multiple locations on the display, and in response to detecting a change in the user position and / or orientation in the SR volumetric environment, the first SR group representations 541A-541D are displayed at the same multiple locations in the SR volumetric environment at a second multiple locations on the display.

[0082] In various embodiments, the positions of the first SR group representations 541A-541D in the SR volumetric environment 500 are persistent over time. For example, in various embodiments, the positions are stored in non-transitory memory. Thus, even when the user stops using the application and / or device presenting the SR user interface and later returns to the application and / or device presenting the SR user interface, the first SR group representations 541A-541D are displayed at the same positions in the SR volumetric environment 500. In this way, the SR volumetric environment 500 is similar to a computer desktop, allowing the user to rearrange and organize the first SR group representations 541A-541D at any subsequent time as they see fit (e.g., using a layout such as relative to the layout). Figure 5B as described above).

[0083] Fig. 5F A diagram showing a user input pointing to text group representation 541D is shown. Figure 5E The volume environment is 500. Fig. 5F , the user input includes the user touching the text group representation 541D with a single finger of his right hand. Fig. 5F The user input in (touch text group representation 541D) is different from Figure 5B 541D). Fig. 5F User input that middle-points to text group representation 541D includes touching text group representation 541D (like a pop-up bubble), but other types of user input that select an SR group representation are described further below.

[0084] Figure 5G In response to detecting a user input pointing to text group representation 541D, Fig. 5F The volume environment is 500. Figure 5G, text group representation 541D is replaced with a plurality of second SR group representations 542A-542D. Similar to first SR group representations 541A-541D, second SR group representations 542A-542D correspond to a document file group having a document type metadata field and an author metadata field. Second SR group representations 542A-542D correspond to a document file group that includes a document type metadata field value of "text" and different values ​​included in the author metadata field.

[0085] The respective second SR group representations 542A-542D correspond to the respective author metadata field values ​​(and a selected one of the document type metadata field values). In various embodiments, the second SR group representations 542A-542D are displayed together with a tag having the respective author metadata field value, such as an avatar of the author of the document file. In various embodiments, the second SR group representations 542A-542D are also displayed together with a tag having the selected one of the document type metadata field values.

[0086] Thus, the second SR group representations 542A-542D include an Alice-text group representation 542A corresponding to a set of document files having a document type metadata field including a value of “text” and an author metadata field having an author metadata field value of “Alice”. The second SR group representations 542A-542D include a Bob-text group representation 542B corresponding to a set of document files having a document type metadata field including a value of “text” and an author metadata field having an author metadata field value of “Bob”. The second SR group representations 542A-542D include a Carl-text group representation 542C corresponding to a set of document files having a document type metadata field including a value of “text” and an author metadata field having an author metadata field value of “Carl”. The second SR group representations 542A-542D include a Dave-text group representation 542D corresponding to a set of document files having a document type metadata field including a value of “text” and an author metadata field having an author metadata field value of “Dave”.

[0087] Despite Figure 5G Other first SR group representations 541A-541C are displayed, but in various embodiments, in response to user input directed to a particular first SR group representation, the other SR group representations cease to be displayed or are displayed in a different manner (e.g., smaller, farther away, grayed out, more transparent, etc.) to focus the user's attention on the second SR group representation that replaces the particular first SR group representation.

[0088] Figure 5GA user input pointing to Dave-text group representation 542D is shown. Figure 5G , the user input includes the user touching Dave-text group representation 542D with a single finger of his right hand.

[0089] Figure 5H Response to detecting user input directed to Dave-text group representation 542D is shown. Figure 5G The SR volume environment is 500. Figure 5H , the Dave-text group representation 542D is replaced with multiple SR item representations 543A-543D. The SR item representations 543A-543D correspond to a document file having a document type metadata field and an author metadata field. Specifically, each SR item representation 543A-543D corresponds to a document file having a document type metadata field and an author metadata field, the document type metadata field including a document type metadata field value "text", and the author metadata field including an author metadata field value "Dave". The SR item representations 543A-543D include a first SR item representation 543A corresponding to a document file named "one.txt", a second SR item representation 543B corresponding to a document file named "two.txt", a third SR item representation 543C corresponding to a document file named "three.txt", and a fourth SR item representation 543D corresponding to a document file named "four.txt".

[0090] Despite Figure 5H The other second SR group representations 542A-542C are not shown, but in various embodiments, the other second SR group representations are displayed in response to user input pointing to a particular second SR group representation (either at the same location or at a different location to make room for the SR item representations).

[0091] Figure 5H A user input pointing to the third SR item representation 543C is shown. Figure 5H , the user input includes the user touching the third SR item representation 543C with a single finger of his right hand.

[0092] Fig.5I In response to detecting a user input pointing to the third SR item representation 543C, Figure 5H SR Volume Environment 500. Fig.5I The SR volume environment 500 includes another virtual object, a text display window 544 including the contents 545 of a document file named "three.txt," and a close enable indication 546 that closes the text display window 544 when selected via user input.

[0093] Figure 5Jshows a metadata field where a user enters an indication of the author Figure 5E The SR volume environment is 500. Figure 5J In the embodiment, the user input includes a speech command 591 issued by the user. In various embodiments, the speech command 591 is displayed in the SR volume environment. In various embodiments, the speech command 591 is not displayed.

[0094] Figure 5K In response to detecting user input indicating an author metadata field, Figure 5J The SR volume environment is 500. Figure 5J , first SR group representations 541A-541D are replaced with multiple third SR group representations 551A-551D. Similar to the first SR group representations 541A-541D, the third SR group representations 551A-551D correspond to a document file group having a document type metadata field and an author metadata field. However, where different first SR group representations in the first SR group representations 541A-541D correspond to different document type metadata field values, different third SR group representations in the third SR group representations 551A-551D correspond to different author metadata field values. Therefore, the corresponding third SR group representations 551A-551D correspond to corresponding author metadata field values. In various embodiments, the third SR group representations 551A-551D are displayed together with a mark having the corresponding author metadata field value, such as an avatar of the author of the document file.

[0095] Thus, third SR group representations 551A-551D include Alice group representation 551A corresponding to a set of document files having an author metadata field including the value “Alice”. Third SR group representations 551A-551D include Bob group representation 551B corresponding to a set of document files having an author metadata field including the value “Bob”. Third SR group representations 551A-551D include Carl group representation 551C corresponding to a set of document files having an author metadata field including the value “Carl”. Third SR group representations 551A-551D include Dave group representation 551D corresponding to a set of document files having an author metadata field including the value “Dave”.

[0096] In various embodiments, replacing the first SR group representations 541A-541D with the third SR group representations 551A-551D includes displaying an animation in which the first SR group representations 541A-541D are displayed as decomposing into SR item representations that move to new locations and converging into the third SR group representations 551A-551D. Figure 5L1-5L6 Shown in Figure 5J and Figure 5K SR volume environment 500 during the animation between.

[0097] Figure 5M The diagram shows a user input including one directed to Carl group representation 551C. Figure 5K The SR volume environment is 500. Figure 5M , the user input includes the user touching Carl group representation 551C with a single finger of his right hand.

[0098] Figure 5N 551C in response to detecting a user input directed to Carl group representation 551C. Figure 5M The volume environment is 500. Figure 5N , Carl group representation 551C is replaced with a plurality of fourth SR group representations 552A-552C. Similar to third SR group representations 551A-551C, fourth SR group representations 552A-552C correspond to a group of document files having a document type metadata field and an author metadata field. However, fourth SR group representations 552A-552C correspond to a group of document files that include the author metadata field value "Carl" (and a different value in the document type metadata field).

[0099] The respective fourth SR group representations 552A-552C correspond to the respective document type metadata field values ​​(and the selected one of the author metadata field values). In various embodiments, the fourth SR group representations 552A-552C are displayed together with an indicia having the respective document type metadata field values, such as an icon of an application for opening a document file of the document type. In various embodiments, the fourth SR group representations 552A-552C are also displayed together with an indicia having the selected one of the author metadata field values.

[0100] Thus, the fourth SR group representations 552A-552C include a Text-Carl group representation 552A corresponding to a set of document files having a Document Type metadata field including a value of “Text” and an Author metadata field having an Author metadata field value of “Carl”. The fourth SR group representations 552A-552C include a Presentation-Carl group representation 552B corresponding to a set of document files having a Document Type metadata field including a value of “Slide Presentation” and an Author metadata field having an Author metadata field value of “Carl”. The fourth SR group representations 552A-552C include a Spreadsheet-Carl group representation 552C corresponding to a set of document files having a Document Type metadata field including a value of “Spreadsheet” and an Author metadata field having an Author metadata field value of “Carl”. Figure 5N In the example of , there is no document file that has a document type metadata field that includes the value "Other" and an author metadata field that has the value "Carl". Figure 5N In the example of , the fourth SR group representation 552A-552C does not include an Other-Carl group representation corresponding to a set of document files having a Document Type metadata field including a value of "Other" and an Author metadata field having an Author metadata field value of "Carl".

[0101] Despite Figure 5N Other third SR group representations 551A-551C are displayed, but in various embodiments, in response to user input directed to a particular third SR group representation, the other three SR group representations stop being displayed or are displayed in a different manner (e.g., smaller, farther away, grayed out, more transparent, etc.) to focus the user's attention on the fourth SR group representation that replaces the particular third SR group representation.

[0102] Figure 5N A user input pointing to the Presentation-Carl group representation 552B is shown. Figure 5N , the user input includes the user touching presentation-Carl group representation 552B with a single finger of his right hand.

[0103] Fig.5O Response to detecting user input pointing to presentation-Carl group representation 552B is shown. Figure 5N The SR volume environment is 500. Fig.5O , Presentation-Carl group representation 552B is replaced with multiple SR item representations 553A-553B. The SR item representations 553A-553B correspond to a document file having a document type metadata field and an author metadata field. Specifically, each SR item representation 553A-553B corresponds to a document file having a document type metadata field and an author metadata field, the document type metadata field including a document type metadata field value "Slide Presentation", and the author metadata field including an author metadata field value "Carl". The SR item representations 553A-553B include a first SR item representation 553A corresponding to a document file named "one.ppt" and a second SR item representation 553B corresponding to a document file named "two.ppt".

[0104] Despite Fig.5O The other fourth SR group representations 552A-552C are not shown, but in various embodiments, the other fourth SR group representations are displayed in response to user input pointing to a particular fourth SR group representation (either at the same location or at a different location to make room for the SR item representations).

[0105] Fig.5O A user input pointing to the second SR item representation 553B is shown. Fig.5O , the user input includes the user touching the second SR item representation 553B with a single finger of his right hand.

[0106] Figure 5P In response to detecting a user input pointing to the second SR item representation 553B, Fig.5O SR Volume Environment 500. Figure 5P The SR volume environment 500 includes another virtual object, a slide presentation display window 554 including a portion of the content 555 of a document file named "two.ppt", navigation enable indications 557A-557B for navigating within the content of the document file named "two.ppt", and a close enable indication 556 that closes the slide presentation display window 554 when selected via user input.

[0107] Fig. 6A An SR volumetric environment 600 from a user's perspective is shown according to some embodiments. The SR volumetric environment 600 includes a plurality of objects. The plurality of objects in the SR volumetric environment 600 include a plurality of real objects, such as a table 612 and a lamp 614 corresponding to a real table and a lamp of the real environment. The real objects also include a left hand 690L and a right hand 690R corresponding to the left hand and the right hand of the user. The plurality of objects in the SR volumetric environment 600 include a plurality of virtual objects, such as a plurality of first SR group representations 641A-641D.

[0108] The first SR group representations 641A-641D correspond to a group of audio files having an artist metadata field and an album metadata field. The respective first SR group representations 641A-641D correspond to respective artist metadata field values. In various embodiments, the first SR group representations 641A-641D are displayed with indicia having respective artist metadata field values, such as a photo of the artist of the audio file and / or text indicating the artist metadata field value.

[0109] The first SR group representations 641A-641D include an Artist Name 1 group representation 641A corresponding to a group of audio files having an Artist metadata field including a value of "Artist Name 1". The first SR group representations 641A-641D include an Artist Name 2 group representation 641B corresponding to a group of audio files having an Artist metadata field including a value of "Artist Name 2". The first SR group representations 641A-641D include an Artist Name 3 group representation 641C corresponding to a group of audio files having an Artist metadata field including a value of "Artist Name 3". The first SR group representations 641A-641D include an Artist Name 4 group representation 641D corresponding to a group of audio files having an Artist metadata field including a value of "Artist Name 4".

[0110] Figure 6B Shows Fig. 6A 6, wherein first SR group representations 641A-641D are displayed as spinning balls using indicia with corresponding artist metadata field values ​​in the form of multiple indicia of corresponding album metadata field values ​​associated with the artist metadata field value (e.g., album art). Thus, at various different times, the user's field of view includes first SR group representations 641A-641D displaying various different album art.

[0111] Figure 6C Shows Fig. 6A 6, wherein a first SR group representation 641A-641D is displayed with indicia having a corresponding artist metadata field value in the form of a plurality of indicia of a corresponding album metadata field value (e.g., album cover) associated with the artist metadata field value rotating around a center (in various embodiments, the artist metadata field value includes additional indicia of the corresponding artist metadata field value, such as an artist photo and / or text indicating the artist metadata field value). Thus, the first SR group representation 641A-641D is displayed as a "solar system" in which the planets are the album covers and the sun (if present) is the artist photo.

[0112] Fig.6D Shows Fig. 6A SR volume environment 600, in which a first SR group representation 641A-641D is displayed together with a tag having a corresponding artist metadata field value, the tag being in the form of multiple tags of the corresponding album metadata field value (e.g., album cover), the multiple tags of the corresponding album metadata field value being associated with the artist metadata field value in a cloud (in various embodiments, the artist metadata field value includes additional tags of the corresponding artist metadata field value, such as an artist photo and / or text indicating the artist metadata field value).

[0113] Fig.6D A user input pointing to the artist name 3 group representation 641C is shown. Fig.6D , the user input includes the user touching the artist name 3 group representation 641C with a single finger of his right hand. Although the user input directed to the artist name 3 group representation 641C includes touching the artist name 3 group representation 641C, other types of user inputs that select SR group representations are further described below.

[0114] Fig. 6E 641C in response to detecting user input pointing to artist name 3 group representation 641C Fig.6D The volume environment is 600. Fig. 6E, Artist Name 3 group representation 641C is replaced with a plurality of second SR group representations 642A-642D. Similar to the first SR group representations 641A-641D, the second SR group representations 642A-642D correspond to a group of audio files having an artist metadata field and an album metadata field. The second SR group representations 642A-642D correspond to a group of audio files that include an artist metadata field value "Artist Name 3" (and a different value in the album metadata field).

[0115] The respective second SR group representations 642A-642D correspond to the respective album metadata field values ​​(and a selected one of the artist metadata field values). In various embodiments, the second SR group representations 642A-642D are displayed with an indicia of the respective album metadata field values, such as an album cover.

[0116] Thus, the second SR group representations 642A-642D include an Album Name 1 group representation 642A corresponding to a group of audio files having an artist metadata field including a value of "Artist Name 3" and an album metadata field having an album metadata field value of "Album Name 1". The second SR group representations 642A-642D include an Album Name 2 group representation 642B corresponding to a group of audio files having an artist metadata field including a value of "Artist Name 3" and an album metadata field having an album metadata field value of "Album Name 2". The second SR group representations 642A-642D include an Album Name 3 group representation 642C corresponding to a group of audio files having an artist metadata field including a value of "Artist Name 3" and an album metadata field having an album metadata field value of "Album Name 3". The second SR group representations 642A-642D includes an Album Name 4 group representation 642D corresponding to a group of audio files having an Artist metadata field including a value of "Artist Name 3" and an Album metadata field having an Album metadata field value of "Album Name 4".

[0117] Despite Fig. 6E Other first SR group representations 641A-641C are displayed in the figure, but in various embodiments, in response to user input directed to a particular first SR group representation, the other SR group representations are displayed in the same manner or in a different manner (e.g., smaller, farther away, grayed out, more transparent, etc.) to focus the user's attention on the second SR group representation that replaces the particular first SR group representation.

[0118] In various embodiments, replacing the first SR group representation 641A-641D with the second SR group representation 642A-642D includes displaying an animation in which the first SR group representation 641A-641D (which includes indicia of associated album metadata field values, such as album art) is displayed as decomposing into the second SR group representation 642A-642D. Figures 6F1 to 6F4 Shown in Fig. 6E SR volume environment 600 during the animation between FIG. 6F .

[0119] Figure 6G The SR volume environment 600 of FIG. 6F is shown in which the user input is directed to the album title 4 group representation 642D. Figure 6G , the user input includes the user touching the album name 4 group representation 642D with a single finger of his right hand.

[0120] Figure 6H In response to detecting user input pointing to album name 4 group representation 642D Figure 6G The volume environment is 600. Figure 6H , the album name 4 group representation 642D is replaced with multiple SR item representations 643A-643D. The SR item representations 643A-643D correspond to audio files (e.g., songs) having an artist metadata field and an album metadata field. Specifically, each SR item representation 643A-643D corresponds to an audio file having an artist metadata field and an album metadata field, the artist metadata field including an artist metadata field value "Artist Name 3", and the album metadata field including an album metadata field value "Album Name 4". The SR item representations 643A-643D include a first SR item representation 643A corresponding to an audio file named "Song Name 1", a second SR item representation 643B corresponding to an audio file named "Song Name 2", a third SR item representation 643C corresponding to an audio file named "Song Name 3", and a fourth SR item representation 643D corresponding to an audio file named "Song Name 4".

[0121] Despite Figure 6H Other second SR group representations are not shown, but in various embodiments, other SR group representations are displayed in response to user input pointing to a particular second SR group representation (either at the same location or at a different location to make room for the SR item representations).

[0122] Figure 6H A user input pointing to the fourth SR item representation 643D is shown. Figure 6H , the user input includes the user touching the fourth SR item representation 643D with a single finger of his right hand. In response to detecting the user input directed to the fourth SR item representation, the audio file is played (e.g., on a speaker of the HMD worn by the user).

[0123] Although certain user inputs (e.g., touch Fig. 5F The ball or touch Fig.6D ), but other user inputs for selecting SR group representations are described below.

[0124] Fig. 7A shows a representation of 641D where the user input points to the artist name 4 Fig. 6A The SR volume environment is 600. Fig. 7A , the user input includes the user grabbing the artist name 4-group representation 641D with his right hand and moving it toward the user.

[0125] Figure 7B shows the first portion of the user input pointing to the artist name 4 group representation 641D. Fig. 7A The SR volume environment is 600. Figure 7B , the artist name 4-group representation 641D is moved from its initial position in the SR volume environment 600 to a position closer to the user's position. Fig. 7A In contrast, the artist name 4 group representation 641D appears to be magnified in the user's field of view. In addition, the other representations in the first SR group representations 641A-641C are faded.

[0126] Figure 7C shows the second portion of the user input pointing to the artist name 4 group representation 641D. Figure 7B The SR volume environment is 600. Figure 7C , in response to the location of the Artist Name 4 group representation 641D being within the threshold distance from the user location, the Artist Name 4 group representation 641D is replaced with a plurality of second SR group representations 647A-647D. Similar to the first SR group representations 641A-641D, the second SR group representations 647A-647D correspond to a group of audio files having an artist metadata field and an album metadata field. The second SR group representations 647A-647D correspond to a group of audio files that include an artist metadata field value of "Artist Name 4" (and a different value in the album metadata field).

[0127] Fig. 8A shows a representation of 641D where the user input points to the artist name 4 Fig. 6A The SR volume environment is 600. Fig. 8A , the user input includes the user pulling apart (or spreading out) the artist name 4-group representation 641D with his left hand and his right hand. In various embodiments, the pull-apart gesture may be performed as a spread gesture using a single hand of the user (where the user's index finger and thumb spread out at the location of the artist name 4-group representation 641D).

[0128] Figure 8B shows the first portion of the user input pointing to the artist name 4 group representation 641D. Fig. 8A The SR volume environment is 600. Figure 8B , the artist name 4 group representation 641D is enlarged at its original position in the SR volume environment 600. In addition, the other representations in the first SR group representations 641A-64C fade in and out.

[0129] Figure 8C shows the second portion of the user input pointing to the artist name 4 group representation 641D. Figure 8B The SR volume environment is 600. Figure 8C , in response to the user's left hand and the user's right hand being greater than a threshold distance apart, the Artist Name 4 group representation 641D is replaced with a plurality of second SR group representations 647A-647D. Similar to the first SR group representations 641A-641D, the second SR group representations 647A-647D correspond to a group of audio files having an artist metadata field and an album metadata field. The second SR group representations 647A-647D correspond to a group of audio files that include an artist metadata field value of "Artist Name 4" (and a different value in the album metadata field).

[0130] Figure 8C User input directed to a plurality of second SR group representations 647A-647D is shown. Figure 8C , the user input includes the user pinching (or collapsing) the plurality of second SR group representations 647A-647D with his left hand and his right hand.

[0131] Fig.8D 647A-647D after the first portion of the user input is directed to the plurality of second SR group representations 647A-647D. Figure 8C The SR volume environment is 600. Fig.8D , multiple second SR group representations 647A-647D move toward each other.

[0132] Fig. 8E The second portion of the user input directed to the plurality of second SR group representations 647A-647D is shown. Fig.8D The SR volume environment is 600. Fig. 8E , in response to the user's left hand and the user's right hand being less than a threshold distance apart, the plurality of second SR group representations 647A-647D are replaced with the artist name 4 group representation 641D (and the other first SR group representations 641A-641C are redisplayed).

[0133] Fig.9A shows a representation of 641D where the user input points to the artist name 4 Fig. 6A The SR volume environment is 600. Fig.9A In , the user input includes the user viewing the artist name 4 group representation 641D and performing a trigger gesture (e.g., winking, pressing a button, snapping a finger, etc.). The user's gaze point is Fig.9A 695, which may or may not be displayed depending on the implementation.

[0134] Fig. 9B Response to detecting user input pointing to artist name 4 group representation 641D Fig.9A The volume environment is 600. Fig. 9B , the Artist Name 4 group representation 641D is replaced with a plurality of second SR group representations 647A-647D. Similar to the first SR group representations 641A-641D, the second SR group representations 647A-647D correspond to a group of audio files having an Artist metadata field and an Album metadata field. The second SR group representations 647A-647D correspond to a group of audio files that include an Artist metadata field value of "Artist Name 4" (and a different value in the Album metadata field).

[0135] Fig. 10A shows a representation of 641D where the user input points to the artist name 4 Fig. 6A The SR volume environment is 600. Fig. 10A , the user input includes the user moving within the SR volume environment 600 toward the artist name 4 group representation 641D.

[0136] Fig. 10B shows the user input after pointing to the artist name 4 group representation 641D Fig. 10A The volume environment is 600. Fig. 10B , in response to the user being less than a threshold distance from the artist name 4 group representation 641D, the artist name 4 group representation 641D is replaced with a plurality of second SR group representations 647A-647D. Similar to the first SR group representations 641A-641D, the second SR group representations 647A-647D correspond to a group of audio files having an artist metadata field and an album metadata field. The second SR group representations 647A-647D correspond to a group of audio files that include an artist metadata field value of "Artist Name 4" (and a different value in the album metadata field).

[0137] Fig.11A An SR volume environment 700 is shown from a user's perspective, according to some implementations. Fig.11A The SR volume environment 700 is basically similar to Fig. 6A6, but lacks the artist name 4 group representation 641D. Therefore, the SR volumetric environment 700 includes a plurality of objects. The plurality of objects in the SR volumetric environment 700 include a plurality of real objects, such as a table 712 and a lamp 714 corresponding to a real table and a lamp of the real environment. The real objects also include a left hand 790L and a right hand 790R corresponding to the left hand and the right hand of the user. The plurality of objects in the SR volumetric environment 700 include a plurality of virtual objects, such as a plurality of first SR group representations 741A-741C.

[0138] The first SR group representations 741A-741C correspond to a group of audio files having an artist metadata field and an album metadata field. The respective first SR group representations 741A-741C correspond to respective artist metadata field values. In various embodiments, the first SR group representations 741A-741C are displayed with indicia having respective artist metadata field values, such as a photo of the artist of the audio file and / or text indicating the artist metadata field value.

[0139] In various embodiments, the first SR group representations 741A-741C emit sounds from their respective locations in the SR volumetric environment 700. Recognizing that the first SR group representations 741A-741C are virtual objects, the phrase "emit sounds" and similar phrases are used herein as shorthand for "appear to emit sounds." For example, when speakers within the HMD generate real sounds, the virtual objects emit sounds to simulate the sounds that would emanate from the virtual objects if the virtual objects were real objects at their locations in the real environment. In various embodiments, when the objects emit sounds from the locations in the SR volumetric environment 700, the volume of the sounds is based on the distance between the user and the location of the objects in the SR volumetric environment. In various embodiments, when the objects emit sounds from the locations in the SR volumetric environment 700, the sounds arrive at the user's two ears at different times to indicate the angle (relative to the user) at which the sounds are emitted.

[0140] Therefore, when the SR group representation is not within the user's current field of view (such as Figure 4 SR group representations 440E-440G), the user is aware of the existence of the SR group representations in the SR volume environment.

[0141] In various embodiments, when the SR group representation represents a group of audio files, the sound emitted by the SR group representation is at least a portion of the audio data of one of the audio files. For example, when the SR group representation represents a group of audio files of an artist's songs, the sound emitted by the SR group representation is one of the artist's songs.

[0142] When there are multiple SR group representations in an SR volume environment, different sounds emitted by different SR group representations may produce inconsistent jarring sounds. Therefore, in various embodiments, the sounds emitted by different SR group representations are selected (and optionally processed) to reduce inconsistencies.

[0143] exist Fig.11A , the first SR group representations 741A-741C include an Artist Name 1 group representation 741A corresponding to a group of audio files having an artist metadata field including a value of “Artist Name 1” that emits a sound corresponding to a first audio file having an artist metadata field including a value of “Artist Name 1”. The first SR group representations 741A-741C include an Artist Name 2 group representation 741B corresponding to a group of audio files having an artist metadata field including a value of “Artist Name 2” that emits a sound corresponding to the first audio file having an artist metadata field including a value of “Artist Name 2”. The first SR group representations 741A-741C include an Artist Name 3 group representation 741C corresponding to a group of audio files having an artist metadata field including a value of “Artist Name 3” that emits a sound corresponding to the first audio file having an artist metadata field including a value of “Artist Name 3”.

[0144] In various embodiments, the first audio file is randomly selected. In various embodiments, the second audio file is selected based on the first audio file. In various embodiments, the second audio file is selected to reduce inconsistencies between the first audio file and the second audio file. In various embodiments, the second audio file is based on audio data of the first audio file and the second audio file, such as a pitch, melody, or beat of the audio data. In various embodiments, the second audio file is based on metadata of the first audio file and the second audio file, such as a song name or genre.

[0145] In various embodiments, the second audio file is selected based on the pitch of the first audio file and the second audio file. For example, in various embodiments, the second audio file is more likely to be selected (from the group of audio files having an artist metadata field including the value "Artist Name 2") if the second audio file has the same pitch as the first audio file, has a pitch that is close to the pitch of the first audio file (and can therefore be easily processed to have the same pitch as the first audio file), or has a pitch that is consistent with the first audio file (e.g., differs by one fifth).

[0146] In various embodiments, the second audio file is selected based on the melodies of the first audio file and the second audio file. For example, in various embodiments, the second audio file is more likely to be selected (from the group of audio files having an artist metadata field including the value "Artist Name 2") if the second audio file has the same melody as the first audio file, has a melody that is close to the melody of the first audio file (and can therefore be easily processed to have the same melody as the first audio file), or has a multiple of the melody of the first audio file (e.g., half or twice).

[0147] In various embodiments, the second audio file is selected based on the tempo of the first audio file and the second audio file. For example, in various embodiments, the second audio file is more likely to be selected (from the group of audio files having an artist metadata field including the value "Artist Name 2") if the second audio file has the same tempo as the first audio file, has a tempo that is close to the tempo of the first audio file (and can therefore be easily processed to have the same tempo as the first audio file), or has a multiple of the tempo of the first audio file (e.g., half or twice).

[0148] In various embodiments, the second audio file is selected based on the instruments of the first audio file and the second audio file. For example, in various embodiments, if the second audio file uses one or more of the same instruments as the first audio file (e.g., guitar, violin, flute, electric bass, harp, etc.), then the second audio file is more likely to be selected (from the group of audio files having an artist metadata field including the value "Artist Name 2").

[0149] In various embodiments, the second audio file is selected based on samples common to the first audio file and the second audio file. For example, in various embodiments, the second audio file is selected as a second audio file that includes samples of the first audio file or samples of the artist of the first audio file.

[0150] In various embodiments, the second audio file is selected based on the names of the first audio file and the second audio file. For example, in various embodiments, the second audio file is selected as an audio file having the same name as the first audio file. Thus, the second audio file may be a cover version of the first audio file. In various embodiments, the second audio file is selected based on the artist and backing artists of the first song of the second file. Thus, the second audio file may be a duet or a collaboration between Artist Name 2 and Artist Name 1.

[0151] In various embodiments, the second audio file is at least partially randomly selected. Therefore, the first audio file and the second audio file are not always the same two songs.

[0152] As described above, in various embodiments, the first audio file is randomly selected (from the set of audio files having an artist metadata field including the value "Artist Name 1"). In various embodiments, the first audio file and the second audio file are simultaneously selected to find two audio files with matching characteristics. Thus, in various embodiments, the first audio file is selected based on the second audio file.

[0153] In various embodiments, the second audio file is processed or modified to reduce the inconsistency between the first audio file and the second audio file. For example, in various embodiments, the pitch and / or tempo of the second audio file is changed to better match the pitch and / or tempo of the first audio file. Thus, in various embodiments, the sound corresponding to the second audio file is emitted by the artist name 2SR representation 741B based on the first audio file.

[0154] In various implementations, the third audio file is selected based on the first audio file and the second audio file.In various implementations, the SR group representation emits sound only when a distance between a user in the SR volume environment 700 and the SR group representation is less than a threshold amount.

[0155] Fig.11A A user is shown moving within the SR volume environment 700 . Fig. 11B The SR volumetric environment 700 is shown after the user has moved within the SR volumetric environment 700. The first SR group representations 741A-741C are displayed at the same location in the SR volumetric environment 700 (but possibly at different locations on the display). In addition, the distance between the user and the respective first SR group representations 741A-741C has changed. As a result, the volume (and direction) of the sounds emitted by the first SR group representations 741A-741C has also changed.

[0156] Fig. 12A An SR volume environment 800 is shown from a user's perspective, according to some implementations. Fig. 12A The SR volume environment 800 is basically similar to Fig. 6A The SR volume environment 600 of FIG. 800 includes a plurality of objects. The plurality of objects in the SR volume environment 800 include a plurality of real objects, such as a table 812 and a lamp 814 corresponding to a real table and a lamp of the real environment. The real objects also include a left hand 890L and a right hand 890R corresponding to a left hand and a right hand of the user. The plurality of objects in the SR volume environment 800 include a plurality of virtual objects, such as a plurality of first SR group representations 841A-841D.

[0157] The first SR group representations 841A-841D correspond to a group of audio files having an artist metadata field and an album metadata field. The respective first SR group representations 841A-841D correspond to respective artist metadata field values. In various embodiments, the first SR group representations 841A-841D are displayed with indicia having respective artist metadata field values, such as a photo of the artist of the audio file and / or text indicating the artist metadata field value.

[0158] The first SR group representations 841A-841D include an Artist Name 1 group representation 841A corresponding to a group of audio files having an Artist metadata field including a value of "Artist Name 1". The first SR group representations 841A-841D include an Artist Name 2 group representation 841B corresponding to a group of audio files having an Artist metadata field including a value of "Artist Name 2". The first SR group representations 841A-841D include an Artist Name 3 group representation 841C corresponding to a group of audio files having an Artist metadata field including a value of "Artist Name 3". The first SR group representations 841A-841D include an Artist Name 4 group representation 841D corresponding to a group of audio files having an Artist metadata field including a value of "Artist Name 4".

[0159] The user's gaze is Fig. 12A 895, which may or may not be displayed, depending on the implementation. Fig. 12A , the user is viewing the artist name 4 group representation 841D. In various embodiments, when the user is viewing the first SR group representation, the sound corresponding to one of the audio files in the group of audio files is presented to the user. In various embodiments, when the SR group representation represents a group of audio files, the sound emitted by the SR group representation is at least a portion of the audio data of one of the audio files. For example, when the SR group representation represents a group of audio files of an artist's songs, the sound emitted by the SR group representation is one of the artist's songs. Therefore, in Fig. 12A In the figure, artist name 4 group indicates that 841D emits the sound corresponding to the song of artist name 4.

[0160] Fig. 12B 841C. Fig. 12AIn various embodiments, the sound presented fades from a first audio file (e.g., a song by artist name 4) to a second audio file (e.g., a song by artist name 3). In various embodiments, the second audio file is based on the first audio file (e.g., as described above with respect to Fig.11A described above).

[0161] Fig. 12C It shows that when the user's gaze point reaches the artist name 3 group representation 841C Fig. 12B At this point, the rendered sound has been fully cross-faded from the first audio file to the second audio file.

[0162] Fig.13 A flowchart representation of a method 1300 for moving an object in an SR user interface according to some embodiments is shown. In various embodiments, the method 1300 is performed by a device (e.g., Figure 3 In some embodiments, the method 1300 is performed by a processing logic component, including hardware, firmware, software, or a combination thereof. In some embodiments, the method 1300 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).

[0163] The method 1300 begins in block 1310 with the device identifying a plurality of data items, each of the plurality of data items having a first metadata field (including a corresponding first metadata field value). In various embodiments, each of the plurality of data items also has a second metadata field (including a corresponding second metadata field value). For example, in various embodiments, the data item is an audio file, and the first metadata field is an artist metadata field (including values ​​such as "Artist Name 1", "Artist Name 2", and "Artist Name 3"), and the second metadata field is an album metadata field (including values ​​such as "Album Name 1", "Album Name 2", and "Album Name 3"). In various embodiments, the data item is a document file, and the first metadata field is a document type field (including values ​​such as "text", "spreadsheet", and "slideshow"), and the second metadata field is an author metadata field (including values ​​such as "Alice", "Bob", and "Carl"). In various embodiments, the data item is a movie file, and the first metadata field is a genre metadata field, and the second metadata field is a release year metadata field. In various implementations, the data item is a web page of an online store, and the first metadata field is a brand metadata field, and the second metadata field is a price metadata field. In various implementations, the data item may have additional and / or other metadata fields.

[0164] In various implementations, identifying the plurality of data items includes populating the first metadata field with the first metadata field value for each data item in the plurality of data items. In various implementations, the populating is performed based on an inspection of the data items, such as by a clustering algorithm.

[0165] In various implementations, a plurality of data items are stored with metadata including a first metadata field value in a first metadata field and a second metadata field value in a second metadata field.

[0166] Method 1300 continues in box 1320, where the device displays an SR volume environment including multiple first SR group representations corresponding to multiple first group data items, each of the multiple first group data items respectively including a data item among the multiple data items having a corresponding first metadata field value among multiple first metadata field values ​​of the first metadata field.

[0167] For example, in Figure 5A , the device displays an SR volume environment 500 that includes a plurality of first SR group representations 541A-541D that respectively correspond to a plurality of first group data items (e.g., document files). Each of the plurality of first group data items respectively includes a data item having a corresponding first metadata field value (e.g., "text," "spreadsheet," and "slide presentation") of a plurality of first metadata field values ​​for a first metadata field (e.g., a document type metadata field) in the plurality of data items.

[0168] For example, in Fig. 6A , the device displays an SR volume environment 600 that includes a plurality of first SR group representations 641A-641D that respectively correspond to a plurality of first group data items (e.g., audio files). Each of the plurality of first group data items respectively includes a data item having a corresponding first metadata field value (e.g., "Artist Name 1," "Artist Name 2," and "Artist Name 3") of a plurality of first metadata field values ​​of a first metadata field (e.g., an artist metadata field) in the plurality of data items.

[0169] In various embodiments, displaying the SR volumetric environment includes displaying a plurality of first SR group representations at a plurality of locations in the SR volumetric environment on the display at a first plurality of locations on the display. In various embodiments, displaying the SR volumetric environment includes detecting a change in a user's position and / or orientation in the SR volumetric environment. For example, Figure 5DIn the embodiment, the first SR group representations 541A-541D are displayed at a plurality of locations in the SR volume environment 500 at a first plurality of locations on the display, and the user changes the position and / or orientation in the SR volume environment 500. In various embodiments, displaying the SR volume environment includes, in response to detecting a change in the user's position and / or orientation in the SR volume environment, displaying the plurality of first SR group representations at the plurality of locations in the SR volume environment on the display at a second plurality of locations on the display. For example, in Figure 5E In response to detecting a change in the user's position and / or orientation, the first SR group representations 541A-541D are displayed at a plurality of locations in the SR volume environment 500 at a second plurality of locations on the display.

[0170] In various embodiments, multiple locations in the SR volume environment are saved in association with the user profile. Thus, these locations are persistent over time.

[0171] In various embodiments, each first SR group representation in the plurality of first SR group representations is displayed together with a marker of a corresponding first metadata field value in the plurality of first metadata field values. Figure 5A In FIG. 5 , the first SR group representations 541A-541D are displayed together with an icon of an application for opening a document file of the document type. Fig. 6A In , the first SR group representation 641A-641D is displayed with the artist photo and text indicating the artist metadata field value. Figure 6B In various embodiments, the indicia of a corresponding first metadata field value in the plurality of first metadata field values ​​is in the form of a plurality of indicia of a corresponding second metadata field value (e.g., album cover) associated with the first metadata field value. Thus, in various embodiments, a first one of the first SR group representations is displayed differently than a second one of the first SR group representations.

[0172] Method 1300 continues at block 1330, where the device detects a first user input directed to a particular first SR group representation of the plurality of first SR group representations. In various embodiments, detecting the first user input includes detecting a position of at least a portion of the user in the SR volume environment. For example, in Figure 5B , the user grabs text group representation 541D and moves it to place it on table 512. In various embodiments, the first user input includes the use of a cursor or the issuance of a verbal command.

[0173] The method 1300 continues at block 1340 where the device, in response to detecting the first user input, moves a particular first SR group representation of the plurality of first SR group representations relative to at least another first SR group representation of the plurality of first SR group representations in the SR volumetric environment. Figure 5C In response to detecting the first user input, the text group representation 541D is moved in the SR volume environment 500 relative to the other first SR group representations 541A- 541C.

[0174] In various embodiments, method 1300 further includes detecting a second user input directed to a specific first SR group representation of the plurality of first SR group representations corresponding to a specific first group data item of the first group of data items, the specific first group data item of the first group of data items including a data item of the plurality of data items having a first metadata field including the specific first metadata field value of the plurality of first metadata field values. For example, Fig. 5F User input is shown directed to text group representation 541D.

[0175] In various embodiments, method 1300 further includes, in response to detecting a second user input, replacing a specific first SR group representation of the plurality of first SR group representations with a plurality of second SR group representations corresponding to a plurality of second group data items, each of the plurality of second group data items respectively including a data item of the plurality of data items having a first metadata field and a second metadata field, the first metadata field including a specific first metadata field value of the plurality of first metadata field values, and the second metadata field including a corresponding second metadata field value of the plurality of second metadata field values. For example, in Figure 5G In response to detecting Fig. 5F , the text group representation 541D is replaced with multiple second SR group representations 542A-542D, and the multiple second SR group representations respectively include data items having a first metadata field (e.g., a document type metadata field) and a second metadata field (e.g., an author metadata field) from multiple data items, the first metadata field including a specific first metadata field value (e.g., "text") from multiple first metadata field values, and the second metadata field including corresponding second metadata field values ​​(e.g., "Alice", "Bob", "Carl", and "Dave") from multiple second metadata field values.

[0176] In various implementations, method 1300 also includes detecting a second user input indicating a second metadata field. Figure 5J In , the user issues a verbal command indicating the author metadata field.

[0177] In various embodiments, method 1300 further includes, in response to detecting a second user input, replacing the plurality of first SR group representations with a plurality of second SR group representations corresponding to a plurality of second group data items, respectively, each second group data item in the plurality of second group data items respectively including a data item in the plurality of data items having a corresponding second metadata field value in a plurality of second metadata field values ​​of the second metadata field. Figure 5K In the embodiment of the present invention, in response to a speech command, the plurality of first SR group representations 541A-541B are replaced with a plurality of second SR group representations (e.g., third SR group representations 551A-551D), wherein the plurality of second SR group representations respectively correspond to a plurality of second group data items, wherein each of the plurality of second group data items respectively includes a data item having a corresponding second metadata field value (e.g., "Alice," "Bob," "Carl," and "Dave") among a plurality of second metadata field values ​​having a second metadata field (e.g., an author metadata field) in the plurality of data items.

[0178] Fig.14 1 is a flowchart representation of a method 1400 for reorganizing data items in an SR user interface according to some embodiments. In various embodiments, the method 1400 is performed by a device (e.g., Figure 3 In some embodiments, the method 1400 is performed by a processing logic component, including hardware, firmware, software, or a combination thereof. In some embodiments, the method 1400 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).

[0179] Method 1400 begins in block 1410, where a device identifies a plurality of data items, each of the plurality of data items having a first metadata field (including a corresponding first metadata field value). In various embodiments, each of the plurality of data items also has a second metadata field (including a corresponding second metadata field value). For example, in various embodiments, the data item is an audio file, and the first metadata field is an artist metadata field (including values ​​such as "Artist Name 1", "Artist Name 2", and "Artist Name 3"), and the second metadata field is an album metadata field (including values ​​such as "Album Name 1", "Album Name 2", and "Album Name 3"). In various embodiments, the data item is a document file, and the first metadata field is a document type field (including values ​​such as "text", "spreadsheet", and "slideshow"), and the second metadata field is an author metadata field (including values ​​such as "Alice", "Bob", and "Carl"). In various embodiments, the data item is a movie file, and the first metadata field is a genre metadata field, and the second metadata field is a release year metadata field. In various implementations, the data item is a web page of an online store, and the first metadata field is a brand metadata field, and the second metadata field is a price metadata field. In various implementations, the data item may have additional and / or other metadata fields.

[0180] In various implementations, identifying the plurality of data items includes populating the first metadata field with the first metadata field value for each data item in the plurality of data items. In various implementations, the populating is performed based on an inspection of the data items, such as by a clustering algorithm.

[0181] In various implementations, a plurality of data items are stored with metadata including a first metadata field value in a first metadata field and a second metadata field value in a second metadata field.

[0182] In various embodiments, method 1400 includes selecting a first metadata field in the plurality of metadata fields as the first metadata field based on a default metadata field for the type of the plurality of data items. For example, in various embodiments, when the plurality of data items are identified as document files, the default metadata field is a document type metadata field. For another example, in various embodiments, when the plurality of data items are identified as audio files, the default metadata field is an artist metadata field (or a genre metadata field). For another example, in various embodiments, when the plurality of data items are identified as web pages of an online store, the default metadata field is a brand metadata field.

[0183] In various embodiments, method 1400 includes selecting a first metadata field from among the plurality of metadata fields as the first metadata field based on the number of data items in the plurality of first groups of data items. For example, the metadata field is selected as the first metadata field to evenly divide the data items among the plurality of groups of data items. In various embodiments, the first metadata is selected based on contextual information such as a calendar invitation or a recently opened document.

[0184] Method 1400 continues in box 1420, where the device displays an SR volume environment including multiple first SR group representations corresponding to multiple first group data items, each of the multiple first group data items respectively including a data item among the multiple data items having a corresponding first metadata field value among multiple first metadata field values ​​of the first metadata field.

[0185] For example, in Figure 5A , the device displays an SR volume environment 500 that includes a plurality of first SR group representations 541A-541D that respectively correspond to a plurality of first group data items (e.g., document files). Each of the plurality of first group data items respectively includes a data item having a corresponding first metadata field value (e.g., "text," "spreadsheet," and "slide presentation") of a plurality of first metadata field values ​​for a first metadata field (e.g., a document type metadata field) in the plurality of data items.

[0186] For example, in Fig. 6A , the device displays an SR volume environment 600 that includes a plurality of first SR group representations 641A-641D that respectively correspond to a plurality of first group data items (e.g., audio files). Each of the plurality of first group data items respectively includes a data item having a corresponding first metadata field value (e.g., "Artist Name 1," "Artist Name 2," and "Artist Name 3") of a plurality of first metadata field values ​​of a first metadata field (e.g., an artist metadata field) in the plurality of data items.

[0187] In various embodiments, displaying the SR volumetric environment includes displaying a plurality of first SR group representations at a plurality of locations in the SR volumetric environment on the display at a first plurality of locations on the display. In various embodiments, displaying the SR volumetric environment includes detecting a change in a user's position and / or orientation in the SR volumetric environment. For example, Figure 5DIn the embodiment, the first SR group representations 541A-541D are displayed at a plurality of locations in the SR volume environment 500 at a first plurality of locations on the display, and the user changes the position and / or orientation in the SR volume environment 500. In various embodiments, displaying the SR volume environment includes, in response to detecting a change in the user's position and / or orientation in the SR volume environment, displaying the plurality of first SR group representations at the plurality of locations in the SR volume environment on the display at a second plurality of locations on the display. For example, in Figure 5E In response to detecting a change in the user's position and / or orientation, the first SR group representations 541A-541D are displayed at a plurality of locations in the SR volume environment 500 at a second plurality of locations on the display.

[0188] In various embodiments, each first SR group representation in the plurality of first SR group representations is displayed together with a marker of a corresponding first metadata field value in the plurality of first metadata field values. Figure 5A In FIG. 5 , the first SR group representations 541A-541D are displayed together with an icon of an application for opening a document file of the document type. Fig. 6A In , the first SR group representation 641A-641D is displayed with the artist photo and text indicating the artist metadata field value. Figure 6B In various embodiments, the indicia of a corresponding first metadata field value in the plurality of first metadata field values ​​is in the form of a plurality of indicia of a corresponding second metadata field value (e.g., album cover) associated with the first metadata field value. Thus, in various embodiments, a first one of the first SR group representations is displayed differently than a second one of the first SR group representations.

[0189] Method 1400 continues at block 1430, where the device detects a first user input indicating a second metadata field. In various embodiments, detecting the first user input includes detecting a user's verbal input. For example, in Figure 5J In the example embodiment, the user issues a verbal command indicating an author metadata field. In various embodiments, detecting the first user input includes detecting a selection of a displayed reorder affordance.

[0190] Method 1400 continues at block 1440, where the device, in response to detecting a second user input, replaces the plurality of first SR group representations with a plurality of second SR group representations corresponding to a plurality of second group data items, respectively, each second group data item in the plurality of second group data items respectively including a data item in the plurality of data items having a corresponding second metadata field value in a plurality of second metadata field values ​​for the second metadata field. Figure 5KIn the embodiment of the present invention, in response to a speech command, the plurality of first SR group representations 541A-541B are replaced with a plurality of second SR group representations (e.g., third SR group representations 551A-551D), wherein the plurality of second SR group representations respectively correspond to a plurality of second group data items, wherein each of the plurality of second group data items respectively includes a data item having a corresponding second metadata field value (e.g., "Alice," "Bob," "Carl," and "Dave") among a plurality of second metadata field values ​​having a second metadata field (e.g., an author metadata field) in the plurality of data items.

[0191] In various embodiments, replacing the plurality of first SR group representations with the plurality of second SR group representations includes displaying an animation, wherein the plurality of first SR representations are replaced with a plurality of SR item representations, the plurality of SR item representations are rearranged in the SR volume environment and replaced with the plurality of second SR group representations. Figure 5L1 to Figure 5L6 An animation is shown in which the plurality of first SR group representations 541A-541D are replaced with a plurality of SR item representations, which are rearranged in the SR volume environment 500 and replaced with a plurality of second SR group representations (eg, third SR group representations 551A-551C).

[0192] In various embodiments, method 1400 includes detecting a second user input directed to a particular second SR group representation of a plurality of second SR group representations corresponding to a particular second group data item of a second group of data items, the particular second group data item of the second group of data items including a data item of the plurality of data items having a second metadata field, the second metadata field including a particular second metadata field value of the plurality of second metadata field values. For example, Figure 5M User input directed to Carl group representation 551D is shown.

[0193] In various embodiments, method 1400 further includes, in response to detecting a second user input, replacing a specific second SR group representation among the plurality of second SR group representations with a plurality of third SR group representations corresponding respectively to a plurality of third group data items, each of the plurality of third group data items respectively including a data item among the plurality of data items having a second metadata field and a first metadata field, the second metadata field including a specific second metadata field value among the plurality of second metadata field values, and the first metadata field including a corresponding first metadata field value among the plurality of first metadata field values.

[0194] For example, in Figure 5N In response to detecting Figure 5M, Carl group representation 551C is replaced with multiple third SR group representations (e.g., fourth SR group representations 552A-552D), which respectively correspond to multiple third group data items, each of the multiple third group data items respectively including a data item having a second metadata field (e.g., an author metadata field) and a first metadata field (e.g., a document type metadata field) among the multiple data items, the second metadata field including a specific second metadata field value (e.g., "Carl") among the multiple second metadata field values, and the first metadata field including a corresponding first metadata field value (e.g., "text", "spreadsheet", and "slide presentation") among the multiple first metadata field values.

[0195] In various embodiments, the method 1400 further includes detecting a third user input directed to a particular third SR group representation in the plurality of third SR group representations corresponding to the particular third group data item in the third group data items. Figure 5N , the user selects Presentation-Carl group representation 552B.

[0196] In various embodiments, method 1400 further includes, in response to detecting a third user input, replacing a specific third SR group representation in the plurality of third SR group representations with a plurality of SR item representations, the plurality of SR item representations corresponding to data items in the plurality of data items having a specific first metadata field value in a plurality of first metadata field values ​​for the first metadata field and a specific second metadata field value in a plurality of second metadata field values ​​for the second metadata field, respectively. Fig.5O In response to Figure 5N In response to user input, the Presentation-Carl group representation 552B is replaced with multiple SR item representations 553A-553B, which respectively correspond to data items having a specific first metadata field value (e.g., "Slide Presentation") among a first metadata field value of a first metadata field (e.g., a document type metadata field) and a specific second metadata field value (e.g., "Carl") among a plurality of data items having a second metadata field (e.g., an author metadata field).

[0197] In various embodiments, the method 1400 further includes detecting a fourth user input pointing to a specific SR item representation in the plurality of SR item representations corresponding to the specific data item in the data items. Fig.5O , the user selects the second SR item representation 553B corresponding to the document file named "two.ppt".

[0198] In various implementations, method 1400 further includes, in response to detecting the fourth user input, opening a specific data item in the data items. Figure 5P, the SR volume environment 500 includes a slide presentation display window 554, which includes a portion of the content 555 of a document file named "two.ppt", navigation enable indications 557A-557B for navigating within the content of the document file named "two.ppt", and a close enable indication 556, which closes the slide presentation display window 554 when selected via user input.

[0199] Fig.15 1 is a flowchart representation of a method 1500 for selecting a data item group in an SR user interface according to some embodiments. In various embodiments, the method 1500 is performed by a device (e.g., Figure 3 In some embodiments, the method 1500 is performed by a processing logic component, including hardware, firmware, software, or a combination thereof. In some embodiments, the method 1500 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).

[0200] Method 1500 begins in block 1510, where a device identifies a plurality of data items, each of the plurality of data items having a first metadata field (including a corresponding first metadata field value). In various embodiments, each of the plurality of data items also has a second metadata field (including a corresponding second metadata field value). For example, in various embodiments, the data item is an audio file, and the first metadata field is an artist metadata field (including values ​​such as "Artist Name 1", "Artist Name 2", and "Artist Name 3"), and the second metadata field is an album metadata field (including values ​​such as "Album Name 1", "Album Name 2", and "Album Name 3"). In various embodiments, the data item is a document file, and the first metadata field is a document type field (including values ​​such as "text", "spreadsheet", and "slideshow"), and the second metadata field is an author metadata field (including values ​​such as "Alice", "Bob", and "Carl"). In various embodiments, the data item is a movie file, and the first metadata field is a genre metadata field, and the second metadata field is a release year metadata field. In various implementations, the data item is a web page of an online store, and the first metadata field is a brand metadata field, and the second metadata field is a price metadata field. In various implementations, the data item may have additional and / or other metadata fields.

[0201] In various implementations, identifying the plurality of data items includes populating the first metadata field with the first metadata field value for each data item in the plurality of data items. In various implementations, the populating is performed based on an inspection of the data items, such as by a clustering algorithm.

[0202] In various implementations, a plurality of data items are stored with metadata including a first metadata field value in a first metadata field and a second metadata field value in a second metadata field.

[0203] In various embodiments, method 1500 includes selecting a first metadata field in the plurality of metadata fields as the first metadata field based on a default metadata field for the type of the plurality of data items. For example, in various embodiments, when the plurality of data items are identified as document files, the default metadata field is a document type metadata field. For another example, in various embodiments, when the plurality of data items are identified as audio files, the default metadata field is an artist metadata field (or a genre metadata field). For another example, in various embodiments, when the plurality of data items are identified as web pages of an online store, the default metadata field is a brand metadata field.

[0204] In various embodiments, method 1500 includes selecting a first metadata field in the plurality of metadata fields as the first metadata field based on the number of data items in the plurality of first groups of data items. For example, the metadata field is selected as the first metadata field to evenly divide the data items in the plurality of groups of data items. In various embodiments, the first metadata is selected based on contextual information such as a calendar invitation or a recently opened document.

[0205] Method 1500 continues in box 1520, where the device displays an SR volume environment including multiple first SR group representations corresponding to multiple first group data items, each of the multiple first group data items respectively including a data item among the multiple data items having a corresponding first metadata field value among multiple first metadata field values ​​of the first metadata field.

[0206] For example, in Figure 5A , the device displays an SR volume environment 500 that includes a plurality of first SR group representations 541A-541D that respectively correspond to a plurality of first group data items (e.g., document files). Each of the plurality of first group data items respectively includes a data item having a corresponding first metadata field value (e.g., "text," "spreadsheet," and "slide presentation") of a plurality of first metadata field values ​​for a first metadata field (e.g., a document type metadata field) in the plurality of data items.

[0207] For example, in Fig. 6A, the device displays an SR volume environment 600 that includes a plurality of first SR group representations 641A-641D that respectively correspond to a plurality of first group data items (e.g., audio files). Each of the plurality of first group data items respectively includes a data item having a corresponding first metadata field value (e.g., "Artist Name 1," "Artist Name 2," and "Artist Name 3") of a plurality of first metadata field values ​​of a first metadata field (e.g., an artist metadata field) in the plurality of data items.

[0208] In various embodiments, displaying the SR volumetric environment includes displaying a plurality of first SR group representations at a plurality of locations in the SR volumetric environment on the display at a first plurality of locations on the display. In various embodiments, displaying the SR volumetric environment includes detecting a change in a user's position and / or orientation in the SR volumetric environment. For example, Figure 5D In the embodiment, the first SR group representations 541A-541D are displayed at a plurality of locations in the SR volume environment 500 at a first plurality of locations on the display, and the user changes the position and / or orientation in the SR volume environment 500. In various embodiments, displaying the SR volume environment includes, in response to detecting a change in the user's position and / or orientation in the SR volume environment, displaying the plurality of first SR group representations at the plurality of locations in the SR volume environment on the display at a second plurality of locations on the display. For example, in Figure 5E In response to detecting a change in the user's position and / or orientation, the first SR group representations 541A-541D are displayed at a plurality of locations in the SR volume environment 500 at a second plurality of locations on the display.

[0209] In various embodiments, each first SR group representation in the plurality of first SR group representations is displayed together with a marker of a corresponding first metadata field value in the plurality of first metadata field values. Figure 5A In FIG. 5 , the first SR group representations 541A-541D are displayed together with an icon of an application for opening a document file of the document type. Fig. 6A , the first SR Group representations 641A-641D are displayed with the artist photo and text indicating the artist metadata field value.

[0210] In various implementations, the indicia of corresponding first metadata field values ​​in the plurality of first metadata field values ​​includes a plurality of indicia of corresponding second metadata field values ​​in the plurality of second metadata field values. FIG. 6B to FIG. 6DIn various embodiments, the indicia of a corresponding first metadata field value in the plurality of first metadata field values ​​is in the form of a plurality of indicia of a corresponding second metadata field value (e.g., album cover) associated with the first metadata field value. Thus, in various embodiments, a first one of the first SR group representations is displayed differently than a second one of the first SR group representations.

[0211] The method 1500 continues at block 1530, where the device detects a first user input directed to a particular first SR group representation of a plurality of first SR group representations corresponding to a particular first group data item of a first group of data items, the particular first group data item of the first group of data items including a data item of a plurality of data items having a first metadata field including the particular first metadata field value of the plurality of first metadata field values. For example, Fig. 5F A user input pointing to text group representation 541D is shown. As another example, Fig.6D A user input is shown directed to the artist name 3 group representation 641C.

[0212] In various embodiments, detecting the first user input includes detecting a position of at least a portion of the user within the SR volume environment. Fig.6D In , the device detects the position of the user's hand when selecting the artist name group 3 representation 641C. Fig. 7A In , the device detects the position of the user's hand when the artist name 4 group representation 641D is pulled closer. Fig. 8A In , the device detects the position of the user's hand when pulling open the artist name 4 group representation 641D. FIG. 10A to FIG. 10B In the embodiment of the present invention, the device detects the position of the user when moving from a first position to a second position.

[0213] In various embodiments, detecting the first user input includes detecting a gaze point of a user in the SR volume environment. Fig.9A In , when a selection trigger is triggered, the device detects the user's gaze point.

[0214] In various implementations, detecting the first user input includes detecting a user's speech input or a position of a cursor.

[0215] Method 1500 continues in box 1540, where the device, in response to detecting the first user input, replaces a specific first SR group representation among the multiple first SR group representations with a plurality of second SR group representations corresponding to a plurality of second group data items, respectively, each of the plurality of second group data items respectively including a data item among the plurality of data items having a first metadata field and a second metadata field, the first metadata field including a specific first metadata field value among the plurality of first metadata field values, and the second metadata field including a corresponding second metadata field value among the plurality of second metadata field values.

[0216] For example, in Figure 5G In response to detecting Fig. 5F , the text group representation 541D is replaced with multiple second SR group representations 542A-542D, each of which includes data items having a first metadata field (e.g., a document type metadata field) and a second metadata field (e.g., an author metadata field) from among the multiple data items, the first metadata field including a specific first metadata field value (e.g., "text") from among the multiple first metadata field values, and the second metadata field including a corresponding second metadata field value (e.g., "Alice", "Bob", "Carl", and "Dave") from among the multiple second metadata field values.

[0217] For example, in Fig. 6E In response to detecting Fig.6D , the artist name 3 group representation 641C is replaced with multiple second SR group representations 642A-642D, each of which includes a data item having a first metadata field (e.g., an artist metadata field) and a second metadata field (e.g., an album metadata field) from among the multiple data items, the first metadata field including a specific first metadata field value (e.g., "Artist Name 3") from among the multiple first metadata field values, and the second metadata field including a corresponding second metadata field value (e.g., "Album Name 1", "Album Name 2", "Album Name 3", and "Album Name 4") from among the multiple second metadata field values.

[0218] In various embodiments, replacing a specific first SR group representation in the plurality of first SR group representations with the plurality of second SR group representations includes displaying an animation in which a plurality of marks of corresponding second metadata field values ​​in the plurality of second metadata field values ​​are expanded and moved to corresponding positions, thereby becoming the plurality of second SR group representations. For example, Figures 6F1 to 6F4An animation is shown in which corresponding tags among multiple tags in multiple second metadata field values ​​(e.g., "Album Name 1," "Album Name 2," "Album Name 3," and "Album Name 4") are expanded and moved to corresponding positions to become multiple second SR group representations 642A-642D.

[0219] In various embodiments, replacing a particular first SR group representation of the plurality of first SR group representations with the plurality of second SR group representations includes stopping displaying other first SR group representations of the plurality of first SR group representations. Fig. 6E In the embodiment, when the artist name 3 group representation 641C is replaced by the plurality of second SR group representations 642A-642D, the other first SR group representations in the first SR group representations 641A, 641B and 641D are stopped from being displayed. In various embodiments, the other SR group representations in the plurality of SR group representations are displayed in a faded, shrunk or grayed state.

[0220] In various embodiments, method 1500 further includes detecting a second user input directed to a specific second SR group representation in the plurality of second SR group representations corresponding to a specific second group data item in the second group of data items, the specific second group data item in the second group of data items including a data item in the plurality of data items having a specific second metadata field value in a plurality of second metadata field values ​​for the second metadata field. Figure 5G In , the user selects Dave-text group representation 542D. For another example, in Figure 6G In the example, the user selects album name 4 groups represented by 642D.

[0221] In various embodiments, method 1500 also includes, in response to detecting a second user input, replacing a specific second SR group representation among the multiple second SR group representations with a plurality of SR item representations, the plurality of SR item representations corresponding to data items having a specific first metadata field value among a plurality of first metadata field values ​​for the first metadata field and a specific second metadata field value among a plurality of second metadata field values ​​for the second metadata field in the plurality of data items, respectively.

[0222] For example, in Figure 5H In response to Figure 5G In response to user input, a specific second SR group representation in the second SR group representation (e.g., Dave-text group representation 542D) is replaced with multiple SR item representations 543A-543D, which respectively correspond to data items in the multiple data items having a specific first metadata field value (e.g., “text”) among first metadata field values ​​for a first metadata field (e.g., document type metadata field) and a specific second metadata field value (e.g., “Dave”) among multiple second metadata field values ​​for a second metadata field (e.g., author metadata field).

[0223] For example, in Figure 6H In response to Figure 6G In response to user input, a specific second SR group representation in the second SR group representation (e.g., Album Name 4 group representation 642D) is replaced with multiple SR item representations 643A-643D, which respectively correspond to data items having a specific first metadata field value (e.g., “Artist Name 3”) among the first metadata field values ​​of the first metadata field (e.g., Artist metadata field) and a specific second metadata field value (e.g., “Album Name 4”) among the multiple data items having a second metadata field (e.g., Album metadata field).

[0224] In various embodiments, the method 1500 further includes detecting a third user input pointing to a specific SR item representation in the plurality of SR item representations corresponding to the specific data item in the data items. Figure 5H In , the user selects the third SR item corresponding to the document file named "three.txt" indicating 543C. Figure 6H , the user selects the fourth SR item representation 643D corresponding to the audio file named "Song Name 4".

[0225] In various implementations, method 1500 further includes, in response to detecting the third user input, opening a specific data item in the data items. Fig.5I In response to detecting Figure 5H In response to the user input in the SR volume environment 500, the SR volume environment 500 includes a text display window 544, which includes the content 545 of the document file named "three.txt" and a close enable indication 546, which closes the text display window 544 when selected via the user input. Figure 6H Based on the user input in , the device plays an audio file named "Song Name 4" through the speaker.

[0226] In various embodiments, method 1500 further includes detecting a third user input to close the second SR group representation while the second SR group representation is displayed. Figure 8C , the user input includes the user pinching (or collapsing) the plurality of second SR group representations 647A-647D with his left hand and his right hand.

[0227] In various implementations, method 1500 further includes, in response to detecting a third user input, replacing the plurality of second SR group representations with the plurality of first SR group representations. Fig. 8E , multiple second SR group representations 647A-647D are replaced with first SR group representations 641A-641D.

[0228] Fig.16 1 shows a flowchart representation of a method 1600 for playing two audio files according to some embodiments. In various embodiments, the method 1600 is performed by a device (e.g., Figure 3 In some embodiments, the method 1600 is performed by a processing logic component, including hardware, firmware, software, or a combination thereof. In some embodiments, the method 1600 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).

[0229] The method 1600 begins in block 1610, where a device displays, on a display, a plurality of SR objects at respective locations in a SR volume space, each SR object in the plurality of SR objects being associated with a respective metadata field value of a metadata field of at least one audio file in a plurality of audio files. Fig.11A , a plurality of SR objects (e.g., first SR group representations 741A-741C) are displayed at corresponding locations in the SR volume space 700. Each of the plurality of objects is associated with a corresponding metadata field value (e.g., “Artist Name 1”, “Artist Name 2”, and “Artist Name 3”) of a metadata field (e.g., an artist metadata field) of at least one of the plurality of audio files.

[0230] The method 1600 continues at block 1620, where the device determines a first distance between the user position in the SR volumetric space and a first position in the SR volumetric space of a first SR object among the plurality of SR objects, where the first SR object is associated with a first metadata value. In various embodiments, the user position corresponds to a position of a user's body in the SR environment, a position of a user's hand in the SR environment, or a position of a user's gaze point in the SR environment. For example, Fig.11A , the volume space includes a first SR object (eg, Artist Name 1 group representation 741A) associated with a first metadata value (eg, “Artist Name 1”).

[0231] The method 1600 continues at block 1630, where the device determines a second distance between the user position in the SR volume space and a second position in the SR volume space of a second SR object of the plurality of SR objects, where the second SR object is associated with a second metadata value. Fig.11A , the volumetric space includes a second SR object (eg, Artist Name 2 Group Representation 741B) associated with a second metadata value (eg, “Artist Name 2”).

[0232] Method 1600 continues at block 1640, where the device selects a first audio file of the plurality of audio files that has a metadata field that includes a first metadata field value. Fig.11A In the embodiment, the device selects an audio file that includes "Artist Name 1" in the artist metadata field. In various embodiments, the first audio file is randomly selected. In various embodiments, the first audio file is selected based on the second audio file.

[0233] The method 1600 continues in block 1650, where the device selects a second audio file of the plurality of audio files having a metadata field including a second metadata field value based on the first audio file. In various embodiments, the second audio file is selected to reduce inconsistencies between the first audio file and the second audio file. In various embodiments, the second audio file is based on audio data of the first audio file and the second audio file. Thus, in various embodiments, the selection of the second audio file is based on one or more of a pitch of the first audio file and the second audio file, a melody of the first audio file and the second audio file, or a beat of the first audio file and the second audio file. In various embodiments, the second audio file is based on metadata of the first audio file and the second audio file, such as a song name or a genre.

[0234] In various embodiments, selecting the second audio file is based on instruments of the first audio file and the second audio file. In various embodiments, selecting the second audio file is based on samples common to the first audio file and the second audio file. In various embodiments, selecting the second audio file is further based on random selection.

[0235] Method 1600 continues in box 1660, where the device includes playing a first audio file at a first volume in a direction starting from a first position based on a first distance via the directional speaker system, and simultaneously playing a second audio file at a second volume in a direction starting from a second position based on a second distance via the directional speaker system.

[0236] In various embodiments, the second audio file is processed or modified to reduce the inconsistency between the first audio file and the second audio file. For example, in various embodiments, the pitch and / or beat of the second audio file is changed to better match the pitch and / or beat of the first audio file. Therefore, in various embodiments, playing the second audio file is based on the first audio file.

[0237] In various embodiments, method 1600 also includes determining a distance between a user position in the SR volume space and a third position in the SR volume space of a third SR object among a plurality of SR objects, wherein the first SR object is associated with a third metadata field value; selecting a third audio file among the plurality of audio files having a metadata field including the third metadata field value based on the first audio file and the second audio file; and playing the third audio file simultaneously with playing the first audio file and the second audio file at a third volume in a direction starting from the third position based on the third distance via a directional speaker system. In various embodiments, the third audio file is selected and played in response to determining that the third distance is below a threshold. Thus, in various embodiments, SR objects far from the user position do not emit sound (or even faint sound).

[0238] In various embodiments, method 1600 further includes detecting a change in the user position in the SR volume space to a new user position, determining an updated first distance between the new user position and a first position of a first SR object, determining an updated second distance between the new user position and the second position of the second SR object, and changing the first volume and the second volume based on the updated first distance and the updated second distance. Fig. 11B In , the user moves to a new position, thereby changing the distance between the user and the first SR volume representation.

[0239] Fig.17 1 is a flowchart representation of a method 1700 for crossfading between two audio files according to some embodiments. In various embodiments, the method 1700 is performed by a device (e.g., Figure 3 In some embodiments, the method 1700 is performed by a processing logic component, including hardware, firmware, software, or a combination thereof. In some embodiments, the method 1700 is performed by a processor that executes instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., a memory).

[0240] Method 1700 continues at block 1710, where the device plays, via a speaker, a first audio file of the plurality of audio files having a metadata field including a first metadata field value. Fig. 12A , in response to the user viewing the Artist Name 4 group representation 841D, the device plays a first audio file from the plurality of audio files having an artist metadata field including a value of “Artist Name 4”.

[0241] Method 1700 continues at block 1720, where the device receives an indication of a second metadata field value while playing the first audio file. Fig. 12B , the user begins looking at Artist Name 3 group representation 841C which indicates the second metadata field value "Artist Name 3".

[0242] The method 1700 continues in block 1730, where the device selects a second audio file from among the plurality of audio files having a metadata field including a second metadata field value based on the first audio file. In various embodiments, the second audio file is selected to reduce inconsistencies between the first audio file and the second audio file. In various embodiments, the second audio file is based on audio data of the first audio file and the second audio file. Thus, in various embodiments, the selection of the second audio file is based on one or more of a pitch of the first audio file and the second audio file, a melody of the first audio file and the second audio file, or a beat of the first audio file and the second audio file. In various embodiments, the second audio file is based on metadata of the first audio file and the second audio file, such as a song name or a genre.

[0243] In various embodiments, selecting the second audio file is based on instruments of the first audio file and the second audio file. In various embodiments, selecting the second audio file is based on samples common to the first audio file and the second audio file. In various embodiments, selecting the second audio file is further based on random selection.

[0244] The method 1700 continues at block 1740 where, in response to receiving an indication of a second metadata field value, the device crossfades from the first audio file to the second audio file. FIG. 12A to FIG. 12C , the device crossfades from Artist Name 4 songs to Artist Name 3 songs (selected based on Artist Name 4 songs). In various implementations, the crossfade includes playing the first audio file and the second audio file simultaneously, while reducing the volume of the first audio file and increasing the volume of the second audio file.

[0245] Although various aspects of the embodiments within the scope of the appended claims are described above, it should be apparent that the various features of the above-described embodiments may be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein may be implemented independently of any other aspects, and two or more of these aspects may be combined in various ways. For example, any number of aspects set forth herein may be used to implement a device and / or a method may be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions may be used to implement such a device and / or such a method may be practiced.

[0246] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0247] The terms used herein are only for describing specific embodiments and are not intended to limit the claims. As used in the description of this embodiment and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.

[0248] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method comprising: At a device comprising a processor, a non-transitory memory, and a display: identifying a plurality of data items, each data item in the plurality of data items having a first metadata field and a second metadata field different from the first metadata field; displaying, on the display, a volumetric environment comprising a first plurality of synthetic reality (SR) objects corresponding to a first plurality of data items of the identified plurality of data items, wherein each data item of the first plurality of data items comprises the first metadata field having a corresponding first metadata field value of a plurality of first metadata field values, and wherein each SR object of the first plurality of SR objects comprises a different graphical indicia for representing a corresponding first metadata field value of the plurality of first metadata field values; detecting a first user input indicating the second metadata field; as well as In response to detecting the first user input, the first plurality of SR objects in the volumetric environment are replaced with a second plurality of SR objects corresponding to a second plurality of data items among the identified plurality of data items, wherein each of the second plurality of data items includes the second metadata field having a corresponding second metadata field value among a plurality of second metadata field values, and wherein each of the second plurality of SR objects includes a different graphical mark for representing a corresponding second metadata field value among the plurality of second metadata field values.

2. The method according to claim 1, further comprising: detecting a second user input directed to a first SR object of the second plurality of SR objects corresponding to a first data item of the second plurality of data items, wherein the first data item includes the second metadata field having a particular second metadata field value of the plurality of second metadata field values; as well as In response to detecting the second user input, the second user input being directed to the first SR object: replacing the first SR object in the volumetric environment with a third plurality of SR objects corresponding to a third plurality of data items, each data item in the third plurality of data items including the second metadata field having the particular second metadata field value in the plurality of second metadata field values; and A balanced display of the second plurality of SR objects excluding the first SR object is maintained in the volumetric environment.

3. The method of claim 1 , wherein replacing the first plurality of SR objects in the volumetric environment with the second plurality of SR objects comprises displaying an animation, wherein the first plurality of SR objects are replaced by a plurality of SR object representations, and the plurality of SR object representations are rearranged in the volumetric environment and replaced with the second plurality of SR objects.

4. The method according to claim 1, further comprising: A first metadata field is selected from a plurality of metadata fields as the first metadata field based on default metadata fields for the identified types of the plurality of data items.

5. The method according to claim 1, further comprising: Based on the number of data items in the identified plurality of data items, a first metadata field is selected from a plurality of metadata fields as the first metadata field.

6. The method according to claim 1, further comprising: Based on the context information, a first metadata field is selected from a plurality of metadata fields as the first metadata field.

7. The method of claim 1, wherein detecting the first user input comprises detecting a speech input from a user of the device.

8. The method of claim 1, wherein identifying the plurality of data items comprises, for each of the plurality of data items, populating at least one of the first metadata field with a first metadata field value or the second metadata field with a second metadata field value.

9. The method of claim 1, wherein each SR object of the first plurality of SR objects is displayed together with a label of the corresponding first metadata field value of the plurality of first metadata field values.

10. The method of claim 1, wherein displaying the volumetric environment comprises: displaying the first plurality of SR objects at a first plurality of locations in the volumetric environment; detecting a change in at least one of a user position or a user orientation; as well as In response to detecting the change in at least one of the user position or the user orientation, displaying the first plurality of SR objects at a second plurality of positions in the volumetric environment that are different from the first plurality of positions based on the change in at least one of the user position or the user orientation.

11. A device, comprising: monitor; one or more processors; non-transitory memory; as well as one or more programs stored in the non-transitory memory, wherein the one or more programs include instructions that, when executed by the one or more processors, cause the device to: identifying a plurality of data items, each data item in the plurality of data items having a first metadata field and a second metadata field different from the first metadata field; displaying, on the display, a volumetric environment comprising a first plurality of synthetic reality (SR) objects corresponding to a first plurality of data items of the identified plurality of data items, wherein each data item of the first plurality of data items comprises the first metadata field having a corresponding first metadata field value of a plurality of first metadata field values, and wherein each SR object of the first plurality of SR objects comprises a different graphical indicia representing a corresponding first metadata field value of the plurality of first metadata field values; detecting a first user input indicating the second metadata field; as well as In response to detecting the first user input, the first plurality of SR objects in the volumetric environment are replaced with a second plurality of SR objects corresponding to a second plurality of data items among the identified plurality of data items, wherein each of the second plurality of data items includes the second metadata field having a corresponding second metadata field value among a plurality of second metadata field values, and wherein each of the second plurality of SR objects includes a different graphical mark for representing a corresponding second metadata field value among the plurality of second metadata field values.

12. The method of claim 11, wherein replacing the first plurality of SR objects in the volumetric environment with the second plurality of SR objects comprises displaying an animation, wherein the first plurality of SR objects are replaced by a plurality of SR object representations, and the plurality of SR object representations are rearranged in the volumetric environment and replaced with the second plurality of SR objects.

13. The method of claim 11, wherein the instructions further cause the device to: A first metadata field is selected from a plurality of metadata fields as the first metadata field based on default metadata fields for the identified types of the plurality of data items.

14. The method of claim 11, wherein the instructions further cause the device to: Based on the number of data items in the identified plurality of data items, a first metadata field is selected from a plurality of metadata fields as the first metadata field.

15. The method of claim 11, wherein the instructions further cause the device to: Based on the context information, a first metadata field is selected from a plurality of metadata fields as the first metadata field.

16. A non-transitory computer-readable storage medium having instructions stored therein, which, when executed by one or more processors of a device having a display, cause the device to: identifying a plurality of data items, each data item in the plurality of data items having a first metadata field and a second metadata field different from the first metadata field; displaying, on the display, a volumetric environment comprising a first plurality of synthetic reality (SR) objects corresponding to a first plurality of data items of the identified plurality of data items, wherein each data item of the first plurality of data items comprises the first metadata field having a corresponding first metadata field value of a plurality of first metadata field values, and wherein each SR object of the first plurality of SR objects comprises a different graphical indicia for representing a corresponding first metadata field value of the plurality of first metadata field values; detecting a first user input indicating the second metadata field; as well as In response to detecting the first user input, the first plurality of SR objects in the volumetric environment are replaced with a second plurality of SR objects corresponding to a second plurality of data items among the identified plurality of data items, wherein each of the second plurality of data items includes the second metadata field having a corresponding second metadata field value among a plurality of second metadata field values, and wherein each of the first plurality of SR objects includes a different graphical mark for representing a corresponding first metadata field value among the plurality of first metadata field values.

17. A non-transitory computer-readable storage medium according to claim 16, wherein replacing the first plurality of SR objects in the volumetric environment with the second plurality of SR objects includes displaying an animation, wherein the first plurality of SR objects are replaced by a plurality of SR object representations, and the plurality of SR object representations are rearranged in the volumetric environment and replaced with the second plurality of SR objects.

18. The non-transitory computer-readable storage medium of claim 16, wherein the instructions further cause the device to: A first metadata field is selected from a plurality of metadata fields as the first metadata field based on default metadata fields for the identified types of the plurality of data items.

19. The non-transitory computer-readable storage medium of claim 16, wherein the instructions further cause the device to: Based on the number of data items in the identified plurality of data items, a first metadata field is selected from a plurality of metadata fields as the first metadata field.

20. The non-transitory computer-readable storage medium of claim 16, wherein the instructions further cause the device to: Based on the context information, a first metadata field is selected from a plurality of metadata fields as the first metadata field.

Images