Multi-user experience coordination system

Abstract

Systems and techniques are provided for coordinating multi-user experiences. For example, a process may include obtaining a plurality of settings associated with a plurality of multi-user experience participants. The plurality of settings includes one or more arbitration settings and one or more non-arbitration settings. The process may include arbitrating, by a setting arbitration engine, the one or more arbitration settings to generate one or more adjusted settings for each arbitration setting. The process may include generating, by an experience adaptation engine, an adapted multi-user experience, wherein the adapted multi-user experience is configured to enforce the one or more adjusted settings for each arbitration setting.

Description

Technical Field

[0001] This disclosure relates to multi-user experiences. In some examples, aspects of this disclosure relate to systems and techniques for multi-user experience management, which can coordinate between different experience settings for multi-user experience participants. Background Technology

[0002] Extended reality (XR) systems (e.g., virtual reality, augmented reality, mixed reality) can provide users with virtual experiences by immersing them in a completely virtual environment (composed of virtual content), and / or can provide users with augmented or mixed reality experiences by combining the real world or physical environment with a virtual environment.

[0003] An example use case for providing users with XR content in the form of virtual reality, augmented reality, or mixed reality is presenting users with a "metaverse" experience. A metaverse is essentially a virtual universe that includes one or more three-dimensional (3D) virtual worlds. For example, a metaverse virtual environment can allow users to virtually interact with other users (e.g., in a social setting, in a virtual meeting, etc.), virtually purchase goods, services, property, or other items, play computer games, and / or experience other services.

[0004] Machine learning models (e.g., deep learning models such as neural networks) can be used to perform a wide variety of tasks, including natural language processing (NLP), image processing, audio processing, depth estimation, detection and / or recognition (e.g., scene or object detection and / or recognition), pose estimation, image reconstruction, classification, 3D modeling, dense regression tasks, data compression and / or decompression, image processing, and more. Machine learning models can be general-purpose and achieve high-quality results across a wide range of tasks. Summary of the Invention

[0005] In some examples, systems and technologies for reconciling multiple user experiences are described. According to at least one example, a method for reconciling multiple user experiences is provided. The method includes: obtaining multiple settings associated with multiple multi-user experience participants, wherein the multiple settings include one or more arbitration settings and one or more non-arbitration settings; arbitrating the one or more arbitration settings by a settings arbitration engine to generate one or more adapted settings for each arbitration setting; and generating an adapted multi-user experience by an experience adaptation engine, wherein the adapted multi-user experience is configured to enforce the one or more adapted settings for each arbitration setting.

[0006] In another example, an apparatus for coordinating multiple user experiences is provided, the apparatus including at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: obtain multiple settings associated with multiple multi-user experience participants, wherein the multiple settings include one or more arbitration settings and one or more non-arbitration settings; arbitrate the one or more arbitration settings by a settings arbitration engine to generate one or more adapted settings for each arbitration setting; and generate an adapted multi-user experience by an experience adaptation engine, wherein the adapted multi-user experience is configured to enforce the one or more adapted settings for each arbitration setting.

[0007] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: obtain multiple settings associated with multiple multi-user experience participants, wherein the multiple settings include one or more arbitration settings and one or more non-arbitration settings; arbitrate the one or more arbitration settings by a settings arbitration engine to generate one or more adapted settings for each arbitration setting; and generate an adapted multi-user experience by an experience adaptation engine, wherein the adapted multi-user experience is configured to enforce the one or more adapted settings for each arbitration setting.

[0008] In another example, an apparatus for coordinating multiple user experiences is provided. The apparatus includes: components for obtaining multiple settings associated with multiple multi-user experience participants, wherein the multiple settings include one or more arbitration settings and one or more non-arbitration settings; components for arbitrating the one or more arbitration settings by a settings arbitration engine to generate one or more adapted settings for each arbitration setting; and components for generating adapted multi-user experiences by an experience adaptation engine, wherein the adapted multi-user experiences are configured to enforce the one or more adapted settings for each arbitration setting.

[0009] In some aspects, one or more of the devices described above are, are part of, or include the following: mobile devices (e.g., mobile phones or so-called "smartphones" or other mobile devices), wearable devices, extended reality devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), personal computers, laptop computers, server computers, vehicles (e.g., computing devices of vehicles), or other devices. In some aspects, a device includes one or more cameras for capturing one or more images. In some aspects, the device includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more sensors. In some cases, the one or more sensors may be used to determine the position and / or orientation of the device, the state of the device, and / or for other purposes.

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to define the scope of the claimed subject matter. This subject matter should be understood with reference to the appropriate portions of the entire specification, any or all drawings, and each claim.

[0011] The foregoing, as well as other features and embodiments, will become more apparent from the following description, claims and drawings. Attached Figure Description

[0012] The exemplary embodiments of this application are described in detail below with reference to the following figures:

[0013] Figure 1 This is a diagram illustrating, according to some examples of the present disclosure, exemplary relationships between machine learning tasks and various types of neural networks;

[0014] Figure 2 This is a block diagram illustrating an example of a multi-user experience coordination system according to some examples of this disclosure;

[0015] Figure 3 These are illustrations of example arbitration preferences and non-arbitration preferences based on some examples of this disclosure;

[0016] Figure 4A These are illustrations illustrating, according to some examples of this disclosure, the transition from an arbitration setting to a non-arbitration setting;

[0017] Figure 4B These are illustrations illustrating, according to some examples of this disclosure, the conversion from a non-arbitration setting to an arbitration setting;

[0018] Figure 5This is a flowchart illustrating an example of a content adaptation process according to some examples of this disclosure;

[0019] Figure 6 This is a block diagram illustrating examples of deep learning networks according to some examples of this disclosure;

[0020] Figure 7 This is a block diagram illustrating examples of convolutional neural networks according to some examples of this disclosure;

[0021] Figure 8 This is a diagram illustrating an example of a computing system used to implement some of the aspects described in this article. Detailed Implementation

[0022] Certain aspects and embodiments of this disclosure are provided below. Some of these aspects and embodiments may be applied independently, and some may be combined, as will be apparent to those skilled in the art. Specific details are set forth in the following description for purposes of explanation in order to provide a thorough understanding of the various embodiments of this application. However, it will be apparent, however, that the various embodiments may be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0023] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the scope of this application as set forth in the appended claims.

[0024] As previously noted, extended reality (XR) systems or devices can provide users with XR experiences (e.g., for fully immersive experiences) by presenting virtual content to them and / or combine a view of a real-world or physical environment with a display of a virtual environment (composed of virtual content). A real-world environment may include real-world objects (also called physical objects), such as people, vehicles, buildings, tables, chairs, and / or other real-world or physical objects. As used herein, the terms XR system and XR device are used interchangeably. Examples of XR systems or devices include head-mounted displays (HMDs), smart glasses (e.g., AR glasses, MR glasses, etc.), and more.

[0025] XR systems may include VR systems that facilitate interaction with virtual reality (VR) environments, AR systems that facilitate interaction with augmented reality (AR) environments, MR systems that facilitate interaction with mixed reality (MR) environments, and / or other XR systems. For example, VR provides a fully immersive experience in a three-dimensional (3D) computer-generated VR environment or video that depicts a virtual version of a real-world environment. VR content may, in some cases, include VR videos that can be captured and rendered at very high quality, potentially providing a truly immersive virtual reality experience. Virtual reality applications may include games, training, education, sports videos, online shopping, and more. VR content may be rendered and displayed using VR systems or devices such as VR HMDs or other VR headsets that completely cover the user's eyes during the VR experience.

[0026] AR is a technology that delivers virtual or computer-generated content (referred to as AR content) onto a user's view of a physical, real-world scene or environment. AR content can include any virtual content, such as video, images, graphic content, location data (e.g., Global Positioning System (GPS) data or other location data), sound, any combination thereof, and / or other augmenting content. AR systems are designed to enhance (or augment) rather than replace a person's current perception of reality. For example, a user may see a real, stationary or moving physical object through an AR device display, but the user's visual perception of the physical object can be enhanced or augmented by a virtual image of that object (e.g., a real-world car replaced by a virtual image of DeLorean), by AR content added to the physical object (e.g., virtual wings added to a live animal), by AR content displayed relative to the physical object (e.g., virtual information displayed near a sign on a building, a virtual coffee cup virtually anchored to one or more images of a real-world table (e.g., placed on top of that real-world table), and / or by displaying other types of AR content. Various types of AR systems can be used for games, entertainment, and / or other applications.

[0027] MR technology can combine aspects of VR and AR to provide users with an immersive experience. For example, in an MR environment, real-world and computer-generated objects can interact (e.g., real people can interact with virtual people as if the virtual people were real people).

[0028] XR environments can interact in a way that appears realistic or physically real. As a user experiencing an XR environment (e.g., an immersive VR environment) moves in the real world, the rendered virtual content (e.g., images rendered within the virtual environment of a VR experience) changes, giving the user the perception that they are moving within the XR environment. For example, a user can turn left or right, look up or down, and / or move forward or backward, thus changing their viewpoint of the XR environment. The XR content presented to the user can change accordingly, making the user's experience in the XR environment as seamless as in the real world.

[0029] In some cases, XR systems can match the relative pose and movement of objects and devices in the physical world. For example, an XR system can use tracking information to calculate the relative pose of features of devices, objects, and / or the real-world environment in order to match the relative positioning and movement of devices, objects, and / or the real-world environment. In some cases, an XR system can use the pose and movement of one or more devices, objects, and / or the real-world environment to render content in a convincing manner relative to the real-world environment. Relative pose information can be used to match virtual content with motion perceived by the user and the spatiotemporal state of devices, objects, and the real-world environment. In some cases, an XR system can track user body parts (e.g., the user's hands and / or fingertips) to allow the user to interact with items in the virtual content.

[0030] XR systems or devices can facilitate interaction with different types of XR environments (e.g., users can use XR systems or devices to interact with XR environments). An example of an XR environment is a metaverse virtual environment. Users can participate in one or more virtual sessions with other users by virtually interacting with them in a metaverse virtual environment (e.g., in a social environment, in a virtual meeting, etc.), virtually purchasing items (e.g., goods, services, property, etc.), virtually playing computer games, and / or experiencing other services. In an exemplary example, virtual sessions provided by an XR system may include 3D collaborative virtual environments for user groups. Users can interact with each other via virtual representations of themselves within the virtual environment. When interacting with virtual representations of other users, users can experience the virtual environment visually, auditorily, tactilely, or otherwise.

[0031] A user's virtual representation can be used to represent the user in a virtual environment. This virtual representation is also referred to herein as an avatar. An avatar representing a user can mimic the user's appearance, movements, habits, and / or other characteristics. The virtual representation or avatar can be generated / animated in real-time based on captured input from the user's device. Avatars can range from a basic synthetic 3D representation of the user to more realistic representations. In some examples, a user might expect an avatar representing a person in a virtual environment to appear as a digital twin of the user. In any virtual environment, efficiently generating high-quality avatars with low latency (e.g., realistically representing a person's appearance, movements, etc.) is important for XR systems. Effectively rendering audio to enhance the XR experience is also important for XR systems.

[0032] For example, in the 3D collaborative virtual environment example above, XR system users from a user group can see virtual representations (avatars) of other users sitting at specific locations in a virtual table or virtual room. The user's virtual representation and the background of the virtual environment should be displayed realistically (e.g., as if the users were sitting together in the real world). As the user moves in the real world, their head, body, arms, and hands can be animated. Audio may need to be spatially rendered or can be rendered in mono. Latency during the rendering and animation of virtual representations should be minimized to maintain a high-quality user experience.

[0033] Machine learning systems (e.g., deep neural network systems or models) can be used to perform a variety of tasks, such as, but not limited to, detection and / or recognition (e.g., scene or object detection and / or recognition, face detection and / or recognition, etc.), depth estimation, pose estimation, image reconstruction, classification, 3D modeling, dense regression tasks, data compression and / or decompression, audio processing, and image processing, etc. Furthermore, machine learning models can be general-purpose and achieve high-quality results across a wide range of tasks.

[0034] There are different types of neural networks, such as deep generative neural network models (e.g., generative pre-trained transformer (GPT), generative adversarial network (GAN)), recurrent neural network (RNN) models, multilayer perceptron (MLP) neural network models, convolutional neural network (CNN) models, and so on.

[0035] Figure 1 This is a diagram 100 illustrating example relationships between machine learning tasks and various types of neural networks. Figure 1In the example, the largest ellipse represents the machine learning system category referred to as generative model 102. As used herein, the term "generative model 102" refers to a model capable of generating new data examples. In some cases, generative model 102 can be implemented using different machine learning architectures. For example, as noted above, GAN and GPT are examples of deep generative neural network models. There exist machine learning system categories other than generative model 102, such as discriminative models. As used herein, discriminative models are models that distinguish between different types of data examples.

[0036] Figure 1 This illustrates another large ellipse representing the neural network architecture of the transformer 104. For example... Figure 1 As illustrated, transformer 104 can be used as a generative model 102 (e.g., for performing generative tasks) and / or a non-generative model (e.g., a discriminative model). Generally, transformer 104 is a deep learning model. Transformers typically perform self-attention (e.g., using at least one self-attention layer), differently weighting the importance of different parts of the input data (including recursive output). Transformers can be used in many contexts, including natural language processing (NLP) 110, image processing 120, or audio processing 130. Like recurrent neural networks (RNNs), transformers are designed to process sequential input data, such as natural language, for applications such as translation and text summarization. However, unlike RNNs, transformers process the entire input at once. The attention mechanism provides context for any localization in the input sequence. For example, if the input data is a natural language sentence, the transformer does not have to process one word at a time. This allows for greater parallelization than RNNs and thus reduces training time. Transformers are more suitable for parallelization than RNN models, allowing training on larger datasets.

[0037] like Figure 1 As illustrated, NLP may include both Natural Language Understanding (NLU) 112 and Natural Language Generation (NLG) 114. NLU 112 refers to understanding the meaning of written and / or spoken language (e.g., text, speech, or a combination thereof). Examples of NLU 112 include text reasoning or email classification. NLG 114 refers to the task of generating written and / or spoken language (e.g., text, speech, or a combination thereof) from structured data, unstructured data, or a combination thereof. Examples of NLG 114 include query-based summarization, story generation, news summarization, conversational artificial intelligence (AI), autocomplete systems, or a combination thereof. In some examples, an NLP system may include a combination of NLU 112 and NLG 114, such as question answering, explanation followed by summarization of content (e.g., news articles or stories), or a combination thereof. In some examples, NLG 114 may include an NLG based on transformer 104, such as... Figure 1exemplified.

[0038] In some cases, image processing 120 may also include understanding and generative aspects. For example, such as Figure 1 As illustrated, image processing 120 includes image understanding (e.g., computer vision (CV) 122) and image generation 124. In one exemplary example, image processing 120 can be used to generate images of virtual environments or avatars in XR environments. Image processing 120 can include processing of single images as well as sequences of images (e.g., consecutive images in a video).

[0039] In another example, audio processing 130 may also include understanding and generation aspects. For example, such as... Figure 1 As illustrated, audio processing 130 includes audio understanding 132 and audio generation 134. For example, audio understanding 132 can be used to interpret audio data waveforms into single words. Examples of audio generation 134 may include synthesizing speech (e.g., text-to-speech conversion), generating music, ambient sounds, and / or sound effects, etc.

[0040] In some cases, a multimodal model (not shown) can combine and / or coordinate the functionality between two or more different tasks. For example, a multimodal model can combine any combination of NLP 110, image processing 120, and / or audio processing 130. In one exemplary example, the multimodal model can receive audio data containing a query, perform audio understanding 132 to generate a sequence of words (e.g., text) representing the query. In some examples, the sequence of words generated by audio understanding 132 can be input to NLP 110, which can utilize NLU 112 to interpret the query and utilize NLG 114 to generate an appropriate response. In some cases, a text response can be output from NLG 114. In some implementations, audio generation 134 can convert the text response output from NLG 114 into an audio response (e.g., synthesized speech). In some implementations, image generation 124 can be used to generate an avatar (e.g., a 2D model or a 3D model) that can be displayed and coordinated with the output of the audio response generated by audio generation 134.

[0041] It should be understood that Figure 1The examples provided are not intended to be limiting and are for illustrative purposes only. Other types of machine learning models, neural network categories, neural network architectures, and / or any combination thereof not described herein may be used without departing from the scope of this disclosure. Systems and techniques are required to coordinate multi-user experiences. In some cases, the experience of different participants in a multi-user experience environment (e.g., an XR environment) may differ based on settings chosen by each individual user. However, in some cases, coordinating multi-user experience environments for different users can benefit the quality of the multi-user experience. For example, for a collaborative meeting in a multi-user experience environment, it may be beneficial to arrange each participant in the same relative position to each other. For example, when a first user speaks to a second user, if the relative posture of the first and second users is different in the XR environment experienced by a third user, it may be unclear whether the first user is speaking to the second user. For example, the first user may appear to be speaking to the third user. Such inconsistencies can lead to confusion. However, in some cases, some inconsistencies between multi-user experience environments can have a positive impact on the multi-user experience of the individual participants. For example, different participants may experience different ambient light settings, different ambient sound level settings, one or more other settings, and / or any combination thereof. In some cases, differences in setup can make each participant more interested in or more engaged with the multi-user experience.

[0042] This document describes systems and techniques for coordinating multi-user experiences. For example, in some cases, the systems and techniques described herein can divide an aggregated setting of a group of participants into arbitrated and non-arbitrated settings. In some cases, arbitrated settings may include preferences that require consistency among all multi-user experience participants. For example, the number and / or location or seating in a meeting, and / or the location of a specific participant in the meeting, may be arbitrated. In some examples, aspects of the multi-user experience environment associated with non-arbitrated settings may differ for different participants. For example, lighting conditions or ambient noise settings, as described above, may be non-arbitrated settings. In some cases, the division between arbitrated and non-arbitrated settings may vary depending on the multi-user experience. For example, a multi-user experience involving many participants with strong setting preferences may include more arbitrated preferences than a multi-user experience involving participants with no strong preferences.

[0043] In some cases, consistency among arbitration preference settings can be made common by enforcing mandatory settings. In other cases, the arbitration engine can determine one or more potential settings for each arbitration setting. For example, participant settings for a specific arbitration setting across all participants can overlap, making one or more potential settings of that arbitration setting available as adjusted settings for all participants. In some cases, two or more potential settings can be combined into a single adjusted setting, which specifies the percentage used to merge the two or more potential settings. For example, a furniture style arbitration setting could indicate that modern furniture and antique furniture are potential settings for all participants. In some cases, the arbitration engine can output two potential settings. In some examples, the arbitration engine can combine two potential settings into one adjusted setting.

[0044] These systems and technologies can generate adapted experiences by allowing participants to use their own settings for non-arbitration settings and enforcing common settings used for arbitration settings. In some cases, these systems and technologies can monitor user engagement with the adapted experience. In some cases, participant settings can be updated based on user engagement monitoring. In some cases, these systems and technologies can leverage machine learning models to generate adapted experiences (e.g., through generative models), monitor participant engagement, and / or update participant settings.

[0045] The various aspects of the technology described in this article will be discussed below with respect to the accompanying figures. Figure 2 An example multi-user experience coordination system 200 is illustrated. As illustrated, the multi-user experience coordination system 200 may include an application engine 210, a settings partitioning engine 220, a settings arbitration engine 230, an experience adaptation engine 240, an adapted experience 250, and an adaptive settings adjustment engine 260. The multi-user experience coordination system 200 can obtain settings for different multi-user experience participants who may participate in a shared multi-user experience (e.g., a metaverse meeting). As used herein, a multi-user experience participant may refer to an individual user, device, user group, and / or any combination thereof.

[0046] In some implementations, the partitioning engine 220, application engine 210, setup arbitration engine 230, experience adaptation engine 240, adapted experience 250, and adaptive setup adjustment engine 260 may be part of the same computing device. For example, in some cases, the partitioning engine 220, application engine 210, setup arbitration engine 230, experience adaptation engine 240, adapted experience 250, and adaptive setup adjustment engine 260 may be integrated into an HMD, extended reality glasses, smartphone, laptop computer, tablet computer, gaming system, and / or any other computing device. However, in some implementations, the partitioning engine 220, application engine 210, setup arbitration engine 230, experience adaptation engine 240, adapted experience 250, and adaptive setup adjustment engine 260 may be part of two or more separate computing devices. For example, in some cases, some components of components 210-260 may be part of or implemented by one computing device, and the remaining components may be part of or implemented by one or more other computing devices.

[0047] In some cases, application engine 210 may include baseline experience 212 and / or experience parameters 214. In some aspects, application engine 210 may include XR applications. Example baseline experience 212 may include, but is not limited to, gaming sessions, conferences, art exhibitions, social gatherings, assemblies, any other type of experience, and / or any combination thereof. Example experience parameters 214 may include, but are not limited to, the number of participants, the experience owner (e.g., the conference creator or administrator), and the natural geography (e.g., in the Sahara Desert, in Southern California, etc.).

[0048] In some cases, each multi-user experience participant may provide one or more settings 201. In some cases, the settings 201 for multi-user experience participants may be explicitly selectable via a user interface (e.g., an HMD settings menu in an accompanying application and / or any other user interface). In some examples, the settings 201 for multi-user experience participants may be generated by one or more machine learning models. For example, a deep learning neural network may be trained to generate and / or update settings based on user feedback regarding different multi-user experiences. In some cases, a deep learning neural network may be trained to detect and / or interpret content interaction information. For example, content interaction information may include, but is not limited to, eye tracking, data shared by multiple connected sensors and / or devices (e.g., smartwatches, vehicles, laptops, mobile devices, HMDs, etc.). In some cases, the values ​​of one or more settings 201 may be suggested by an aggregated recommendation system (e.g., settings used by people like you, trending settings, celebrity settings, and / or any other settings).

[0049] In some examples, setting 201 may include one or more multi-user experience environment appearance settings. For example, multi-user experience environment appearance settings may include, but are not limited to, room decor (e.g., the number and / or location of furniture items (e.g., chairs, tables, etc.)), the number and / or location of virtual displays used for shared content, lighting conditions (e.g., the number and / or location of light sources, brightness and / or color), audio settings (e.g., background noise level, attenuation rate and / or echo behavior), music (e.g., music content, volume and / or location of virtual speakers), sky appearance (e.g., time of day, color, weather, number and / or location of stars, moon and / or planets), the number and / or location of non-player characters (e.g., animals, virtual vendors and / or virtual waiters) and / or non-participant character behavior and / or appearance.

[0050] In some cases, setting 201 may include one or more language settings. For example, language settings may include a preference for automatic translation into a language chosen by multiple user experience participants. In some cases, language settings may include a preference for speech presented in the original spoken language. In some cases, language settings may include explicit language filters.

[0051] In some examples, setting 201 may include one or more avatar settings. For example, avatar settings may include, but are not limited to, clothing, size, realism, or polygon count. For instance, a multi-user experience participant might want to be presented as a ten-foot-tall cartoon bear. As another example, a multi-user experience participant might choose business casual attire as their avatar setting for a business meeting.

[0052] In some aspects, setting 201 may include privacy settings. For example, privacy settings may include whether to use a pseudonym or legal name, whether to display accurate facial expressions or perform facial expression suppression, whether to use a real voice or a voice changer, and / or whether to use a certified avatar. In some cases, participants in a multi-user experience environment may register a certified avatar with an avatar registration service, which can be used when the multi-user experience requires the use of a certified avatar as a condition of participation. For example, a certified avatar may be needed in business negotiations to prevent multi-user experience participants from distorting their identities and compromising the integrity of the negotiations.

[0053] In some cases, setting 201 may include user interface settings. For example, user interface settings may include, but are not limited to, whether to display name tags, whether to display titles / roles (e.g., CEO, supervisor, moderator, administrator, etc.), and / or whether to provide prompts to identify speakers (e.g., arrows pointing to speakers, spotlights, color changes, or person effects, etc.). In some cases, multi-user experience participants may prefer to disable user interface elements to give the multi-user experience a more realistic look and feel. In some examples, multi-user experience participants may prefer to display user interface elements so that they can easily identify other multi-user experience participants in the multi-user experience environment and / or navigate the multi-user experience environment more easily.

[0054] In some implementations, setting 201 may include accessibility settings. For example, accessibility settings may include, but are not limited to, closed captions, colorblind color schemes, minimum audio volume, and / or minimum brightness. In some cases, the multi-user experience coordination system 200 may be configured to enforce the availability of accessibility settings within a multi-user experience.

[0055] In some aspects, setting 201 may include geo-specific settings. For example, recording sessions with a multi-user experience participant may be illegal based on their location. In some examples, setting 201 may include geo-specific settings that require consent before recording sessions with a multi-user experience participant.

[0056] Additional example setting 201 includes settings for the relative positions of specific multi-user experience participants. For example, parents may have the option to require their children to always remain within a specific distance of their parents in a multi-user experience environment. As another example, employees may have the option to sit near, opposite, or next to specific participants in a multi-user experience (e.g., friends, colleagues, potential business contacts).

[0057] In some cases, multi-user experience participants may automatically and / or manually assign importance weights to setting 201. For example, importance weights may include, but are not limited to, strictly required settings (e.g., no recording without consent), extremely important settings, moderately important settings, completely optional settings, any other importance weights, and / or any combination thereof. In some cases, one or more settings in setting 201 may include a range and / or group of related settings. In one exemplary example, a multi-user experience participant may choose setting 201 to allow the sky in the multi-user experience environment to be blue, red, or black instead of any other color. In another exemplary example, a multi-user experience participant may choose to include a setting in setting 201 that specifies the maximum polygon count used for the user avatar.

[0058] In some cases, one or more settings in setting 201 may be conditional (e.g., context-specific settings). In some cases, one or more settings in setting 201 may be conditional on hardware capabilities (e.g., battery life, processing power, any other hardware combination, and / or any combination thereof). In one exemplary example, a setting that allows for an unlimited polygon count may be used when a multi-user experience participant is using a state-of-the-art HMD to participate in the multi-user experience. In some cases, when a multi-user experience participant is using a device with limited capabilities to participate in the multi-user experience, the polygon count setting may be adjusted to match the capabilities of that device. In one exemplary example, multi-user experience participants may have different groups of settings 201 for different contexts. For example, multi-user experience participants may have a first setting for work meetings, a second setting for family and / or friends, and a third setting for multi-user experience participants with more than a certain number of participants.

[0059] like Figure 2 As illustrated, partition 205 indicates that setting 201 can be created separately from the setting arbitration provided by the multi-user experience coordination system 200. For example, setting 201 can be generated over time before the multi-user experience. In some cases, when the multi-user experience coordination system 200 is ready to perform setting partitioning and / or arbitration, the settings 201 of all multi-user experience participants for a particular multi-user experience can be aggregated into a set of settings 207.

[0060] As illustrated, the settings partitioning engine 220 may obtain a set of settings 207, baseline experience 212, and / or experience parameters 214 as input. In some cases, the settings partitioning engine 220 may divide the set of settings 207 into non-arbitration settings 222 and arbitration settings 224. As used herein, the term "arbitration setting" refers to any setting that requires consistency across all multi-user experience participants. In some cases, arbitration settings 224 may include mandatory settings (e.g., because mandatory settings require consistency). In one illustrative example, arbitration setting 224, also a mandatory setting, may require the sky to be blue for all participants. However, not all arbitration settings 224 require mandatory settings. In some examples, arbitration settings 224 can be kept consistent through agreement among multi-user experience participants. In one illustrative example, arbitration setting 224 may require all participants to agree on a consistent sky color.

[0061] In some cases, the settings partitioning engine 220 may initially partition the settings of a multi-user experience based on settings 201, baseline experiences 212, and / or experience parameters 214 for each multi-user experience participant. In some aspects, the settings partitioning engine 220 may identify arbitration settings 224 based on the device capabilities of the devices belonging to the multi-user experience participants. For example, if one or more devices do not support a polygon count higher than the maximum polygon count (e.g., based on the maximum polygon count of the worst-capable device participating in the multi-user experience, the mean device, the median device, any other measurement of device capabilities, and / or any combination thereof), then the polygon count may become the arbitration setting 224. In some cases, one or more arbitration settings 224 may be identified based on one or more baseline experiences 212 and / or experience parameters 214 from the settings partitioning engine 220. For example, experience parameters 214 may specify that a multi-user experience generated by a meeting scheduler used to schedule meetings between company employees needs to maintain consistency of one or more settings 201 across all multi-user experience participants. In one exemplary example, experience parameter 214 could specify that all employee avatars must wear business casual attire, use their legal names, comply with public privacy settings, any other specified settings, and / or any combination thereof. As another example, the baseline experience 212 for an art exhibition could allow the settings partitioning engine 220 to determine that the type of art displayed is arbitration setting 224.

[0062] Figure 3 Examples of initial arbitration settings 324 and non-arbitration settings 322 for a specific multi-user experience are illustrated. For illustrative purposes, the example arbitration settings 324 and non-arbitration settings 322 illustrate a limited number of settings. However, it should be understood that fewer, more, and / or different settings may be included in the arbitration settings 324 and non-arbitration settings 322 without departing from the scope of this disclosure.

[0063] return Figure 2 In some implementations, arbitration setting 224 may be determined by application engine 210 based on analysis of each user's preferences. For example, a first user's setting 201 may include a setting that all multi-user experiences they participate in must have the same sky color for all multi-user experience participants. As another example, a second user's setting 201 may instruct all multi-user experience participants to agree on common privacy settings for the multi-user experience. In some cases, application engine 210 may generate a superset of arbitration settings 224 identified based on a review of the preferences of all multi-user experience participants.

[0064] like Figure 2As illustrated, arbitration settings 224 from application engine 210 can be provided to setting arbitration engine 230 for arbitration. In some cases, setting arbitration engine 230 can arbitrate arbitration settings 224 based on different criteria. In some cases, based on arbitrating arbitration settings 224, setting arbitration engine 230 can output one or more potential settings for each arbitration setting 224 to experience adaptation engine 240. For example, one or more settings indicated as mandatory and / or specified by the meeting owner in baseline experience 212, experience parameters 214 can be resolved by imposing mandatory settings on all multi-user experience participants.

[0065] In some implementations, the arbitration engine 230 may be configured to arbitrate one or more arbitration settings 224 prior to the multi-user experience. For example, the arbitration engine 230 may determine potential settings for a particular arbitration setting 224 based on the expected multi-user experience participants (e.g., a list of invited participants). In some cases, the arbitration engine 230 may be configured to select among potential settings associated with the expected multi-user experience. For example, the arbitration engine 230 may apply potential settings from a first multi-user experience participant to an optional first arbitration setting. In some examples, the arbitration engine 230 may apply settings from a second multi-user experience participant to a second arbitration setting. In some aspects, the arbitration engine 230 may apply default settings to a third arbitration setting. For example, one or more multi-user experience participants may indicate that arbitration should be performed on a third arbitration setting, but without indicating a preference for the actual settings applied to the third arbitration setting, and the arbitration engine 230 may apply default settings to the third arbitration setting. In some cases, arbitrating settings used to generate multi-user experiences from a given set of settings (e.g., potential settings and / or default settings for expected multi-user experience participants) instead of introducing new settings (e.g., by combining settings, generating new settings, and / or any combination thereof) can reduce power consumption, memory usage, computational costs, and / or any combination thereof.

[0066] In some cases, one or more settings can be arbitrated by the settings arbitration engine 230 by dynamically applying one or more preferences of the current speaker. For example, in a conference environment, one or more audio settings can be set to match the preferences of the current speaker and can change dynamically as different multi-user experience participants speak. In an exemplary example, a multi-user experience may include consecutive presentations by two presenters. In some aspects, the first presenter may have one or more settings 201 that make ambient noise audible to all audience members. For example, the first presenter may believe that ambient sound makes the audience feel more like they are participating in a live experience. In some examples, the second presenter may have one or more settings 201 that disable ambient noise for all audience members. For example, the presenter may prefer not to be interrupted or distracted, and / or may prefer to speak at a low volume that might be inaudible in noisy ambient sound.

[0067] In some examples, the setting arbitration engine 230 can automatically arbitrate one or more settings by searching for settings that are suitable for all users. For example, sky color could be the arbitration setting 224 because a user insists that the sky color should reflect their geographical location at the time of day. In some cases, the setting arbitration engine 230 can determine mandatory settings that do not conflict among the remaining multi-user experience participants, so a setting of one multi-user experience participant can be used as a common setting for all multi-user experience participants. In some cases, the setting arbitration engine 230 can determine that two or more potential settings for a particular arbitration setting 224 are acceptable to all multi-user experience participants and output the two or more potential settings to the experience adaptation engine 240.

[0068] In some cases, the setting arbitration engine 230 may determine, for a specific arbitration setting 224, that two or more settings are acceptable to multiple user experience participants. In some cases, as a supplement or alternative to outputting two or more potential settings to the experience adaptation engine 240, the setting arbitration engine 230 may generate an adjusted setting by combining two or more potential settings and may provide that adjusted setting as output to the arbitration setting 224. For example, the setting arbitration engine 230 may generate a blending percentage of two or more potential settings for the arbitration setting 224 to generate the adjusted setting of the arbitration setting 224. In some cases, the blending percentage may be equal for each of the two or more potential settings. In some examples, the blending percentage may be weighted based on one or more weighting criteria. Example weighting criteria may include, but are not limited to, relative preference strength (e.g., strong preference / slight preference), the relative number of multiple user experience participants who prefer one potential setting relative to another, user engagement indicators (e.g., multiple user experience participants who appear to be less engaged may prioritize their preferences), any other weighting criteria, and / or any combination thereof.

[0069] like Figure 2 As illustrated, the experience adaptation engine 240 may receive non-arbitration settings 222 directly from the settings partitioning engine 220, and receive one or more potential settings for each arbitration setting 224 output by the settings arbitration engine 230. In some cases, the experience adaptation engine 240 may generate an adapted experience 250 based on the non-arbitration settings 222 and one or more potential settings from each arbitration setting 224 of the settings arbitration engine 230.

[0070] In some respects, the experience adaptation engine 240 can create an adapted experience 250 that allows individual multi-user experience participants to use their own settings 201 for any non-arbitration settings 222. In some cases, for arbitration settings 224, the experience adaptation engine 240 can enforce settings from the settings arbitration engine 230 for all multi-user experience participants.

[0071] In some cases, the Experience Adaptation Engine 240 can directly manipulate the multi-user experience to meet settings from the Settings Arbitration Engine 230. For example, but not limited to, the Experience Adaptation Engine 240 can adjust light levels, music volume, the number of non-participant characters, enable or disable automatic language translation, any other settings adjustments, and / or any combination thereof. In some cases, the Experience Adaptation Engine 240 can select content for the multi-user experience from a database based on potential settings from a specific arbitration setting 224 from the Settings Arbitration Engine 230. In some cases, when two or more potential settings for a specific arbitration setting 224 exist, a mix of content elements that satisfy two or more potential settings can be selected. For example, if the two potential settings for furniture style are modern and antique, the Experience Adaptation Engine 240 can select a mix of modern and antique furniture.

[0072] In some cases, the experience adaptation engine 240 can divide the adapted experience 250 into different partitions, each partition utilizing one of two or more potential settings. For example, in a multi-user experience, one side of a meeting room may include all antique furniture, while the other side may include all modern furniture.

[0073] In some cases, the Experience Adaptation Engine 240 can perform environment fusion to generate smoother transitions between partitions in a multi-user experience environment. For example, a multi-user experience environment might be partitioned such that a blue sky appears to a multi-user experience participant standing in one partition, and a red sky appears to a multi-user experience participant standing in another partition. In some cases, as a multi-user experience participant moves from one partition to another, the Experience Adaptation Engine 240 can gradually transition the sky color between blue and red.

[0074] In some cases, when two or more potential settings exist for a particular arbitration setting 224, the experience adaptation engine 240 may attempt to generate new content that combines the two or more different potential settings into a new setting that combines the two or more potential settings together. For example, as noted above, the setting arbitration engine 230 may generate an adjusted setting for the arbitration setting 224 that specifies the blending percentage used to combine the two or more potential settings. In some cases, the experience adaptation engine 240 may include one or more generative neural networks (e.g., NLG 114, image generation 124, audio generation 134, and / or multi-model neural networks) capable of generating new content for multi-user experience environments. For example, the generative neural network may receive the blending percentage used to combine the two or more settings as input, and the generative neural network may output a blend of new content that merges preferences (e.g., combined modern / antique furniture).

[0075] In some implementations, the experience adaptation engine 240 may be configured to generate multiple potential multi-user experience environments based on one or more potential settings for arbitration setting 224 prior to the multi-user experience, rather than dynamically generating the adapted experience 250. In some cases, the multi-user experience can be selected from multiple potential multi-user experience environments through voting by multi-user experience participants (e.g., the option to vote on the multi-user experience environment can be given to a list of invited participants). In some cases, the experience adaptation engine 240 may also be configured to select an environment from multiple generated environments based on the multi-user experience participants' prior experience, analysis of the generated multi-user experience environment, and the multi-user experience participants' expected user engagement, any other selection criteria, and / or any combination thereof.

[0076] In some cases, the partitioning engine 220 and / or the arbitration engine 230 can receive feedback from the adapted experience 250, which can dynamically influence the preference partitioning and / or preference arbitration results. For example, as a multi-user experience participant leaves the multi-user experience, the setting can change from arbitration setting 224 to non-arbitration setting 222. For example, if the leaving participant is the only one with a setting that requires all multi-user experience participants to share a common sky color, the sky color setting can become non-arbitration setting 222.

[0077] Figure 4A An example is shown where arbitration setting 424 becomes non-arbitration setting 422. Figure 4A This illustrates a change in the arbitration status of the sky color, as indicated by moving the sky color from arbitration setting 424 to non-arbitration setting 422.

[0078] return Figure 2 In some cases, non-arbitration setting 222 can become arbitration setting 224 based on the addition of new multi-user experience participants. For example, a new multi-user experience participant might join to initiate an art exhibition. In some cases, the art selection can change from non-arbitration setting 222 to arbitration setting 224.

[0079] Figure 4B An example is shown where non-arbitration setting 423 becomes arbitration setting 425. Figure 4B An example is given of a change in the arbitration status of an art style, such as moving the art style from non-arbitration setting 423 to arbitration setting 425 as indicated.

[0080] return Figure 2In some cases, the Experience Adaptation Engine 240 may update the adapted experience 250 based on updates to the potential settings of each arbitration setting 224 provided to the Experience Adaptation Engine 240 by the Arbitration Engine 230, and / or based on a change in arbitration state from non-arbitration setting 222 to arbitration setting 224 or vice versa. For example, if the setting remains arbitration setting 224, and one or more potential settings of arbitration setting 224 provided to the Experience Adaptation Engine 240 by the Arbitration Engine 230 change, the Experience Adaptation Engine 240 may update the adapted experience 250 according to the updated potential settings of any arbitration setting 224. Similarly, if the setting changes in either direction between non-arbitration setting 222 and arbitration setting 224, the Experience Adaptation Engine 240 may update the adapted experience 250 accordingly.

[0081] In some cases, the arbitration engine 230 may use hysteresis to suppress potential changes to a particular arbitration setting 224. For example, when multi-user experience participants enter and exit the multi-user experience, the arbitration engine 230 may not immediately change the number of chairs in the multi-user experience environment. In an exemplary example, the arbitration engine 230 may only update the number of chairs if the difference between the number of chairs and the number of multi-user experience participants exceeds two. Similarly, in some cases, the experience adaptation engine 240 may use hysteresis to suppress potential changes to the arbitration setting 224 and / or changes in the arbitration state between the non-arbitration setting 222 and the arbitration setting 224 in either direction.

[0082] In some implementations, the adaptive settings adjustment engine 260 may obtain data from the adapted experience 250. For example, the adapted experience 250 may provide content interaction information, user engagement data, and / or any other data from the adapted experience 250. As an illustrative example, the adapted experience 250 may provide data to the adaptive settings adjustment engine 260 indicating that specific settings applied in the multi-user experience environment of the adapted experience 250 increase the engagement of the target multi-user experience participant. In some cases, the adaptive settings adjustment engine 260 may determine whether the target multi-user experience participant's existing settings reflect settings that lead to increased engagement that are preferred by the target multi-user experience participant. In some cases, the adaptive settings adjustment engine 260 may add one or more new settings for the target multi-user experience participant and / or update one or more existing settings for the target multi-user experience participant based on the determination that the target multi-user experience participant's existing settings do not properly address the target multi-user experience participant's preference for a particular setting. In some implementations, machine learning models (e.g., reinforcement learning models trained based on engagement statistics) can determine whether to change the preferences of target multi-user experience participants and / or the specific types of changes to be applied to the settings of target multi-user experience participants.

[0083] In some cases, the multi-user experience coordination system 200 may fail to honor one or more preferences of a particular multi-user experience participant. For example, mandatory settings for a particular multi-user experience participant may conflict with mandatory settings imposed by the meeting owner. In some cases, the meeting owner's mandatory settings may take precedence over the settings of a particular multi-user experience participant. In some cases, the multi-user experience coordination system 200 may output a warning to the particular multi-user experience participant that the settings will not be honored and ask the particular multi-user experience participant whether they still wish to join the multi-user experience.

[0084] In some cases, the multi-user experience coordination system 200 may provide a user interface for multi-user experience participants to inspect the content of the multi-user experience environment to determine whether the inspected content is included in the multi-user experience environment based on the arbitration setting 224. For example, the adapted experience 250 may display metadata and / or tags indicating the source of one or more settings. In some cases, metadata and / or tags may be overlaid on objects in the multi-user experience environment. In some cases, the user interface may contain a menu displaying all settings of the multi-user experience environment (whether they have been arbitrated) and / or one or more potential settings of the arbitration setting 224.

[0085] In some cases, the multi-user experience coordination system 200 may allow multi-user experience participants to save a set of arbitration settings 224 and non-arbitration settings 222 of the environment as settings for future multi-user experiences.

[0086] As noted above, the multi-user experience coordination system 200 and related technologies described herein provide coordinated multi-user experiences. These systems and technologies can divide settings into arbitrated settings and non-arbitrated settings. An arbitration engine can arbitrate the arbitrated settings to determine one or more adjusted settings for each arbitrated setting. An experience adaptation engine can generate adapted experiences that enforce the adjusted settings for each arbitrated setting while allowing participants to use their own settings for non-arbitrated preferences.

[0087] Figure 5 This is a flowchart illustrating an example of a process 500 for coordinating multiple user experiences. Process 500 and / or other processes described herein may be performed by a computing device (or apparatus) or a component of a computing device (e.g., chipset, codec, etc.). A computing device may be an extended reality (XR) device (e.g., a virtual reality (VR) device or an extended reality (AR) device), a mobile device (e.g., a mobile phone), a network-connected wearable device (such as a watch), a vehicle or a component or system of a vehicle, or other types of computing devices. In one example, process 500 and / or other processes described herein may be performed by... Figure 2The multi-user experience coordination system 200 is executed. In another example, one or more processes in these processes may be executed by... Figure 8 The computing system 800 shown is executed. For example, it has... Figure 8 The computing device of the computing system 800 shown may include components of the multi-user experience coordination system 200, and can achieve... Figure 5 The operation of process 500 and / or other processes described herein. The operation of process 500 may be implemented in one or more processors (e.g., Figure 8 The processor 810 is configured to execute machine learning models or algorithms (such as...) Figure 6 Deep learning networks 600 or Figure 7 Software components that execute and run on a processor (such as a DSP, GPU, NPU, etc.) or other processor (such as a CNN 700). Furthermore, the sending and receiving of signals via the computing device in process 500 may be, for example, via one or more antennas, one or more transceivers (e.g., wireless transceivers) and / or other communication components of the computing device (e.g., [missing information]). Figure 8 The communication interface 840 facilitates this.

[0088] At box 502, the computing device (or a component thereof) can obtain multiple settings associated with multiple multi-user experience participants (e.g., Figure 2 (Setting 201). In some examples, these multiple settings include one or more arbitration settings (e.g., Figure 2 Arbitration setting 224) and one or more non-arbitration settings (e.g., Figure 2 (Non-arbitration setting 222). In some cases, the multiple settings include one or more arbitration settings for a first multi-user experience participant and one or more arbitration settings for a second multi-user experience participant. In some aspects, the one or more arbitration settings include a single arbitration setting. In some implementations, the one or more arbitration settings for the first multi-user experience participant include a first setting of the single arbitration setting; and the one or more arbitration settings for the second multi-user experience participant include a second setting of the single arbitration setting.

[0089] At box 504, the computing device (or a component thereof) may be configured by setting an arbitration engine (e.g., Figure 2The setting arbitration engine 230 arbitrates the one or more arbitration settings to generate one or more adjusted settings for each arbitration setting. In some examples, arbitrating a single arbitration setting by the setting arbitration engine includes selecting between a first setting and a second setting of the single arbitration setting. In some cases, arbitrating a single arbitration setting by the setting arbitration engine includes generating an adjusted setting for the single arbitration setting based on the first setting and the second setting of the single arbitration setting. In some aspects, the adjusted setting of the single arbitration setting differs from the first setting and the second setting of the single arbitration setting. In some cases, arbitrating a single arbitration setting by the setting arbitration engine includes selecting the second setting of the single arbitration setting. In some implementations, the second setting of the single arbitration setting includes a preference for the single arbitration setting, and the first setting of the single arbitration setting does not include a preference for the single arbitration setting. In some examples, generating an adjusted setting for a single arbitration setting includes combining the first setting and the second setting of the single arbitration setting. In some aspects, combining the first setting of the single arbitration setting with the second setting of the single arbitration setting includes generating a percentage combination of the first setting and the second setting of the single arbitration setting. In some examples, the multiple settings associated with the multiple user experience participants include multiple preference intensities. In some examples, generating the percentage combination includes applying weights based on the multiple preference intensities.

[0090] In some examples, the first setting of the single arbitration setting includes a first preference range of the single arbitration setting, and the second setting of the single arbitration setting includes a second preference range of the single arbitration setting. In some aspects, arbitrating the single arbitration setting by the setting arbitration engine includes selecting from overlapping preference ranges associated with the first preference range and the second preference range of the single arbitration setting. In some cases, the second preference range of the single arbitration setting is a superset of the first preference range of the single arbitration setting. In some aspects, arbitrating the single arbitration setting by the setting arbitration engine includes selecting the first preference range of the single arbitration setting.

[0091] At box 506, the computing device (or its components) may be configured by an experience adaptation engine (e.g., Figure 2 The experience adaptation engine 240 generates adapted multi-user experiences (e.g., Figure 2The adapted multi-user experience (250). In some examples, the adapted multi-user experience is configured to enforce the one or more adapted settings for each arbitration setting. Enforcing a single arbitration setting among the one or more arbitration settings includes at least one or more of the following: directly manipulating the adapted multi-user experience to match the single arbitration setting; selecting a portion of the adapted multi-user experience from a database based on the single arbitration setting; or partitioning the adapted multi-user experience. In some examples, a first partition of the adapted multi-user experience includes a first setting of the single arbitration setting associated with the first multi-user experience participant, and a second partition of the adapted multi-user experience includes a second setting of the single arbitration setting associated with the second multi-user experience participant; or generating a merged portion of the adapted multi-user experience. In some cases, the merged portion of the adapted multi-user experience includes the first setting of the single arbitration setting associated with the first multi-user experience participant and the second setting of the single arbitration setting associated with the second multi-user experience participant.

[0092] In some cases, a first multi-user experience participant and a second multi-user experience participant participate in the adapted multi-user experience; and the adapted multi-user experience of the first multi-user experience participant includes one or more adjusted settings for each arbitration setting and a first setting associated with a first non-arbitration setting for the first multi-user experience participant. In some implementations, the adapted multi-user experience of the second multi-user experience participant includes one or more adjusted settings for each arbitration setting and a second setting associated with the first non-arbitration setting for the second multi-user experience participant. In some aspects, the second setting associated with the first non-arbitration setting for the second multi-user experience participant differs from the first setting associated with the first non-arbitration setting for the first multi-user experience participant. In some examples, the second setting associated with the first non-arbitration setting for the second multi-user experience participant matches the first setting associated with the first non-arbitration setting for the first multi-user experience participant. In some cases, a computing device (or a component thereof) may partition the multiple settings into one or more arbitration settings and one or more non-arbitration settings via a partitioning engine. In some aspects, dividing a setting among multiple settings into one or more arbitration settings includes determining whether that setting among the multiple settings needs to be consistent across all multi-user experience participants. In some examples, enforcing a single arbitration setting among the one or more arbitration settings includes generating a portion of the adapted multi-user experience from a generative model. In some cases, generating that portion of the adapted multi-user experience includes merging combinations of settings from two or more multi-user experience participants for that single arbitration setting among the one or more arbitration settings.

[0093] As noted above, the processes described herein (e.g., process 500 and / or other processes described herein) may be performed by a computing device or apparatus. In some cases, a computing device or apparatus may include various components such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components configured to perform the steps of the processes described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0094] A component capable of implementing a computing device in a circuit. For example, the component may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0095] Process 500 is illustrated as a logic flowchart, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or combinations thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0096] Additionally, process 500 and / or other processes described herein may be executed under the control of one or more computer systems configured using executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that executes jointly on one or more processors, by hardware, or a combination thereof. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0097] As noted above, various aspects of this disclosure may utilize machine learning models or systems. Figure 6 This is an exemplary example of a deep learning neural network 600, which can be used to implement the machine learning-based feature extraction and / or activity recognition (or classification) described above. Input layer 620 includes input data. In one exemplary example, input layer 620 may include data representing pixels of an input video frame. Neural network 600 includes multiple hidden layers 622a, 622b through 622n. Hidden layers 622a, 622b through 622n include “n” hidden layers, where “n” is an integer greater than or equal to one. Multiple hidden layers can be made to include as many layers as needed for a given application. Neural network 600 also includes an output layer 621, which provides the output produced by the processing performed by hidden layers 622a, 622b through 622n. In one exemplary example, output layer 621 can provide a classification of objects in an input video frame. The classification may include categories identifying activity types (e.g., looking up, looking down, eyes closed, yawning, etc.).

[0098] Neural network 600 is a multi-layered neural network composed of interconnected nodes. Each node can represent a piece of information. The information associated with these nodes is shared between different layers, and each layer retains information while processing it. In some cases, neural network 600 may include a feedforward network, in which case there are no feedback connections where the network's output is fed back into itself. In some cases, neural network 600 may include a recurrent neural network, which may have loops that allow information to be carried across nodes as input is read.

[0099] Information can be exchanged between nodes via node-to-node interconnects between layers. Nodes in input layer 620 can activate the node set in the first hidden layer 622a. For example, as shown, each input node in input layer 620 is connected to each node in the first hidden layer 622a. Nodes in the first hidden layer 622a can transform the information of each input node by applying an activation function to the input node information. The information derived from this transformation can then be passed to nodes in the next hidden layer 622b, activating those nodes, which can then perform their own specified functions. Example functions include convolution, upsampling, data transformation, and / or any other suitable function. The output of hidden layer 622b can then activate nodes in the next hidden layer, and so on. Finally, the output of hidden layer 622n can activate one or more nodes in output layer 621, providing the output at those nodes. In some cases, although a node in neural network 600 (e.g., node 626) is shown as having multiple output lines, the node has a single output and is shown as all lines output from the node representing the same output value.

[0100] In some cases, each node or the interconnection between nodes may have weights, which are a set of parameters derived from the training of the neural network 600. Once the neural network 600 is trained, it can be called a trained neural network, which can be used to classify one or more activities. For example, the interconnection between nodes may represent a piece of information learned about the interconnected nodes. The interconnection may have tunable numerical weights that can be tuned (e.g., based on the training dataset), allowing the neural network 600 to adapt to the input and learn as more and more data is processed.

[0101] The neural network 600 is pre-trained to process features from the data in the input layer 620 using different hidden layers 622a, 622b to 622n, in order to provide an output through the output layer 621. In an example where the neural network 600 is used to identify activities performed by a driver in a frame, the neural network 600 can be trained using training data that includes both frames and labels, as described above. For example, training frames can be input into the network, where each training frame has a label indicating a feature in the frame (for a feature extraction machine learning system) or a label indicating the category of activity in each frame. In an example where object classification is used for illustrative purposes, the training frame may include an image of the number 2, in which case the image label could be [0 0 1 00 0 0 0 0 0].

[0102] In some cases, the neural network 600 can use a training process called backpropagation to adjust the weights of its nodes. As noted above, the backpropagation process includes forward pass, loss function, back pass, and weight update. For each training iteration, forward pass, loss function, back pass, and parameter update are performed. This process can be repeated a certain number of iterations for each training image set until the neural network 600 is trained well enough that the weights of each layer are accurately tuned.

[0103] For an example of identifying objects in a frame, the forward pass may include passing a training frame through a neural network 600. The weights are initially randomized before training the neural network 600. As an illustrative example, a frame may include a numerical array representing pixels of an image. Each number in the array may include a value from 0 to 255 describing the intensity of the pixel at that location in the array. In one example, the array may include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or lightness and two chroma components, etc.).

[0104] As noted above, for the first training iteration of the neural network 600, the output will likely include values ​​due to the weights being randomly chosen during initialization, without prioritizing any particular class. For example, if the output is a vector with probabilities that an object includes different classes, the probability values ​​for each class may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). Using the initial weights, the neural network 600 cannot determine low-level features and therefore cannot make an accurate determination of what the object's classification might be. A loss function can be used to analyze the error in the output. Any suitable loss function can be defined, such as cross-entropy loss. Another example of a loss function includes mean squared error (MSE), which is defined as... The loss can be set to equal to The value of .

[0105] For the first training image, the loss (or error) will be high because the actual value will be significantly different from the predicted output. The goal of training is to minimize the loss so that the predicted output matches the training labels. The Neural Network 600 performs backpropagation by determining which inputs (weights) contribute most to the network's loss and can adjust the weights to reduce and eventually minimize this loss. The derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight at a specific layer) can be calculated to determine the weights that contribute most to the network's loss. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. A weight update can be represented as... Where w represents the weight, wi Let represent the initial weights, and η represent the learning rate. The learning rate can be set to any suitable value, where a high learning rate includes larger weight updates, while a lower value indicates smaller weight updates.

[0106] Neural Network 600 can include any suitable deep network. An example includes a Convolutional Neural Network (CNN), which includes an input layer and an output layer, with multiple hidden layers between them. The hidden layers of a CNN include a series of convolutional layers, non-linear layers, pooling layers (for downsampling), and fully connected layers. Neural Network 600 can include any other deep network besides CNNs, such as autoencoders, deep belief networks (DBNs), recurrent neural networks (RNNs), and so on.

[0107] Figure 7 This is an exemplary example of a Convolutional Neural Network (CNN) 700. The input layer 720 of the CNN 700 includes data representing an image or frame. For example, the data may include a numerical array representing pixels of an image, where each number in the array includes a value from 0 to 255 describing the pixel intensity at that location in the array. Using the previous example from above, the array may include a 28×28×3 numerical array with 28 rows and 28 columns of pixels and 3 color components (e.g., red, green, and blue, or lightness and two chroma components, etc.). The image may be passed through a convolutional hidden layer 722a, an optional non-linear activation layer, a pooling hidden layer 722b, and a fully connected hidden layer 722c to obtain an output at the output layer 724. Although... Figure 7 Only one hidden layer from each hidden layer is shown in the diagram, but those skilled in the art will understand that multiple convolutional hidden layers, non-linear layers, pooling hidden layers, and / or fully connected layers may be included in the CNN 700. As previously described, the output may indicate a single category of an object, or may include probabilities that best describe the category of an object in an image.

[0108] The first layer of CNN 700 is a convolutional hidden layer 722a. Convolutional hidden layer 722a analyzes the image data input to layer 720. Each node in convolutional hidden layer 722a is connected to a node (pixel) region in the input image called a receptive field. Convolutional hidden layer 722a can be thought of as one or more filters (each filter corresponding to a different activation or feature map), where each convolutional iteration of the filter is a node or neuron in convolutional hidden layer 722a. For example, the region of the input image covered by the filter at each convolutional iteration will be the filter's receptive field. In an exemplary example, if the input image consists of a 28×28 array and each filter (and its corresponding receptive field) is a 5×5 array, then there will be 24×24 nodes in convolutional hidden layer 722a. Each connection between a node and its receptive field learns weights and, in some cases, learns an overall bias, allowing each node to learn to analyze its specific local receptive field in the input image. Each node in hidden layer 722a will have the same weights and biases (referred to as shared weights and shared biases). For example, the filter has a weight (digital) array and the same depth as the input. For the video frame example, the filter would have a depth of 3 (based on the three color components of the input image). An exemplary example of the filter array size is 5×5×3, corresponding to the size of the receptive field of a node.

[0109] The convolutional property of the convolutional hidden layer 722a is due to the fact that each node of the convolutional layer is applied to its corresponding receptive field. For example, the filters of the convolutional hidden layer 722a may begin at the top left corner of the input image array and may convolve around the input image. As noted above, each convolutional iteration of the filter can be considered as a node or neuron of the convolutional hidden layer 722a. In each convolutional iteration, the value of the filter is multiplied by the corresponding number of original pixel values ​​of the image (e.g., a 5×5 filter array is multiplied by a 5×5 array of input pixel values ​​at the top left corner of the input image array). The multiplications from each convolutional iteration can be summed to obtain the sum of that iteration or node. Next, the process continues at the next position in the input image based on the receptive field of the next node in the convolutional hidden layer 722a. For example, the filter may move a step size (called stride) to the next receptive field. The stride may be set to 1 or other suitable amounts. For example, if the stride is set to 1, the filter will move 1 pixel to the right in each convolutional iteration. Processing the filter at each unique location in the input volume produces a number representing the filter result at that location, thus producing a sum value determined for each node of the convolutional hidden layer 722a.

[0110] The mapping from the input layer to the convolutional hidden layer 722a is called an activation map (or feature map). An activation map includes the value of each node, representing the filter result at each location in the input volume. Activation maps can include arrays containing various sums of values ​​produced by the filter on each iteration of the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a stride of 1), the activation map would consist of a 24×24 array. The convolutional hidden layer 722a can include several activation maps to identify multiple features in the image. Figure 7 The example shown includes three activation maps. Using the three activation maps, the convolutional hidden layer 722a can detect three different types of features, each of which is detectable across the entire image.

[0111] In some examples, nonlinear hidden layers can be applied after convolutional hidden layer 722a. Nonlinear layers can be used to introduce nonlinearity into a system that has already computed linear operations. An exemplary example of a nonlinear layer is the Corrected Linear Unit (ReLU) layer. A ReLU layer applies the function f(x) = max(0, x) to all values ​​in the input volume, which changes all negative activations to 0. Therefore, ReLU can increase the nonlinearity of CNN 700 without affecting the receptive field of convolutional hidden layer 722a.

[0112] A pooling hidden layer 722b can be applied after the convolutional hidden layer 722a (and, in use, after the non-linear hidden layer). The pooling hidden layer 722b is used to simplify the information in the output of the convolutional hidden layer 722a. For example, the pooling hidden layer 722b can take each activation map output from the convolutional hidden layer 722a and use a pooling function to generate a condensed activation map (or feature map). Max pooling is an example of a function performed by the pooling hidden layer. The pooling hidden layer 722a uses other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions. Pooling functions (e.g., max pooling filters, L2 norm filters, or other suitable pooling filters) are applied to each activation map included in the convolutional hidden layer 722a. Figure 7 In the example shown, three pooling filters are used to convolve the three activation maps in the hidden layer 722a.

[0113] In some examples, max pooling can be used by applying a max pooling filter (e.g., of size 2×2) with a stride (e.g., equal to the dimension of the filter, such as stride 2) to the activation map output from the convolutional hidden layer 722a. The output from the max pooling filter includes the maximum number in each sub-region of the filter convolution. Using a 2×2 filter as an example, each unit in the pooling layer can summarize a region of 2×2 nodes from the previous layer (where each node is a value in the activation map). For example, four values ​​(nodes) in the activation map will be analyzed by the 2×2 max pooling filter at each iteration of the filter, with the maximum of the four values ​​being output as the "maximum" value. If such a max pooling filter is applied to an activation filter of size 24×24 nodes from the convolutional hidden layer 722a, the output from the pooling hidden layer 722b will be an array of 12×12 nodes.

[0114] In some examples, L2 norm pooling filters may also be used. L2 norm pooling filters involve calculating the square root of the sum of squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (instead of calculating the maximum value as done in max pooling), and using the calculated value as the output.

[0115] Intuitively, pooling functions (e.g., max pooling, L2-norm pooling, or other pooling functions) determine whether a given feature is found anywhere in a region of an image, discarding the exact location information. This can be done without affecting the results of feature detection, because once a feature has been found, its exact location is less important than its approximate location relative to other features. Max pooling (and other pooling methods) offers the benefit of having far fewer pooling features, thus reducing the number of parameters required in subsequent layers of a CNN 700.

[0116] The final connection in the network is a fully connected layer, which connects each node from the pooling hidden layer 722b to each output node in the output layer 724. Using the example above, the input layer comprises 28×28 nodes encoding the pixel intensity of the input image, the convolutional hidden layer 722a comprises 3×24×24 hidden feature nodes based on applying a 5×5 local receptive field (for filtering) to three activation maps, and the pooling hidden layer 722b comprises a layer based on applying a max-pooling filter to a 2×2 region across each of the three feature maps. Extending this example, the output layer 724 may comprise ten output nodes. In such an example, each node of the 3×12×12 pooling hidden layer 722b is connected to each node of the output layer 724.

[0117] The fully connected layer 722c takes the output of the previous pooling hidden layer 722b (which should represent the activation map of high-level features) and determines the features most relevant to a particular class. For example, the fully connected layer 722c can determine the high-level features most relevant to a particular class and may include weights (nodes) for those high-level features. The product between the weights of the fully connected layer 722c and the pooling hidden layer 722b can be computed to obtain the probabilities for different classes. For example, if CNN 700 is being used to predict whether an object in a video frame is a person, there will be high values ​​in the activation map representing the high-level features of a person (e.g., two legs, a face at the top of the object, two eyes at the upper left and upper right of the face, a nose in the middle of the face, a mouth at the bottom of the face, and / or other features common to people).

[0118] In some examples, the output from output layer 724 may include an M-dimensional vector (M=10 in the previous example). M indicates the number of classes the CNN 700 must choose from when classifying objects in an image. Other example outputs may also be provided. Each number in the M-dimensional vector represents the probability that an object belongs to a certain class. In an exemplary example, if the 10-dimensional output vector representing objects of ten different classes is [0 0 0.05 0.8 0 0.15 0 0 0 0], then the vector indicates a 5% probability that the image is an object of the third class (e.g., a dog), an 80% probability that the image is an object of the fourth class (e.g., a person), and a 15% probability that the image is an object of the sixth class (e.g., a kangaroo). The probability of a class can be considered as the confidence level that an object is part of that class.

[0119] Figure 8 This is a diagram illustrating an example of a system used to implement certain aspects of this technology. Specifically, Figure 8 An example of a computing system 800 is illustrated. This computing system can be any computing device, such as constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using a connection 805. The connection 805 can be a physical connection using a bus, or a direct connection to a processor 810, such as in a chipset architecture. The connection 805 can also be a virtual connection, a networking connection, or a logical connection.

[0120] In some embodiments, the computing system 800 is a distributed system, wherein the functions described herein may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a plurality of such components, each of which performs some or all of the functions of the described components. In some embodiments, the components may be physical devices or virtual devices.

[0121] Example system 800 includes at least one processing unit (CPU or processor) 810 and a connection 805 that couples various system components, including system memories 815 such as read-only memory (ROM) 820 and random access memory (RAM) 825, to processor 810. Computing system 800 may include a cache 812 of high-speed memory that is directly connected to, closely adjacent to, or integrated into processor 810.

[0122] Processor 810 may include any general-purpose processor and hardware or software services, such as services 832, 834, and 836 stored in storage device 830, which are configured to control processor 810 and dedicated processors in which software instructions are incorporated into the actual processor design. Processor 810 may be a substantially completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0123] To enable user interaction, the computing system 800 includes an input device 845 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. The computing system 800 may also include an output device 835 that can be one or more of a plurality of output mechanisms. In some instances, a multi-mode system allows a user to provide multiple types of input / output to communicate with the computing system 800. The computing system 800 may include a communication interface 840, which typically governs and manages user input and system output. The communication interface can perform or facilitate the receipt and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including utilizing audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple... ® Lightning ® Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, dedicated wired ports / plugs, Bluetooth ® Wireless signal transmission, Bluetooth ® Low-power (BLE) wireless signal transmission, IBEACON ®The communication interface 840 may include wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), microwave access global interoperability (WiMAX), infrared (IR) wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 840 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for determining the location of the computing system 800 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the U.S. Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There are no limitations on operation on any particular hardware configuration, and therefore the underlying features here can be easily replaced to obtain improved hardware or firmware configurations as they are developed.

[0124] Storage device 830 may be a non-volatile and / or non-transitory and / or computer-readable storage device, and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as magnetic tape cassettes, flash memory cards, solid-state storage devices, digital multifunction disks, cassette tapes, floppy disks, flexible disks, hard disks, magnetic tapes, magnetic stripes / strips, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, CD-ROM, rewritable CD, digital video disc (DVD), Blu-ray disc (BDD), holographic disc, another optical medium, secure digital (SD) cards, micro-secure digital (microSD) cards, Memory Stick. ®Cards, smart card chips, EMV chips, Subscriber Identity Module (SIM) cards, mini / micro / nano / micro SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase-change memory (PCM), spin-transfer torque RAM (STT-RAM), another memory chip or cassette and / or combinations thereof.

[0125] Storage device 830 may include software services, servers, etc., which enable the system to perform functions when the code defining such software is executed by processor 810. In some embodiments, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 810, connection 805, output device 835, etc.

[0126] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and which do not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted using any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0127] In some implementations, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as power consumption, carrier signals, electromagnetic waves, and the signals themselves.

[0128] Specific details are provided in the foregoing description to provide a thorough understanding of the embodiments and examples presented herein. However, those skilled in the art will understand that embodiments can be practiced without these specific details. For clarity, in some instances, the technology may be presented as comprising individual functional blocks, including functional blocks containing devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid these embodiments becoming obscure with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without necessary details to avoid obscuring the embodiments.

[0129] Individual implementations may be described above as processes or methods depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although flowcharts may describe operations as sequential processes, many operations within an operation may be executed in parallel or concurrently. Furthermore, the order of operations may be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, process, subroutine, subroutine, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0130] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure, or otherwise configure, a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion may be accessible via a network of the computer resources used. Computer-executable instructions may be, for example, binary, intermediate-format instructions, such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical discs, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0131] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or intercalation cards. By way of additional examples, such functionality may also be implemented on circuit boards of different chips or different processes executed on a single device.

[0132] Instructions, media for delivering such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0133] In the foregoing description, various aspects of this application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, although exemplary embodiments of this application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in a variety of other ways, and the appended claims are intended to be construed as including such variations, unless limited by prior art. The various features and aspects of the applications described above may be used individually or in combination. Furthermore, without departing from the scope of this specification, the embodiments can be used in any number of environments and applications beyond those described herein. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in a different order than described.

[0134] Those skilled in the art will understand that, without departing from the scope of this description, the less than (“<”) and greater than (“>”) symbols or terms used herein may be replaced with less than or equal to (“>”) respectively. ") and greater than or equal to (" The symbol ) is used instead.

[0135] When a component is described as being “configured” to perform certain operations, such a configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0136] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0137] The claim language or other language that states "at least one of" and / or "one or more of" in a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, the claim language that states "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, the claim language that states "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language that states "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the claim language that states "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0138] Claims using phrases such as "at least one processor, the at least one processor being configured to," or other languages ​​indicate that one or more processors (in any combination) are capable of performing associated operations. For example, a claim stating "at least one processor, the at least one processor being configured to: X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each assigned a specific subset of tasks involving operations X, Y, and Z, such that the multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, a claim stating "at least one processor, the at least one processor being configured to: X, Y, and Z" could mean that any single processor can perform only a subset of operations X, Y, and Z.

[0139] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been broadly described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0140] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0141] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein.

[0142] The exemplary aspects of this disclosure include:

[0143] Aspect 1. A method for coordinating multiple user experiences, the method comprising: obtaining multiple settings associated with multiple multiple user experience participants, wherein the multiple settings include one or more arbitration settings and one or more non-arbitration settings; arbitrating the one or more arbitration settings by a settings arbitration engine to generate one or more adapted settings for each arbitration setting; and generating an adapted multi-user experience by an experience adaptation engine, wherein the adapted multi-user experience is configured to enforce the one or more adapted settings for each arbitration setting.

[0144] Aspect 2. The method according to aspect 1, wherein the plurality of settings includes one or more arbitration settings for a first multi-user experience participant and one or more arbitration settings for a second multi-user experience participant.

[0145] Aspect 3. The method according to any one of Aspects 1 to 2, wherein the one or more arbitration settings include a single arbitration setting, wherein: the one or more arbitration settings for the first multi-user experience participant include a first setting of the single arbitration setting; and the one or more arbitration settings for the second multi-user experience participant include a second setting of the single arbitration setting.

[0146] Aspect 4. The method according to any one of Aspects 1 to 3, wherein arbitrating the individual arbitration settings by the setting arbitration engine comprises: selecting between a first setting of the individual arbitration settings and a second setting of the individual arbitration settings.

[0147] Aspect 5. The method according to any one of Aspects 1 to 4, wherein arbitrating the individual arbitration setting by the setting arbitration engine comprises: generating an adjusted setting of the individual arbitration setting based on the first setting and the second setting of the individual arbitration setting, wherein the adjusted setting of the individual arbitration setting is different from the first setting and the second setting of the individual arbitration setting.

[0148] Aspect 6. The method according to any one of Aspects 1 to 5, wherein arbitrating the individual arbitration setting by the setting arbitration engine comprises: selecting a second setting of the individual arbitration setting, wherein the second setting of the individual arbitration setting includes a preference for the individual arbitration setting, and the first setting of the individual arbitration setting does not include a preference for the individual arbitration setting.

[0149] Aspect 7. The method according to any one of Aspects 1 to 6, wherein: the first setting of the single arbitration setting includes a first preference range of the single arbitration setting; the second setting of the single arbitration setting includes a second preference range of the single arbitration setting; and arbitrating the single arbitration setting by the setting arbitration engine includes: selecting from an overlapping preference range associated with the first preference range and the second preference range of the single arbitration setting.

[0150] Aspect 8. The method according to any one of Aspects 1 to 7, wherein the second preference range of the single arbitration setting is a superset of the first preference range of the single arbitration setting, and wherein arbitrating the single arbitration setting by the setting arbitration engine comprises: selecting the first preference range of the single arbitration setting.

[0151] Aspect 9. The method according to any one of Aspects 1 to 8, wherein generating the adjusted settings of the single arbitration setting comprises: combining the first setting of the single arbitration setting with the second setting of the single arbitration setting.

[0152] Aspect 10. The method according to any one of Aspects 1 to 9, wherein combining the first setting of the single arbitration setting with the second setting of the single arbitration setting comprises: generating a percentage combination of the first setting of the single arbitration setting and the second setting of the single arbitration setting.

[0153] Aspect 11. The method according to any one of Aspects 1 to 10, wherein the plurality of settings associated with the plurality of multi-user experience participants include a plurality of preference intensities, wherein generating the percentage combination includes: applying a weighting based on the plurality of preference intensities.

[0154] Aspect 12. The method according to any one of Aspects 1 to 11, wherein: a first multi-user experience participant and a second multi-user experience participant participate in the adapted multi-user experience; and the adapted multi-user experience of the first multi-user experience participant includes one or more adjusted settings for each arbitration setting and a first setting for the first multi-user experience participant associated with a first non-arbitration setting.

[0155] Aspect 13. The method according to any one of Aspects 1 to 12, wherein the adapted multi-user experience of the second multi-user experience participant includes the one or more adjusted settings for each arbitration setting and a second setting for the second multi-user experience participant associated with the first non-arbitration setting.

[0156] Aspect 14. The method according to any one of Aspects 1 to 13, wherein the second setting associated with the first non-arbitration setting for the second multi-user experience participant is different from the first setting associated with the first non-arbitration setting for the first multi-user experience participant.

[0157] Aspect 15. The method according to any one of Aspects 1 to 14, wherein the second setting associated with the first non-arbitration setting for the second multi-user experience participant matches the first setting associated with the first non-arbitration setting for the first multi-user experience participant.

[0158] Aspect 16. The method according to any one of Aspects 1 to 15, the method further comprising: dividing the plurality of settings into one or more arbitrated settings and one or more non-arbitrated settings by a partitioning engine, wherein dividing the settings among the plurality of settings into the one or more arbitrated settings comprises: determining whether the settings among the plurality of settings need to be consistent across all multi-user experience participants.

[0159] Aspect 17. The method according to any one of Aspects 1 to 16, wherein enforcing a single arbitration setting among the one or more arbitration settings comprises: generating a portion of the adapted multi-user experience by a generative model, wherein generating the portion of the adapted multi-user experience comprises: merging a combination of settings from two or more multi-user experience participants for the single arbitration setting among the one or more arbitration settings.

[0160] Aspect 18. The method according to any one of Aspects 1 to 17, wherein enforcing a single arbitration setting among the one or more arbitration settings comprises at least one or more of the following: directly manipulating the adapted multi-user experience to match the single arbitration setting; selecting a portion of the adapted multi-user experience from a database based on the single arbitration setting; partitioning the adapted multi-user experience, wherein a first partition of the adapted multi-user experience includes a first setting of the single arbitration setting associated with a first multi-user experience participant, and a second partition of the adapted multi-user experience includes a second setting of the single arbitration setting associated with a second multi-user experience participant; or generating a fused portion of the adapted multi-user experience, wherein the fused portion of the adapted multi-user experience includes the first setting of the single arbitration setting associated with the first multi-user experience participant and the second setting of the single arbitration setting associated with the second multi-user experience participant.

[0161] Aspect 19. An apparatus for coordinating multiple user experiences. The apparatus includes at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to: obtain a plurality of settings associated with a plurality of multiple user experience participants, wherein the plurality of settings includes one or more arbitration settings and one or more non-arbitration settings; arbitrate the one or more arbitration settings by a settings arbitration engine to generate one or more adapted settings for each arbitration setting; and generate an adapted multi-user experience by an experience adaptation engine, wherein the adapted multi-user experience is configured to enforce the one or more adapted settings for each arbitration setting.

[0162] Aspect 20. The apparatus according to aspect 19, wherein the plurality of settings includes one or more arbitration settings for a first multi-user experience participant and one or more arbitration settings for a second multi-user experience participant.

[0163] Aspect 21. The apparatus according to any one of Aspects 19 to 20, wherein the one or more arbitration settings include a single arbitration setting, wherein the one or more arbitration settings for the first multi-user experience participant include a first setting of the single arbitration setting; and the one or more arbitration settings for the second multi-user experience participant include a second setting of the single arbitration setting.

[0164] Aspect 22. The apparatus according to any one of aspects 19 to 21, wherein, in order for the arbitration engine to arbitrate the individual arbitration settings, the at least one processor is configured to select between a first setting of the individual arbitration settings and a second setting of the individual arbitration settings.

[0165] Aspect 23. The apparatus according to any one of Aspects 19 to 22, wherein, in order for the setting arbitration engine to arbitrate the individual arbitration settings, the at least one processor is configured to: generate an adjusted setting of the individual arbitration settings based on the first setting and the second setting of the individual arbitration settings, wherein the adjusted setting of the individual arbitration settings is different from the first setting and the second setting of the individual arbitration settings.

[0166] Aspect 24. The apparatus according to any one of Aspects 19 to 23, wherein, in order for the single arbitration setting to be arbitrated by the setting arbitration engine, the at least one processor is configured to: select a second setting of the single arbitration setting, wherein the second setting of the single arbitration setting includes a preference for the single arbitration setting, and the first setting of the single arbitration setting does not include a preference for the single arbitration setting.

[0167] Aspect 25. The apparatus according to any one of Aspects 19 to 24, wherein the first setting of the single arbitration setting includes a first preference range of the single arbitration setting, and the second setting of the single arbitration setting includes a second preference range of the single arbitration setting, and wherein, in order for the single arbitration setting to be arbitrated by the setting arbitration engine, the at least one processor is configured to select from an overlapping preference range associated with the first preference range of the single arbitration setting and the second preference range of the single arbitration setting.

[0168] Aspect 26. The apparatus according to any one of Aspects 19 to 25, wherein the second preference range of the single arbitration setting is a superset of the first preference range of the single arbitration setting, and wherein arbitrating the single arbitration setting by the setting arbitration engine comprises: selecting the first preference range of the single arbitration setting.

[0169] Aspect 27. The apparatus according to any one of aspects 19 to 26, wherein generating the adjusted setting of the single arbitration setting includes: combining the first setting of the single arbitration setting with the second setting of the single arbitration setting.

[0170] Aspect 28. The apparatus according to any one of Aspects 19 to 27, wherein combining the first setting of the single arbitration setting with the second setting of the single arbitration setting comprises: generating a percentage combination of the first setting of the single arbitration setting and the second setting of the single arbitration setting.

[0171] Aspect 29. The apparatus according to any one of aspects 19 to 28, wherein the plurality of settings associated with the plurality of multi-user experience participants include a plurality of preference intensities, wherein generating the percentage combination includes applying a weighting based on the plurality of preference intensities.

[0172] Aspect 30. The apparatus according to any one of Aspects 19 to 29, wherein: a first multi-user experience participant and a second multi-user experience participant participate in the adapted multi-user experience; and the adapted multi-user experience of the first multi-user experience participant includes one or more adjusted settings for each arbitration setting and a first setting for the first multi-user experience participant associated with a first non-arbitration setting.

[0173] Aspect 31. The apparatus according to any one of Aspects 19 to 30, wherein the adapted multi-user experience of the second multi-user experience participant includes the one or more adjusted settings for each arbitration setting and a second setting for the second multi-user experience participant associated with the first non-arbitration setting.

[0174] Aspect 32. The apparatus according to any one of Aspects 19 to 31, wherein the second setting associated with the first non-arbitration setting for the second multi-user experience participant is different from the first setting associated with the first non-arbitration setting for the first multi-user experience participant.

[0175] Aspect 33. The apparatus according to any one of Aspects 19 to 32, wherein the second setting associated with the first non-arbitration setting for the second multi-user experience participant matches the first setting associated with the first non-arbitration setting for the first multi-user experience participant.

[0176] Aspect 34. The apparatus according to any one of Aspects 19 to 33, wherein the at least one processor is configured to: divide the plurality of settings into one or more arbitration settings and one or more non-arbitration settings, wherein dividing the settings of the plurality of settings into one or more arbitration settings includes: determining whether the settings of the plurality of settings need to be consistent across all multi-user experience participants.

[0177] Aspect 35. The apparatus according to any one of aspects 19 to 34, wherein enforcing a single arbitration setting among the one or more arbitration settings comprises: generating a portion of the adapted multi-user experience from a generative model, wherein generating the portion of the adapted multi-user experience comprises: merging a combination of settings from two or more multi-user experience participants for the single arbitration setting among the one or more arbitration settings.

[0178] Aspect 36. The apparatus according to any one of Aspects 19 to 35, wherein enforcing a single arbitration setting among the one or more arbitration settings comprises at least one or more of the following: directly manipulating the adapted multi-user experience to match the single arbitration setting; selecting a portion of the adapted multi-user experience from a database based on the single arbitration setting; partitioning the adapted multi-user experience, wherein a first partition of the adapted multi-user experience includes a first setting of the single arbitration setting associated with a first multi-user experience participant, and a second partition of the adapted multi-user experience includes a second setting of the single arbitration setting associated with a second multi-user experience participant; or generating a fused portion of the adapted multi-user experience, wherein the fused portion of the adapted multi-user experience includes the first setting of the single arbitration setting associated with the first multi-user experience participant and the second setting of the single arbitration setting associated with the second multi-user experience participant.

[0179] Aspect 37: A non-transitory computer-readable storage medium having instructions stored thereon, the instructions causing the one or more processors, when executed, to perform any of the operations described in aspects 1 to 36.

[0180] Aspect 38: An apparatus comprising components for performing any of the operations described in aspects 1 to 36.

Claims

1. A method for coordinating multiple user experiences, the method comprising: Obtain multiple settings associated with multiple user experience participants, wherein the multiple settings include one or more arbitration settings and one or more non-arbitration settings; The arbitration engine arbitrates the one or more arbitration settings to generate one or more adjusted settings for each arbitration setting; as well as An adapted multi-user experience is generated by the experience adaptation engine, wherein the adapted multi-user experience is configured to enforce one or more adapted settings for each arbitration setting.

2. The method of claim 1, wherein the plurality of settings includes one or more arbitration settings for a first multi-user experience participant and one or more arbitration settings for a second multi-user experience participant.

3. The method of claim 2, wherein the one or more arbitration settings comprise a single arbitration setting, wherein: The one or more arbitration settings for the first multi-user experience participant include a first setting of the single arbitration setting; and The one or more arbitration settings for the second multi-user experience participants include a second setting of the single arbitration setting.

4. The method of claim 3, wherein arbitrating the individual arbitration settings by the arbitration engine comprises: Choose between the first setting and the second setting of the individual arbitration settings.

5. The method of claim 3, wherein arbitrating the individual arbitration settings by the arbitration engine comprises: An adjusted setting for a single arbitration setting is generated based on the first setting and the second setting of the single arbitration setting, wherein the adjusted setting of the single arbitration setting is different from the first setting and the second setting of the single arbitration setting.

6. The method of claim 3, wherein arbitrating the individual arbitration settings by the arbitration engine comprises: The second setting of the individual arbitration setting is selected, wherein the second setting of the individual arbitration setting includes a preference for the individual arbitration setting, and the first setting of the individual arbitration setting does not include a preference for the individual arbitration setting.

7. The method according to claim 3, wherein: The first setting of the individual arbitration setting includes a first preference range of the individual arbitration setting; The second setting of the individual arbitration setting includes a second preference range for the individual arbitration setting; and Arbitrating a single arbitration setting by the arbitration engine includes selecting from an overlapping range of preferences associated with a first preference range and a second preference range of the single arbitration setting.

8. The method of claim 7, wherein the second preference range of the single arbitration setting is a superset of the first preference range of the single arbitration setting, and wherein arbitration of the single arbitration setting by the setting arbitration engine comprises: Select the first preference range for the individual arbitration settings.

9. The method of claim 3, wherein generating the adjusted settings for the single arbitration setting comprises: The first setting of the single arbitration setting is combined with the second setting of the single arbitration setting.

10. The method of claim 9, wherein combining the first setting of the single arbitration setting with the second setting of the single arbitration setting comprises: Generate a percentage combination of the first setting and the second setting of the single arbitration setting.

11. The method of claim 10, wherein the plurality of settings associated with the plurality of multi-user experience participants include a plurality of preference intensities, wherein generating the percentage combination comprises: Weighting is applied based on the strength of the multiple preferences.

12. The method according to claim 1, wherein: First and second multi-user experience participants engage in the adapted multi-user experience; and The adapted multi-user experience of the first multi-user experience participant includes one or more adjusted settings for each arbitration setting and a first setting associated with the first non-arbitration setting for the first multi-user experience participant.

13. The method of claim 12, wherein the adapted multi-user experience of the second multi-user experience participant includes the one or more adjusted settings for each arbitration setting and a second setting for the second multi-user experience participant associated with the first non-arbitration setting.

14. The method of claim 13, wherein the second setting associated with the first non-arbitration setting for the second multi-user experience participant is different from the first setting associated with the first non-arbitration setting for the first multi-user experience participant.

15. The method of claim 13, wherein the second setting associated with the first non-arbitration setting for the second multi-user experience participant matches the first setting associated with the first non-arbitration setting for the first multi-user experience participant.

16. The method according to claim 1, wherein the method further comprises: The partitioning engine divides the plurality of settings into one or more arbitration settings and one or more non-arbitration settings, wherein dividing the settings among the plurality of settings into one or more arbitration settings includes: determining whether the settings among the plurality of settings need to be consistent across all multi-user experience participants.

17. The method of claim 1, wherein enforcing a single arbitration setting among the one or more arbitration settings comprises: A portion of the adapted multi-user experience is generated by a generative model, wherein generating the portion of the adapted multi-user experience includes: merging a combination of settings from two or more multi-user experience participants for the single arbitration setting in the one or more arbitration settings.

18. The method of claim 1, wherein enforcement of a single arbitration setting among the one or more arbitration settings comprises at least one or more of the following: Directly manipulate the adapted multi-user experience to match the single arbitration setting; Based on the single arbitration setting, a portion of the adapted multi-user experience is selected from the database; or The adapted multi-user experience is divided into two parts: a first partition of the adapted multi-user experience includes the first setting associated with the first multi-user experience participant of the single arbitration setting, and a second partition of the adapted multi-user experience includes the second setting associated with the second multi-user experience participant of the single arbitration setting; or a fused portion of the adapted multi-user experience is generated, wherein the fused portion of the adapted multi-user experience includes the first setting associated with the first multi-user experience participant of the single arbitration setting and the second setting associated with the second multi-user experience participant of the single arbitration setting.

19. An apparatus for coordinating multiple user experiences, the apparatus comprising: At least one memory; and At least one processor, the at least one processor being coupled to the at least one memory and being configured to: Obtain multiple settings associated with multiple user experience participants, wherein the multiple settings include one or more arbitration settings and one or more non-arbitration settings; The arbitration engine arbitrates the one or more arbitration settings to generate one or more adjusted settings for each arbitration setting; as well as An adapted multi-user experience is generated by the experience adaptation engine, wherein the adapted multi-user experience is configured to enforce one or more adapted settings for each arbitration setting.

20. The apparatus of claim 19, wherein the plurality of settings includes one or more arbitration settings for a first multi-user experience participant and one or more arbitration settings for a second multi-user experience participant.

21. The apparatus of claim 20, wherein the one or more arbitration settings comprise a single arbitration setting, wherein: The one or more arbitration settings for the first multi-user experience participant include a first setting of the single arbitration setting; and The one or more arbitration settings for the second multi-user experience participants include a second setting of the single arbitration setting.

22. The apparatus according to claim 21, wherein, In order for the arbitration engine to arbitrate the individual arbitration settings, the at least one processor is configured to select between the first setting and the second setting of the individual arbitration settings.

23. The apparatus according to claim 21, wherein, In order for the arbitration engine to arbitrate the individual arbitration settings, the at least one processor is configured to generate an adjusted setting for the individual arbitration settings based on the first setting and the second setting of the individual arbitration settings, wherein the adjusted setting of the individual arbitration settings is different from the first setting and the second setting of the individual arbitration settings.

24. The apparatus according to claim 21, wherein, In order for the arbitration engine to arbitrate the individual arbitration settings, the at least one processor is configured to: select a second setting of the individual arbitration settings, wherein the second setting of the individual arbitration settings includes a preference for the individual arbitration settings, and the first setting of the individual arbitration settings does not include a preference for the individual arbitration settings.

25. The apparatus of claim 21, wherein the first setting of the single arbitration setting includes a first preference range of the single arbitration setting, and the second setting of the single arbitration setting includes a second preference range of the single arbitration setting, and wherein, In order for the arbitration engine to arbitrate the individual arbitration settings, the at least one processor is configured to select from an overlapping range of preferences associated with the first preference range and the second preference range of the individual arbitration settings.

26. The apparatus of claim 25, wherein the second preference range of the single arbitration setting is a superset of the first preference range of the single arbitration setting, and wherein arbitration of the single arbitration setting by the setting arbitration engine comprises: Select the first preference range for the individual arbitration settings.

27. The apparatus of claim 21, wherein the adjusted settings for generating the single arbitration setting comprise: The first setting of the single arbitration setting is combined with the second setting of the single arbitration setting.

28. The apparatus of claim 27, wherein the first setting combining the single arbitration setting and the second setting combining the single arbitration setting comprises: Generate a percentage combination of the first setting and the second setting of the single arbitration setting.

29. The apparatus of claim 28, wherein the plurality of settings associated with the plurality of multi-user experience participants include a plurality of preference intensities, wherein generating the percentage combination comprises: Weighting is applied based on the strength of the multiple preferences.

30. The apparatus according to claim 19, wherein: First and second multi-user experience participants engage in the adapted multi-user experience; and The adapted multi-user experience of the first multi-user experience participant includes one or more adjusted settings for each arbitration setting and a first setting associated with the first non-arbitration setting for the first multi-user experience participant.

31. The apparatus of claim 30, wherein the adapted multi-user experience of the second multi-user experience participant includes the one or more adjusted settings for each arbitration setting and a second setting for the second multi-user experience participant associated with the first non-arbitration setting.

32. The apparatus of claim 31, wherein the second setting associated with the first non-arbitration setting for the second multi-user experience participant is different from the first setting associated with the first non-arbitration setting for the first multi-user experience participant.

33. The apparatus of claim 31, wherein the second setting associated with the first non-arbitration setting for the second multi-user experience participant matches the first setting associated with the first non-arbitration setting for the first multi-user experience participant.

34. The apparatus of claim 19, wherein the at least one processor is configured to: divide the plurality of settings into one or more arbitration settings and one or more non-arbitration settings, wherein dividing the settings of the plurality of settings into the one or more arbitration settings includes: Determine whether the settings among the multiple settings need to be consistent across all multi-user experience participants.

35. The apparatus of claim 19, wherein enforcing a single arbitration setting among the one or more arbitration settings comprises: A portion of the adapted multi-user experience is generated by a generative model, wherein generating the portion of the adapted multi-user experience includes: merging a combination of settings from two or more multi-user experience participants for the single arbitration setting in the one or more arbitration settings.

36. The apparatus of claim 19, wherein enforcing a single arbitration setting among the one or more arbitration settings comprises at least one or more of the following: Directly manipulate the adapted multi-user experience to match the single arbitration setting; Based on the single arbitration setting, a portion of the adapted multi-user experience is selected from the database; The adapted multi-user experience is divided into two parts: a first partition of the adapted multi-user experience includes the first setting associated with the first multi-user experience participant of the single arbitration setting, and a second partition of the adapted multi-user experience includes the second setting associated with the second multi-user experience participant of the single arbitration setting; or a fused portion of the adapted multi-user experience is generated, wherein the fused portion of the adapted multi-user experience includes the first setting associated with the first multi-user experience participant of the single arbitration setting and the second setting associated with the second multi-user experience participant of the single arbitration setting.

Images