Aavatar representation and audio generation

Abstract

An apparatus includes a memory and one or more processors configured to process image data corresponding to a face of a user to generate face data. The one or more processors are configured to process the sensor data to generate feature data, and generate a representation of the avatar based on the facial data and the feature data. The one or more processors are also configured to generate an audio output for the avatar based on the sensor data.

Description

[0001] I. Cross-Reference to Related Applications

[0002] This application claims the benefit of priority from commonly owned U.S. non-provisional patent application No. 17 / 930,257, filed on September 7, 2022, the entire contents of which are expressly incorporated herein by reference.

[0003] II. Technical field

[0004] The present disclosure generally relates to generating representations of avatars.

[0005] III. Related Technology

[0006] Technological advances have enabled smaller and more powerful computing devices. For example, there are currently a variety of portable personal computing devices, including wireless phones (such as mobile and smart phones, tablet computers, and laptop computers) that are small, lightweight, and easy for users to carry. These devices can communicate voice and data packets over wireless networks. In addition, many such devices incorporate additional functions, such as digital still cameras, digital video cameras, digital recorders, and audio file players. In addition, such devices can process executable instructions that can be used to access the Internet, including software applications, such as web browser applications. As such, these devices can include significant computing capabilities.

[0007] One common use of such devices is to enable a user to interact with one or more other users or computer-generated users via an avatar. For example, an avatar may represent a user in a multiplayer online game, a virtual conference, or other application in which participants may interact with one another. While in some cases, an avatar may be used to mimic the appearance of the user represented by the avatar, such as a photo-realistic avatar, in other cases, as a non-limiting example, the avatar may not mimic the appearance of the user, but may have the appearance of a fictional character or fantasy creature.

[0008] Regardless of whether the avatar mimics the user's appearance, it is often beneficial to increase the perceived realism of the avatar, such as by having the avatar accurately convey emotional aspects associated with the user to participants interacting with the avatar. For example, if the facial expressions of a photorealistic avatar do not represent the user's face with sufficient accuracy, participants viewing the avatar may become uneasy due to experiencing the avatar as almost but not quite lifelike, a phenomenon that has been referred to as the "uncanny valley." Similarly, if the facial expressions associated with the avatar speaking are inconsistent with the avatar's voice, the perceived realism of the avatar is also affected. Therefore, the experience of participants interacting with a user's avatar, whether photorealistic or fanciful, can be improved by improving the accuracy with which the avatar conveys such expressions and emotions of the user.

[0009] IV. Summary of the Invention

[0010] According to a specific implementation of the present disclosure, a device includes a memory configured to store instructions. The device also includes one or more processors configured to process image data corresponding to a user's face to generate facial data. The one or more processors are configured to process sensor data to generate feature data. The one or more processors are also configured to generate a representation of an avatar based on the facial data and the feature data. The one or more processors are also configured to generate an audio output for the avatar based on the sensor data.

[0011] According to another specific implementation of the present disclosure, a method for avatar generation includes processing image data corresponding to a user's face at one or more processors to generate facial data. The method includes processing sensor data at the one or more processors to generate feature data. The method includes generating a representation of the avatar based on the facial data and the feature data at the one or more processors. The method also includes generating an audio output for the avatar based on the sensor data at the one or more processors.

[0012] According to another specific implementation of the present disclosure, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to process image data corresponding to a user's face to generate facial data. When executed by the one or more processors, the instructions cause the one or more processors to process sensor data to generate feature data. When executed by the one or more processors, the instructions cause the one or more processors to generate a representation of an avatar based on the facial data and the feature data. When executed by the one or more processors, the instructions also cause the one or more processors to generate an audio output for the avatar based on the sensor data.

[0013] According to another specific implementation of the present disclosure, a device includes a component for processing image data corresponding to a user's face to generate facial data. The device includes a component for processing sensor data to generate feature data. The device includes a component for generating a representation of an avatar based on the facial data and the feature data. The device also includes a component for generating an audio output for the avatar based on the sensor data.

[0014] Other aspects, advantages, and features of the present disclosure will become apparent upon review of the entire application, including the Brief Description of the Drawings, Detailed Description, and Claims.

[0015] V. Description of the Figures

[0016] Figure 1 is a block diagram of certain illustrative aspects of a system configured to generate a representation of an avatar according to some examples of the present disclosure.

[0017] Figure 2 is a block diagram of another illustrative aspect of a system configured to generate a representation of an avatar according to some examples of the present disclosure.

[0018] Figure 3 is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate adjusted facial data corresponding to an avatar facial expression based on audio data, according to some examples of the present disclosure.

[0019] Figure 4 is a block diagram of another illustrative aspect of components that may be included in a system configured to generate adjusted facial data corresponding to an avatar facial expression based on audio data, according to some examples of the present disclosure.

[0020] Figure 5 is a block diagram of another illustrative aspect of components that may be included in a system configured to generate adjusted facial data corresponding to an avatar facial expression based on audio data, according to some examples of the present disclosure.

[0021] Figure 6 is a block diagram of another illustrative aspect of components that may be included in a system configured to generate adjusted facial data corresponding to an avatar facial expression based on audio data, according to some examples of the present disclosure.

[0022] Figure 7 is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate adjusted facial data corresponding to an avatar's facial expression based on audio data and image data, according to some examples of the present disclosure.

[0023] Figure 8 is a block diagram of another illustrative aspect of components that may be included in a system configured to generate adjusted facial data corresponding to an avatar facial expression based on audio data and image data, according to some examples of the present disclosure.

[0024] Fig. 9 is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate adjusted facial data corresponding to an avatar's facial expressions in conjunction with a user profile, according to some examples of the present disclosure.

[0025] Fig.10 is a block diagram of another illustrative aspect of components that may be included in a system configured to generate adjusted facial data corresponding to an avatar facial expression in conjunction with a user profile, according to some examples of the present disclosure.

[0026] Fig.11is a diagram of another illustrative aspect of components that may be included in a system configured to generate adjusted facial data corresponding to an avatar facial expression in conjunction with a user profile, according to some examples of the present disclosure.

[0027] Fig.12 is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate adjusted facial data corresponding to an avatar facial expression based on speech prediction according to some examples of the present disclosure.

[0028] Fig.13 is a block diagram of another illustrative aspect of components that may be included in a system configured to generate adjusted facial data corresponding to an avatar facial expression based on speech prediction according to some examples of the present disclosure.

[0029] Fig.14 is a block diagram of another illustrative aspect of components that may be included in a system configured to generate adjusted facial data corresponding to an avatar facial expression based on speech prediction according to some examples of the present disclosure.

[0030] Fig.15 is a diagram of certain illustrative aspects of a facial data adjuster that may be included in a system configured to generate adjusted facial data, according to some examples of the present disclosure.

[0031] Fig.16 is a diagram of specific illustrative aspects of combining representations of multimodal data that may be included in a system configured to generate adjusted facial data, according to some examples of the present disclosure.

[0032] Fig.17 is a diagram of another illustrative aspect of combining representations of multimodal data that may be included in a system configured to generate adjusted facial data according to some examples of the present disclosure.

[0033] Fig.18 is a diagram of another illustrative aspect of combining representations of multimodal data that may be included in a system configured to generate adjusted facial data according to some examples of the present disclosure.

[0034] Fig.19 is a diagram of another illustrative aspect of combining representations of multimodal data that may be included in a system configured to generate adjusted facial data according to some examples of the present disclosure.

[0035] Fig. 20 is a diagram of another illustrative aspect of combining representations of multimodal data that may be included in a system configured to generate adjusted facial data according to some examples of the present disclosure.

[0036] Fig.21 is a diagram of another illustrative aspect of combining representations of multimodal data that may be included in a system configured to generate adjusted facial data according to some examples of the present disclosure.

[0037] Fig. 22 is a block diagram of certain illustrative aspects of a system configured to generate adjusted facial data corresponding to an avatar's facial expression based on semantic context associated with motion sensor data, according to some examples of the present disclosure.

[0038] Fig.23 is a block diagram of certain illustrative aspects of a system configured to generate a representation of an avatar and audio associated with the avatar, according to some examples of the present disclosure.

[0039] Fig.24 is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate a representation of an avatar and audio associated with the avatar, according to some examples of the present disclosure.

[0040] Fig.25 is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate a representation of an avatar and audio associated with the avatar, according to some examples of the present disclosure.

[0041] Fig.26 is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate a representation of an avatar and audio associated with the avatar, according to some examples of the present disclosure.

[0042] Fig. 27 is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate a representation of an avatar and audio associated with the avatar, according to some examples of the present disclosure.

[0043] Fig.28 is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate a representation of an avatar and audio associated with the avatar, according to some examples of the present disclosure.

[0044] Fig.29 is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate a representation of an avatar and audio associated with the avatar, according to some examples of the present disclosure.

[0045] Fig.30 is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate a representation of an avatar and audio associated with the avatar, according to some examples of the present disclosure.

[0046] Fig.31is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate a representation of an avatar and audio associated with the avatar, according to some examples of the present disclosure.

[0047] Fig.32 is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate a representation of an avatar and audio associated with the avatar, according to some examples of the present disclosure.

[0048] Fig.33 is a block diagram of certain illustrative aspects of components that may be included in a system configured to generate a representation of an avatar and audio associated with the avatar, according to some examples of the present disclosure.

[0049] Fig.34 is a block diagram of specific illustrative aspects of components that may be included in a system configured to generate a representation of an avatar associated with an avatar, according to some examples of the present disclosure.

[0050] Fig.35 An example of an integrated circuit including a sensor-based avatar generator according to some examples of the present disclosure is illustrated.

[0051] Fig.36 is a diagram of a mobile device including a sensor-based avatar generator according to some examples of the present disclosure.

[0052] Fig.37 is a diagram of a headset including a sensor-based avatar generator according to some examples of the present disclosure.

[0053] Fig.38 is a diagram of a wearable electronic device including a sensor-based avatar generator according to some examples of the present disclosure.

[0054] Fig.39 is a diagram of a voice-controlled speaker system including a sensor-based avatar generator according to some examples of the present disclosure.

[0055] Fig.40 is a diagram of a camera including a sensor-based avatar generator according to some examples of the present disclosure.

[0056] Fig.41 is a diagram of an extended reality headset (such as a virtual reality, mixed reality, or augmented reality headset) including a sensor-based avatar generator according to some examples of the present disclosure.

[0057] Fig.42 is a diagram of a mixed reality or augmented reality eyewear device including a sensor-based avatar generator according to some examples of the present disclosure.

[0058] Fig.43 is a diagram of an earbud headset including a sensor-based avatar generator according to some examples of the present disclosure.

[0059] Fig.44 is a diagram of a first example of a vehicle including a sensor-based avatar generator according to some examples of the present disclosure.

[0060] Fig.45 is a diagram of a second example of a vehicle including a sensor-based avatar generator according to some examples of the present disclosure.

[0061] Fig.46 is a diagram of a specific implementation of a method for avatar generation according to some examples of the present disclosure.

[0062] Fig.47 is a diagram of another specific implementation of the method of avatar generation according to some examples of the present disclosure.

[0063] Fig.48 is a block diagram of a specific illustrative example of a device operable to generate adjusted facial data corresponding to an avatar facial expression based on semantic context associated with motion sensor data in accordance with some examples of the present disclosure.

[0064] VI. Specific Implementation Methods

[0065] Systems and methods for generating avatar facial expressions are disclosed. Because interaction with avatars whose facial expressions do not accurately convey aspects such as emotions can be disconcerting, such avatars impair the user experience. By improving the perceived realism of avatars, such as by improving the avatar's ability to convey emotional aspects and facial expressions of the user represented by the avatar, the experience of a user interacting with the avatar can be improved.

[0066] Conventional camera-based avatar solutions often fail to accurately simulate the characteristics and movements of a human face. For example, some camera-based solutions require extensive registration of a user's face (e.g., requiring the user to provide a series of pictures or videos) as a starting point for recreating the user's face for use as an avatar. However, even after registration, such camera-based solutions often fail to provide sufficient realism in reproducing the user's actual facial behavior because the previously collected registration data fails to account for the many possible forms that the user's real face may take during the myriad social situations, emotional reactions, facial expressions, etc. that the user may exhibit.

[0067] In other conventional solutions, a camera attached to a head mounted display (HMD) points downward and captures the characteristics, movements, and behaviors of parts of the face in an attempt to animate the avatar. However, due to the limited field of view of the camera, the camera cannot capture all aspects of the user's face, so the resulting avatar usually lacks sufficient realism.

[0068] The disclosed systems and methods enable the creation of more realistic representations of user facial behaviors than the conventional solutions described above. For example, the disclosed systems and methods enable improved realism of facial parts (e.g., eyes, nose, skin, lips, etc.), facial expressions (e.g., smiling, laughing, crying, etc.), and emotional states involving multiple parameters of the face to consistently convey accurate emotions (e.g., happy, sad, angry, etc.).

[0069] According to some aspects, sensor data associated with a user (such as audio data representing the user's voice, image data representing one or more portions of the user's face, motion data corresponding to movement of the user or the user's head, or a combination thereof) is used to determine a semantic context associated with such data. For example, the semantic context may correspond to the meaning of a word, phrase, or sentence spoken (or predicted to be spoken) by the user, which may be used to inform the facial expression of the avatar. In some examples, the semantic context may be based on characteristics of a conversation in which the user is participating, such as the type of relationship between the conversation participants (e.g., business, friends, family, parent / child, etc.), the social context of the conversation (e.g., professional, friendly, etc.), or both.

[0070] In some examples, the semantic context may correspond to an emotion detected based on the user's voice, based on image data of the user's face, or a combination of both. In some examples, the semantic context may be associated with an audio event detected in the audio data, such as the sound of breaking glass near the user.

[0071] According to some aspects, the facial expressions of the avatar are modified to more accurately represent the emotions or expressions of the user based on the semantic context. For example, facial data representing the avatar may be generated from an image of a portion of the user's face captured by a camera of the HMD, but as explained above, such facial data may not be sufficient to generate sufficiently realistic facial expressions of the avatar. However, the facial data may be adjusted based on feature data derived from the sensor data so that the avatar's facial expressions are more realistic in view of the semantic context.

[0072] According to some aspects, the disclosed systems and methods enable prediction of a user's future expressions or emotions based on semantic context. For example, a future speech prediction of the most likely word to be spoken by the user can be generated, which can enable prediction of the facial expression involved in pronouncing the word in addition to predicting the emotional tone associated with the meaning of the word. As another example, a user's future emotions or expressions can be predicted based on detected audio events (such as breaking glass or the sound of a car horn). Accurate future predictions of facial expressions, emotions, and the like enable generation of transitions between avatar expressions with reduced latency and improved accuracy.

[0073] According to some aspects, the disclosed systems and methods include modifying a user's voice to generate an audio output for an avatar. For example, the audio output may be generated by capturing the user's voice via sensor data and performing a voice conversion to a voice associated with a virtual avatar or adjusting the user's voice to make it more understandable, more comfortable, etc. According to some aspects, because such modification may cause the avatar's facial expressions to become misaligned with the avatar's voice, the avatar facial data is adjusted to more accurately correspond to the avatar's voice, which may increase the perceived accuracy and realism of the avatar's facial expressions.

[0074] Improving the realism of the avatar's facial expressions improves the user experience for participants interacting with the avatar. In addition, future predictions of speech, expression, or emotion can improve the accuracy and reduce the latency associated with generating the avatar's facial expressions. Other advantages and examples of applications in which the disclosed technology can be used are described in more detail below with reference to the accompanying drawings.

[0075] Certain aspects of the present disclosure are described below with reference to the accompanying drawings. In this specification, common features are designated by common reference numerals. As used herein, various terms are used only for the purpose of describing a particular implementation and are not intended to limit the implementation. For example, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, some features described herein are singular in some implementations and plural in other implementations. For the purpose of illustration, Figure 1 Depicts a system including one or more processors ( Figure 1 116), indicating that in some implementations, the device 102 includes a single processor 116, while in other implementations, the device 102 includes multiple processors 116. For ease of reference herein, such features are generally introduced as "one or more" features and are subsequently referred to in the singular unless aspects are described that relate to multiple ones of the features.

[0076] It will be further understood that the term "including" can be used interchangeably with "comprising". In addition, it will be understood that the term "wherein" can be used interchangeably with "wherein", and the term "based on" can be used interchangeably with "based at least in part on", or "based in part on". As used herein, "exemplary" can indicate examples, specific implementations and / or aspects, and should not be understood as restrictive or indicating preferences or preferred specific implementations. As used herein, ordinal terms (e.g., "first", "second", "third", etc.) used to modify elements (such as, structures, components, operations, etc.) do not themselves indicate any priority or order of the element relative to another element, but only distinguish the element from another element with the same name (but using ordinal terms). As used herein, the term "set" refers to one or more specific elements in a specific element, and the term "plurality" refers to multiple (e.g., two or more) specific elements.

[0077] As used herein, "coupling" may include "communicatively coupled", "electrically coupled" or "physically coupled", and may also (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicatively coupled, electrically coupled or physically coupled) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof), etc. As illustrative, non-limiting examples, two electrically coupled devices (or components) may be included in the same device, or in different devices, and may be connected via electronic devices, one or more connectors, or inductive coupling. In some specific implementations, two devices (or components) that are communicatively coupled (such as electrically connected) may directly or indirectly transmit and receive signals (e.g., digital signals or analog signals) via one or more wires, buses, networks, etc. As used herein, "direct coupling" may include two devices that are coupled (e.g., communicatively coupled, electrically coupled or physically coupled) without intermediate components.

[0078] In the present disclosure, terms such as "determine", "calculate", "estimate", "shift", "adjust", etc. may be used to describe how to perform one or more operations. It should be noted that such terms should not be interpreted as limiting, and other technologies may be used to perform similar operations. In addition, as mentioned herein, "generate", "calculate", "estimate", "use", "select", "access", and "determine" may be used interchangeably. For example, "generating", "calculating", "estimating" or "determining" a parameter (or signal) may refer to actively generating, estimating, calculating or determining the parameter (or signal), or may refer to using, selecting or accessing a parameter (or signal) that has already been generated (such as by another component or device).

[0079] refer to Figure 1, discloses certain illustrative aspects of a system 100 configured to generate data corresponding to an avatar's facial expression. The system 100 includes a device 102, which includes a memory 112 and one or more processors 116. In some specific implementations, as illustrative non-limiting examples, the device 102 corresponds to a computing device such as a mobile phone, a laptop computer, a server, a headset or other head-mounted device, or a vehicle.

[0080] The one or more processors 116 include a feature data generator 120 and a facial data adjuster 130. According to some specific implementations, one or more of the components of the one or more processors 116 may be implemented using dedicated circuits. As a non-limiting example, one or more of the components of the one or more processors 116 may be implemented using a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc. According to another specific implementation, one or more of the components of the one or more processors 116 may be implemented by executing instructions 114 stored in the memory 112. For example, the memory 112 may be a non-transitory computer-readable medium storing instructions 114 that can be executed by the one or more processors 116 to perform the operations described herein.

[0081] The one or more processors 116 are configured to process the sensor data 106 to generate feature data 124. For purposes of illustration, the feature data generator 120 is configured to process the sensor data 106 to determine a semantic context 122 associated with the sensor data 106. As used herein, "semantic context" refers to one or more meanings or emotions that can be determined based on the sensor data 106 or predicted from the sensor data. In the illustrative example where the sensor data 106 includes audio data, the semantic context 122 is based on the meaning of the speech represented in the audio data, based on the emotions associated with the speech represented in the audio data, based on audio events detected in the audio data, or a combination thereof. In some specific implementations, the sensor data 106 includes image data (e.g., video data), and the semantic context 122 is based on the emotions associated with the expressions on the user's face represented in the image data. In some examples, the sensor data 106 includes motion sensor data, and the semantic context 122 is based on the motion sensor data. Example references for determining semantic context 122 based on audio data, image data, motion data, or a combination thereof Figure 2 Further description.

[0082] Feature data 124 includes information that enables facial data adjuster 130 to adjust one or more aspects of the expression of avatar 154. In some examples, feature data 124 indicates an expression condition, emotion, audio event, or other information conveying or based on semantic context 122. In some examples, feature data 124 includes codes, audio features, voice tags / phonemes, audio event tags, emotion indicators, expression indicators, or a combination thereof, as described in further detail below.

[0083] In some implementations, the feature data generator 120 determines a semantic context 122 based on processing the sensor data 106, and generates the feature data 124 based on the semantic context 122. For purposes of illustration, in some examples, the feature data generator 120 includes an indicator or encoding of the semantic context 122 in the feature data 124. In other examples, the feature data generator 120 generates an expression condition (e.g., facial expression information, emotion data, etc.) based on the semantic context 122, and includes the expression condition in the feature data 124. However, in other implementations, the feature data generator 120 does not explicitly determine the semantic context 122. For example, the feature data generator 120 may include one or more feature generation models that are configured to bypass explicitly determining the semantic context 122 and directly map the sensor data 106 to values ​​of the feature data 124 that are appropriate for the semantic context 122 implicit in the sensor data 106. As an example, feature data generator 120 may process audio data representing a user's voice having a happy tone, and thus feature data generator 120 may output feature data 124 encoding or indicating a facial expression associated with conveying happiness without explicitly determining that semantic context 122 corresponds to "happy."

[0084] The one or more processors 116 are configured to generate adjusted facial data 134 based on the feature data 124. For example, the facial data adjuster 130 may receive facial data 132, such as data corresponding to a coarse grid representing the face of the user 108 and used as a reference for generating the face of the avatar 154. In some implementations, the facial data 132 is generated based on image data from one or more cameras that capture portions of the face of the user 108, and the face of the avatar 154 is generated to substantially match the face of the user 108 (such as a photo-realistic avatar). Alternatively, the face of the avatar 154 may be based on the face of the user 108, but may include one or more modifications (e.g., adding or removing facial hair or tattoos, changing hairstyle, eye color or skin tone, etc.), such as based on user preferences. In some implementations where the avatar 154 corresponds to a virtual avatar selected by the user (such as a fantasy computer-generated character or creature), the facial data 132 of the avatar 154 may be generated by the one or more processors 116 (e.g., via a game engine) or retrieved from the memory 112.

[0085] The adjusted facial data 134 corresponds to the avatar's facial expression 156 based on the semantic context 122. For example, while the facial data 132 may not include sufficient information to accurately reproduce the expression or emotion exhibited by the user 108, the feature data 124 generated based on the sensor data 106 may provide additional information about the expression or emotion of the user 108. For example, the feature data 124 may directly include an indication of the semantic context 122, or may include expression data, emotion data, or both based on the semantic context 122.

[0086] In some implementations, facial data adjuster 130 generates adjusted facial data 132 by modifying facial data 134 based on feature data 124. In some implementations, facial data adjuster 130 generates adjusted facial data 132 by merging facial data 134 with facial expression data corresponding to feature data 124. In some implementations, such as reference Figures 15 to 19 As described, the facial data adjuster 130 includes a neural network having an encoder portion that processes the facial data 132 and is coupled to a decoder portion. The output of the encoder portion is combined with the feature data 124 at the decoder portion, such as via concatenation or fusion in a latent space, which causes the decoder portion to generate adjusted facial data 134.

[0087] During operation, in a particular example, the feature data generator 120 processes the user's voice, speech, or both, and detects emotions and behaviors that may correspond to the semantic context 122. Such emotions and behaviors may be encoded in the feature data 124 and used by the facial data adjuster 130 to generate adjusted facial data 134. For example, the facial data adjuster 130 may cause the adjusted facial data 134 to represent or express the emotions or behaviors indicated in the feature data 124. For purposes of illustration, when a laugh of the user 108 has been identified, the facial data adjuster 130 may cause the avatar 154's mouth to smile wider, tighten the eyes, add or enlarge dimples, etc. In another particular example, the semantic context 122 may correspond to a type of relationship between the user 108 and another participant engaged in a conversation with the user 108 (e.g., family, intimate, professional, formal, friends, etc.), and the facial data adjuster 130 may cause the avatar facial expression 156 to exhibit one or more characteristics appropriate to the type of relationship (e.g., by increasing focus, reducing or amplifying emotional expressions, etc.). Other examples of generating adjusted facial data 134 based on semantic context 122 are provided with reference to the various implementations described below.

[0088] Optionally, one or more sensors 104 are coupled to or integrated into the device 102 and are configured to generate sensor data 106. In some examples, the one or more sensors 104 include one or more microphones configured to capture the voice of the user 108, background audio, or both. In some examples, the one or more sensors 104 include one or more cameras configured to capture the facial expressions of the user 108, one or more other visual characteristics of the user 108 (e.g., posture, gesture, movement, etc.), or a combination thereof. In some examples, the one or more sensors 104 include one or more motion sensors, such as an inertial measurement unit (IMU) or other sensors configured to detect movement, acceleration, orientation, or a combination thereof. In an exemplary implementation, the one or more processors 116 are integrated into an extended reality ("XR") device, which also includes one or more microphones, multiple cameras, and an IMU.

[0089] Alternatively or additionally, the one or more processors 116 may receive at least a portion of the sensor data 106 from recorded sensor data stored at the memory 112, from a second device (not shown) via an optional modem 140, or a combination thereof. For example, the device 102 may correspond to a mobile phone or a computer device (e.g., a laptop or a server), and the one or more sensors 104 may be coupled to or integrated in an extended reality ("XR") headset, such as a virtual reality ("VR"), augmented reality ("AR"), or mixed reality ("MR") headset device (e.g., HMD) worn by the user 108. In some scenarios, the device 102 uses a wired connection, a wireless connection (e.g., Bluetooth ® (a registered trademark of the Bluetooth SIG, Inc., Washington, D.C.) connection) or both to receive sensor data 106. In some examples, device 102 may use a low-power protocol (e.g., Bluetooth ® In some examples, the wireless connection corresponds to sending and receiving signals according to an IEEE 802.11-type (e.g., WiFi) wireless local area network or one or more other wireless radio frequency (RF) communication protocols.

[0090] Optionally, device 102 may include or be coupled to a user interface device, such as display device 150 or other visual user interface device, configured to display a representation 152 of an avatar 154 having an avatar facial expression 156 based on the adjusted facial data 134. For example, one or more processors 116 may be configured to generate the representation 152 of the avatar 154 based on the adjusted facial data 134 and having an appropriate data format to be sent to and displayed at display device 150. In other specific implementations, device 102 may alternatively (or additionally) transmit the representation 152 of the avatar 154 to a second device (e.g., a server, or another user's headset device or computer device) to enable one or more other geographically remote users to view the avatar 154.

[0091] By generating adjusted facial data 134 based on feature data 124 , the resulting avatar facial expressions 156 may more accurately or realistically convey the expressions or emotions of user 108 than may be generated from facial data 132 alone, thereby improving the user experience.

[0092] Figure 2Another particular illustrative aspect of a system 200 configured to generate data corresponding to an avatar's facial expression is depicted. The system 200 includes a device 102 and optionally includes a display device 150, a sensor 104, or both. The sensor 104 optionally includes one or more microphones 202, one or more cameras 206, and one or more motion sensors 210. The one or more processors 116 include a feature data generator 120, a facial data generator 230, a facial data adjuster 130, and an avatar generator 236.

[0093] The one or more microphones 202 are configured to generate audio data 204 included in the sensor data 106. For example, the one or more microphones 202 may include a microphone configured to capture the voice of the user 108 (e.g., a directional microphone), one or more microphones configured to capture ambient sounds near the user 108 (e.g., one or more directional or omnidirectional microphones), or a combination thereof. In an implementation where the one or more microphones 202 are omitted, as an illustrative example, the audio data 204 may be received from another device (e.g., a headphone device or other device including a microphone) via the modem 140 or retrieved from a memory (e.g., the memory 112 or another memory, such as a network storage device).

[0094] The one or more cameras 206 are configured to generate image data 208 included in the sensor data 106. In the illustrated implementation, the image data 208 includes multiple regions of the user's face captured by corresponding cameras in the one or more cameras 206. The image data 208 includes first image data 208A, which includes a representation of a first portion of the user's face, illustrated as a profile view of the region of the user's left eye. The image data 208 includes second image data 208B, which includes a representation of a second portion of the user's face, illustrated as a profile view of the region of the user's right eye. The image data 208 includes third image data 208C, which includes a representation of a third portion of the user's face, illustrated as a front view of the region of the user's mouth.

[0095] For purposes of illustration, one or more cameras 206 may be integrated into a head-mounted device such as an XR headset or glasses, and various cameras may be positioned at various locations of the XR headset or glasses (e.g., at the user's temples and in front of the user's nose) to enable the capture of image data 208A, 208B, and 208C without substantially protruding from the XR headset or glasses or detracting from the aesthetic appearance of the XR headset or glasses. However, it should be understood that in other implementations, the image data 208 may include more than three portions of the user's face or less than three portions of the user's face, one or more other portions of the user's face (instead of or in addition to the portions illustrated), or a combination thereof. In implementations where one or more cameras 206 are omitted, as an illustrative example, the image data 208 may be received from another device (e.g., a headset device or other device including a camera) via the modem 140 or retrieved from a memory (e.g., the memory 112 or another memory, such as a network storage device).

[0096] The one or more processors 116 include a facial data generator 230 configured to process the image data 208 corresponding to a person's face to generate the facial data 132. In an illustrative, non-limiting example, the facial data generator 230 includes a three-dimensional morphable model (3DMM) encoder configured to input the image data 208 and generate the facial data 132 as a coarse mesh representation of the user's face. Although the image data 208 is described as including the face of a user (e.g., the user 108 wearing the XR headset or glasses), in other specific implementations, the image data 208 may include the faces of one or more persons who are not "users" of the device 102, such as when the one or more cameras 206 capture the faces of multiple persons (e.g., the user 108 and one or more other persons near the user 108), and the facial data 132 is generated based on the faces of the "non-user" persons in the image data 208.

[0097] The facial data adjuster 130 is configured to generate adjusted facial data 134 based on the feature data 124 and further based on the facial data 132. For example, the facial data adjuster 130 may include a deep learning architecture neural network. In an illustrative, non-limiting example, the facial data adjuster 130 corresponds to a skin U-Net, which includes a convolutional neural network contracting path or encoder followed by a convolutional network expanding path or decoder. The contracting path or encoder may include repeated applications (e.g., layers) of convolutions, each followed by a rectified linear unit (ReLU) and a maximum pooling operation, which reduces spatial information while increasing feature information. The expanding path or decoder may include repeated applications (e.g., layers) of up-convolutions and concatenations with high-resolution features from the contracting path, from the feature data 124, or both.

[0098] The avatar generator 236 is configured to generate a representation 152 of an avatar 154 having an avatar facial expression 156 based on the adjusted facial data 134. In an illustrative, non-limiting example, the avatar generator 236 includes a U-Net implementation, such as an NRA U-Net.

[0099] The feature data generator 120 includes an audio unit 222 that is configured to process the audio data 204 and generate an audio representation 224 based on the audio data 204, and that may be indicative of or used to determine the semantic context 122. Although not illustrated, in some implementations, the feature data generator 120 is configured to perform pre-processing of the audio data 204 into a format that is more useful for processing at the audio unit 222. In some implementations, as illustrative, non-limiting examples, the audio unit 222 includes a deep learning neural network, such as an audio variational autoencoder (VAE), that is trained to identify characteristics of speech in the audio 204, and the audio representation 224 includes one or more of expression conditions, audio phonemes, or a mel-spectrogram. Alternatively or additionally, the audio unit 222 is configured to determine one or more signal processing speech representations, such as mel-frequency cepstral coefficients (MFCCs), MFCCs and pitch information, spectrogram information, or a combination thereof, as described in reference to Figure 4 Alternatively or additionally, the audio unit 222 is configured to determine one or more speech representations or labels based on automatic speech recognition (ASR), such as reference Figure 5 Alternatively or additionally, as an illustrative non-limiting example, the audio unit 222 is configured to determine one or more deep learned speech representations from self-supervised learning, such as based on Wav2vec, VQ-Wav2vec, Wav2vec2.0 or Hubert implementations, such as reference Figure 6 Further described.

[0100] In the illustrative example, the semantic context 122 is based on the meaning of the speech 258 represented in the audio data 204 (e.g., the emotional content associated with the user's speech 258). In some examples, the semantic context 122 is based on the meaning of the words 260 detected in the speech 258. In some examples, the semantic context 122 is based on the meaning of at least one phrase or sentence 262 detected in the speech 258. For purposes of illustration, the audio unit 222 may include a dictionary or other data structure or model that maps words, phrases, sentences, or combinations thereof to meanings associated with the words, phrases, or sentences. As used herein, the "meaning" associated with a word, phrase, or sentence may include an emotion associated with the word, phrase, or sentence. For purposes of illustration, as illustrative, non-limiting examples, the audio unit 222 may scan the audio data 204 for specific keywords or phrases that convey specific contexts or emotions, such as "budget," "bandwidth," "action items," and "schedule" associated with business language, "great," "excellent," and "can't wait to see you" associated with happiness, and "oh no," "sorry," and "that's too bad" associated with sadness.

[0101] In some examples, speech 258 includes at least a portion of a conversation 264, and semantic context 122 is based on characteristics of conversation 264. For purposes of illustration, in some implementations, the characteristics include the type of relationship 266 between user 108 and another participant participating in conversation 264 (e.g., family, intimate, professional, formal, casual, etc.). In some implementations, characteristics of conversation 264 include social context 268 of conversation 264 (e.g., at work, at home, shopping, traveling, etc.). Relationship 266 and social context context 268 can be used to determine the type of contact (e.g., the person involved in the conversation). According to one aspect, knowing the type of contact can help feature data generator 120 predict the type of conversation that may occur, which may affect the type of facial expression that the user's avatar 154 may make. In some examples, the type of contact is determined based on a list of contacts in device 102. As illustrative, non-limiting examples, "business" type contacts may include co-workers, clients / customers, or suppliers; "friend" type contacts may include purely spiritual friendships, romantic, elderly, or children; and "family" type contacts may include elderly, adults, children, spouses, wives, and husbands. In some implementations, one or more processors 116 are configured to build a history of the user's interactions with various contacts, create a model of each contact, and predict the types of interactions that may occur in future interactions. Therefore, the resulting facial expressions of avatar 154 may be different for various contacts.

[0102] Optionally, the semantic context 122 is based on the emotions 270 associated with the speech 258 represented in the audio data 204. In the illustrative example, the one or more processors 116 are configured to process the audio data 204 to predict the emotions 270. For example, in addition to detecting emotions associated with the meaning of words, phrases, and sentences of the user's speech 258, the audio unit 222 may also include one or more machine learning models configured to detect audible emotions such as happiness, sadness, anger, fun, romance, seriousness, frustration, etc. based on the speaking characteristics of the user 108 (e.g., based on tone, pitch, rhythm, volume, etc.). The feature data generator 120 can be configured to associate specific facial expressions or characteristics with various audible emotions. In some specific implementations, the adjusted facial data 134 causes the avatar facial expression 156 to represent the emotions 270 (e.g., smiling to express happiness, narrowing eyes to express anger, widening eyes to express surprise, etc.).

[0103] In some examples, the semantic context 122 is based on audio events 272 detected in the audio data 204. For example, the audio unit 222 may include an audio event detector that may access a database (not shown) that includes models of different audio events, such as car horns, dog barks, alarms, etc. In certain aspects, an "audio event" may correspond to a specific audio signature or set of sound characteristics that may indicate an event of interest. In some implementations, audio events exclude speech, and thus detecting audio events is different from keyword detection or speech recognition. In some implementations, detection of audio events may include detection of specific types of voices (e.g., shouting, screaming, baby crying, etc.), without keyword detection or determining the content of the voice. In response to sound characteristics in the audio data 204 that match (or substantially match) a specific model, the audio event detector may generate audio event information indicating that the audio data 204 represents an audio event 272 associated with the specific model. As used herein, if the pitch and frequency components of the audio data 204 are within the thresholds of the pitch and frequency components of the particular sound model, then the sound characteristics in the audio data 204 can "match" the particular sound model. In some specific implementations, the audio unit 222 includes one or more classifiers that are configured to process the audio data 204 to determine the associated category from a plurality of categories supported by the one or more classifiers. In an example, the one or more classifiers operate in conjunction with the above-mentioned audio event model to determine the category (e.g., type, such as "dog barking", "glass breaking", "baby crying", etc.) of the sound represented in the audio data 204 and associated with the audio event 272. For example, one or more classifiers may include a neural network that has been trained using labeled sound data to distinguish between sounds corresponding to various categories, and the neural network is configured to process the audio data 204 to determine the specific category of the sound represented by the audio data 204 (or determine the probability of the sound belonging to the category for each category).

[0104] The semantic context 122 associated with the detected audio event may correspond to an emotion associated with the audio event, such as fear or surprise at “glass breaking,” sympathy or frustration at “baby crying,” etc. In other examples, the semantic context 122 associated with the detected audio event may correspond to other aspects, such as the location or environment of the user 108 (e.g., on a busy street, in an office, at a restaurant), which may be determined based on detecting the audio event 272.

[0105] Optionally, the sensor data 106 includes image data 208, and the feature data generator 120 includes an image unit 226 that is configured to generate facial expressions 228 based on the image data 208 and that may be indicative of or used to determine the semantic context 122. Although not illustrated, in some implementations, the feature data generator 120 is configured to perform pre-processing of the image data 208 into a format more useful for processing at the image unit 226. The image unit 226 may include one or more neural networks (e.g., a face part VAE) that are configured to specifically process the image data 208 to detect facial expressions and movements in the image data 208 with greater accuracy than the facial data generator 230. For example, because the facial data generator 230 may not be able to generate sufficiently accurate representations of the user's facial expressions to be perceived as realistic, the resulting adjusted facial data 134 may provide more accurate and realistic facial expressions of the avatar 154 by also processing the image data 208 using a neural network of the image unit 226 that is trained to specifically detect facial expressions and movements associated with speaking, conveying emotions, etc., such as near the eyes and mouth, and using such detected facial expressions and movements when generating the feature data 124. In a particular implementation, the facial expressions 228 include an indication of one or more expressions, movements, or other features of the user 108. The image unit 226 may detect facial expressions and movements in the image data 208, such as smiling, winking, grimacing, etc., while the user 108 is not speaking and would otherwise not be detected by the audio unit 222, thereby further enhancing the accuracy and realism of the avatar 154.

[0106] In some examples, semantic context 122 is based on emotion 270, and emotion 270 is associated with an expression on the user's face represented in image data 208, rather than or in addition to audible emotions detected in the user's voice or emotional content associated with user speech 258. In some implementations, audio data 204 and image data 208 are input to a neural network configured to detect emotion 270, such as a neural network ... Figure 7 Further described.

[0107] Optionally, the sensor data 106 includes motion sensor data 212, and the semantic context 122 is based on the motion sensor data 212. In some examples, the motion sensor data 212 is received from one or more motion sensors 210 coupled to or integrated in the device 102. For purposes of illustration, the one or more sensors 104 optionally include one or more motion sensors 210, such as one or more accelerometers, gyroscopes, magnetometers, inertial measurement units (IMUs), one or more cameras configured to detect user movement, one or more other sensors configured to detect movement, acceleration, orientation, or a combination thereof. For example, the motion sensor data 212 may include head tracker data associated with the movement of the user 108, such as a reference Fig. 22 Further described.

[0108] The feature data generator 120 may include a motion unit 238 configured to process the motion sensor data 212 and determine a motion representation 240 based on the motion sensor data 212, and may indicate or be used to determine the semantic context 122. Although not illustrated, in some implementations, the feature data generator 120 is configured to perform pre-processing of the motion sensor data 212 into a format more useful for processing at the motion unit 238. The motion unit 238 may be configured to identify head movements that indicate meaning or emotion, such as nodding (indicating agreement) or shaking the head (indicating disagreement), sudden shaking of the head (indicating surprise), etc. In another example, the motion sensor data 212 corresponds at least in part to the movement of a vehicle (e.g., a car) that the user 108 is operating or traveling in, and the motion unit 238 may be configured to identify vehicle movements that may provide context information. For example, motion sensor data 212 indicating sudden lateral or rotational movement (e.g., caused by a collision) or sudden deceleration (e.g., indicating an emergency stop) may be associated with fear or surprise, while relatively rapid acceleration may be associated with excitement.

[0109] Thus, the system 200 allows audio-based, and optionally camera-based and motion-based techniques to increase the realism of the avatar 154. In addition, the system 200 enables the use of predictive methods to reduce the latency associated with displaying the facial features of the avatar 154, and the reduced latency also increases the perceived realism of the avatar 154.

[0110] In some implementations, the system 200 uses one or more microphones 202 to capture / record the user's auditory behavior to identify sounds made by the user and identify emotions. The identified auditory information can inform the system 200 about the current behavior, emotion, or both that the user's face is showing, and the user's face also has a facial expression associated with the behavior or emotion. For example, if the user is laughing, the system 200 can exclude certain facial expressions that are not associated with laughter, and therefore can select from a smaller set of specific facial expressions when determining the avatar facial expression 156. Reducing the number of possible emotion types exhibited by the user 108 is advantageous because the system 200 can apply selected audio information to increase the accuracy of the facial expression of the avatar 154. For example, the system 200 can identify laughter in the audio data 204, and in response to identifying laughter, the system 200 can adjust the avatar facial expression 156 to make the mouth smile wider, tighten the eyes, enhance the crow's feet around the eyes, show dimples, etc.

[0111] Machine learning models associated with audible cues and emotions can be included in the system 200 (e.g., in the audio unit 222) to convert audio information into an accurate understanding of the user's emotions. Converting the user's auditory behavior (e.g., laughter) into associated emotions allows the system 200 to increase the accuracy of the avatar's facial expressions 156 with targeted (e.g., higher probability) information. For example, the audio unit 222 can create and extract audio codes associated with specific emotions (e.g., audio representations 224), and associate the audio codes with facial codes and expressions (e.g., in facial expressions 228 and feature data 124).

[0112] In some implementations, the device 102 may use the audio data 204 to enhance the quality of the avatar's expression without using any image data 208. For purposes of illustration, the device 102 may identify the various users participating in the interaction, and the previously registered avatars of the users using images or videos may be used as a baseline. However, the facial expressions of each avatar in the avatar may be based on audio input from the user, as described above. In some implementations, the device 102 may intermittently use one or more cameras 206 to enhance the audio data 204 to help create the avatar's expression. The above two implementations enable the use of cameras to be reduced, which results in power savings due to one or more cameras 206 being used less, turned off, or completely omitted from the system 200.

[0113] According to an aspect, processing of the audio data at audio unit 222 enables device 102 to determine the magnitude (from low to high, amplified or reduced) of the expression to be expressed by avatar 154. The context and volume of the speech, emotional response, or both presented in audio data 204 are examples of information that may be used to determine the magnitude of the expression expressed by avatar 154. For example, a loud laugh by user 108 may cause avatar 154 to display a larger open mouth, and other facial aspects associated with loud laughter may also be increased.

[0114] Using auditory information from the interaction between avatars (users) in a given interaction can achieve improved prediction, anticipation, or both of the user's facial expressions and increase the accuracy of the avatar's facial expressions. Device 102 can "listen" to the conversations (e.g., to detect keywords, determine the meaning of sentences, etc.) and behavioral interactions (e.g., the tone of voice, emotional reactions, etc.) of one or more avatars or users to create a model of the context of such conversations. In some specific implementations, device 102 can determine the semantic context 122 of the conversation and predict future emotions that may be exhibited by one or more of the participants in the conversation based on the model and the semantic context 122.

[0115] According to some aspects, the feature data generator 120 is configured to change one or more behaviors or characteristics of the avatar 154 to suit certain social situations. For example, the feature data generator 120 may determine to change such behaviors or characteristics based on an analysis of the conversation 264 (e.g., based on the relationship 266, the social context 268, or both), based on user preferences (e.g., according to preference settings in a user profile), or both. In some implementations, the feature data generator 120 includes one or more models or information that limit the range of expressions or emotions that can be expressed by the avatar 154 based on the semantic context 122 and the characteristics of the conversation 264. For example, the feature data generator 120 may adjust the feature data 124 to prevent the avatar 154 from displaying one or more emotions and expression extremes that the user 108 may exhibit during a conversation with the user's colleagues in a professional context, such as by preventing the avatar 154 from expressing some emotions such as anger or love and limiting the magnitude of other emotions such as boredom, excitement, or frustration. However, during a conversation with the same coworker in a casual setting, the feature data generator 120 may allow the avatar to exhibit a wider range of emotions and facial expressions.

[0116] In some implementations, user 108 may select a “personality setting” that indicates a user preference for the behavior of avatar 154 for a particular social situation, such as to ensure that avatar 154 is socially appropriate, or is “better” in some way than user 108 for a particular social situation (e.g., making avatar 154 look “calm,” “contemplative,” “excited,” or “interested,” etc.). For example, user 108 may set parameters before interacting with others (e.g., selecting a personality profile for avatar 154 via a user interface of device 102), and device 102 alters the avatar’s behavior based on the parameters. As a result, avatar 154 may not exactly match the behavior of user 108, but may instead exhibit “appropriate” behavior for the situation. For example, device 102 may prevent avatar 154 from expressing behavior that indicates user 108 is not paying attention during an interaction, such as when user 108 is checking the user’s phone (e.g., head tilted downward, eye focus reduced, facial expression suddenly changes, etc.). The feature data generator 120 may adjust the feature data 124 to cause the avatar 154 to express subtle visual facial cues to make the communication more comfortable, to exhibit polite behaviors that the user 108 does not actually express, and so on.

[0117] Figure 3 An example of a component 300 that may be implemented in a system configured to generate adjusted facial data corresponding to an avatar's facial expression, such as in device 102, is illustrated. Component 300 includes an audio network 310, an image unit 226, and a facial data adjuster 130.

[0118] The audio network 310 corresponds to a deep learning neural network, such as an audio variational autoencoder, that can be implemented in the feature data generator 120. The audio network 310 is trained to identify characteristics of speech in the audio 204 and determine an audio representation 324 that includes one or more of an expression condition, an audio phoneme, or a mel-spectrogram. In the illustrative example, the audio network 310 corresponds to or is included in the audio unit 222, and the audio representation 324 corresponds to or is included in Figure 2 The audio representation is 224.

[0119] The audio network 310 outputs one or more audio-based features 320 included in the feature data 124. In some examples, the one or more audio-based features 320 correspond to the audio representation 324, such as by including the audio representation 324 or an encoded version of the audio representation 324. Alternatively or additionally, the one or more audio-based features 320 may include one or more expression characteristics associated with the audio representation 324. For example, the audio network 310 may map specific values ​​of the audio representation 324 to one or more emotions or expressions. For purposes of illustration, the audio network 310 may be trained to identify a specific value or set of values ​​in the audio representation 324 as corresponding to laughter, and the audio network 310 may include in the audio-based features 320 an indication of one or more facial expressions associated with laughter, an indication of laughter itself (e.g., a code indicating laughter), or a combination thereof.

[0120] Similarly, the image unit 226 outputs one or more image-based features 322 included in the feature data 124. In some examples, the one or more image-based features 322 correspond to the facial representation 228, such as by including the facial representation 228 or an encoded version of the facial representation 228. Alternatively or additionally, the one or more image-based features 322 may include one or more expression characteristics associated with the facial expression 228. For example, the image unit 226 may map specific values ​​of the facial representation 228 to one or more emotions or expressions. For purposes of illustration, the image unit 226 may include a network that is trained to identify a specific value or set of values ​​in the facial representation 228 as corresponding to laughter, and the image unit 226 may include in the image-based features 322 an indication of one or more facial expressions associated with laughter, an indication of laughter itself (e.g., a code indicating laughter), or a combination thereof.

[0121] The one or more audio-based features 320 and the one or more image-based features 322 are combined (e.g., concatenated, fused, etc.) in the feature data 124 for use by the facial data adjuster 130 in generating the adjusted facial data 134, such as reference data. Figures 16 to 21 Further described.

[0122] Figure 4 An example of a component 400 that may be implemented in a system configured to generate adjusted facial data corresponding to an avatar's facial expression, such as in device 102, is illustrated. Component 400 includes a speech signal processing unit 410, an image unit 226, and a facial data adjuster 130. In a particular implementation, image unit 226 and facial data adjuster 130 operate substantially as described above.

[0123] The speech signal processing unit 410 includes one or more components configured to process the audio data 204 and detect, generate, or otherwise determine characteristics of speech in the audio data 204 and determine an audio representation 424. As an illustrative, non-limiting example, the audio representation 424 includes one or more signal processed speech representations, such as Mel-frequency cepstral coefficients (MFCCs), MFCCs, and pitch information or spectrogram information (e.g., a regular spectrogram, a logarithmic Mel-spectrogram, or one or more other types of spectrograms). In the illustrative example, the speech signal processing unit 410 corresponds to or is included in the audio unit 222, and the audio representation 424 corresponds to or is included in Figure 2 The audio representation is 224.

[0124] The speech signal processing unit 410 outputs one or more audio-based features 420 included in the feature data 124. In some examples, the one or more audio-based features 420 correspond to the audio representation 424, such as by including the audio representation 424 or an encoded version of the audio representation 424. Alternatively or additionally, the one or more audio-based features 420 may include one or more expression characteristics associated with the audio representation 424. For example, the speech signal processing unit 410, the audio unit 222, or both may map a particular value of the audio representation 424 to one or more emotions or expressions. For purposes of illustration, the speech signal processing unit 410 may include one or more components (e.g., one or more lookup tables, one or more trained networks, etc.) configured to identify a particular value or set of values ​​in the audio representation 424 as corresponding to laughter, and the speech signal processing unit 410 may include in the audio-based features 420 an indication of one or more facial expressions associated with laughter, an indication of laughter itself (e.g., a code indicating laughter), or a combination thereof.

[0125] The one or more audio-based features 420 and the one or more image-based features 322 from the image unit 226 are combined (e.g., concatenated, fused, etc.) in the feature data 124 for use by the facial data adjuster 130 in generating adjusted facial data 134, such as reference data. Figures 16 to 21 Further described.

[0126] The example implementation 450 depicts components that may be included in the speech signal processing unit 410 to perform speech signal processing. The pre-emphasis filter 454 is configured to perform pre-emphasis filtering on the speech signal 452 included in the audio data 204. The window block 456 performs a windowing operation on the output of the pre-emphasis filter 454, and the transform block 458 performs a transform operation (e.g., a fast Fourier transform (FFT)) on each of the windows. The resulting transformed data is processed at the Mel filter bank 460 and the log block 462. The transform block 464 (e.g., a discrete cosine transform (DCT) or an inverse FFT (IFFT)) performs an inverse transform on the output of the log block 462, and the resulting time domain data is processed at the Mel cepstrum block 466 to generate MFCC 480. A spectrogram 482 (e.g., a Mel log spectrogram) may be generated based on the frequency domain output of the log block 462. Pitch 484 may be determined based on autocorrelation block 470 that determines the autocorrelation (R) of a plurality of offset periods of the time domain output of window block 456 and find max R block 472 that determines the offset period associated with the maximum autocorrelation.

[0127] Figure 5 Another example of a component 500 that may be implemented in a system configured to generate adjusted facial data corresponding to an avatar's facial expression, such as in device 102, is illustrated. Component 500 includes an automatic speech recognition (ASR) based processing unit 510, image unit 226, and facial data adjuster 130. In a particular implementation, image unit 226 and facial data adjuster 130 operate substantially as described above.

[0128] The ASR-based processing unit 510 includes one or more components configured to process the audio data 204 and detect, generate, or otherwise determine characteristics of speech in the audio data 204 and determine an audio representation 524. As an illustrative, non-limiting example, the audio representation 524 includes one or more speech representations or labels based on automatic speech recognition (ASR), such as one or more phonemes, diphones, or triphones, associated stress or prosody (e.g., duration, pitch), one or more words, or a combination thereof. In the illustrative example, the ASR-based processing unit 510 corresponds to or is included in the audio unit 222, and the audio representation 524 corresponds to or is included in Figure 2 The audio representation is 224.

[0129] The ASR-based processing unit 510 outputs one or more audio-based features 520 included in the feature data 124. In some examples, the audio-based features 520 are outputted in the same manner as those for Figure 4In a similar manner as described above for the speech signal processing unit 410 , the one or more audio-based features 520 include an audio representation 524 or an encoded version of the audio representation 524 , one or more expression characteristics associated with the audio representation 524 , or a combination thereof.

[0130] Figure 6 Another example of a component 600 that can be implemented in a system configured to generate adjusted facial data corresponding to an avatar's facial expression, such as in device 102, is illustrated. Component 600 includes a self-supervised learning based deep learning model 610, an image unit 226, and a facial data adjuster 130. In a particular implementation, image unit 226 and facial data adjuster 130 operate substantially as described above.

[0131] The deep learning model 610 is configured to determine an audio representation 624. As illustrative, non-limiting examples, the audio representation 624 includes speech representations from one or more deep learning methods such as self-supervised learning based on Wav2vec, VQ-Wav2vec, Wav2vec2.0, or Hubert implementations. In the illustrative example, the deep learning model 610 corresponds to or is included in the audio unit 222, and the audio representation 624 corresponds to or is included in Figure 2 The audio representation is 224.

[0132] The deep learning model 610 outputs one or more audio-based features 620 included in the feature data 124. In some examples, the audio-based features 620 are outputted in the same manner as those for Figure 4 In a similar manner as described above for the speech signal processing unit 410 , the one or more audio-based features 620 include an audio representation 624 or an encoded version of the audio representation 624 , one or more expression characteristics associated with the audio representation 624 , or a combination thereof.

[0133] Figure 7 Another example of a component 700 that may be implemented in a system configured to generate adjusted facial data corresponding to an avatar's facial expression, such as in device 102, is illustrated. Component 700 includes an audio / image network 710, an image unit 226, and a facial data adjuster 130. In a particular implementation, image unit 226 and facial data adjuster 130 operate substantially as described above.

[0134] The audio / image network 710 is configured to determine an audio / image representation 724. In the illustrative example, the audio / image representation 724 includes a deep learning architecture neural network that receives the audio data 204 and the image data 208 as input and is configured to determine the audio / image representation 724 as a result of jointly processing the audio data 204 and the image data 208. In the illustrative example, the audio / image network 710 is included in the feature data generator 120 and may correspond to, be included in the audio unit 222, and the audio / image representation 724 corresponds to or is included in Figure 2 In another illustrative example, the feature data generator 120 includes an audio / image network 710 instead of or in addition to the audio unit 222 .

[0135] The audio / image network 710 outputs one or more audio and image based features 720 included in the feature data 124. In some examples, the audio / image network 710 outputs one or more audio and image based features 720 included in the feature data 124. Figure 4 In a similar manner as described for the speech signal processing unit 410 , the one or more audio and image based features 720 include an audio / image representation 724 or an encoded version of the audio / image representation 724 , one or more expression characteristics associated with the audio / image representation 724 , or a combination thereof.

[0136] The audio / image network 710 enables the system to listen to the user's voice and analyze the user's image to interpret emotions and behavior. In some implementations, the audio / image network 710 is configured to detect emotions based on the audio data 204, the image data 208, or both (e.g., Figure 2 By using both audio data 204 and image data 208, emotions can be detected based on visual cues (e.g., facial expressions) that are not present in audio data 204 and also based on audible cues (e.g., vocal intonation) that are not present in image data 208, enabling more accurate detection than if detection were performed using only audio data 204 or only image data 208. The audio / image network 710 also enables more robust detection in low signal-to-noise audio conditions that can reduce detection based on audio data 204, and in poor lighting or image capture conditions that can hinder detection based on image data 208.

[0137] According to some aspects, joint processing of audio data 204 and image data 208 may also enable greater accuracy in disambiguating emotions that may have similar auditory or visual cues. As a simplified, non-limiting example, emotion "A" (e.g., blues) may be associated with an audible cue similar to emotion "B" (e.g., sad), and may be associated with a visual cue similar to emotion "C" (e.g., happy). Speech analysis alone may mispredict a user's emotion A as emotion B, and visual analysis alone may mispredict a user's emotion A as emotion C, but the combined speech and visual analysis performed by the audio / image network 710 may correctly predict emotion A.

[0138] Figure 8 Another example of a component 800 that may be implemented in a system configured to generate adjusted facial data corresponding to an avatar's facial expression, such as in device 102, is illustrated. Component 800 includes event detector 810, audio / image network 710, image unit 226, and facial data adjuster 130. In a particular implementation, audio / image network 710, image unit 226, and facial data adjuster 130 operate substantially as described above.

[0139] The event detector 810 is configured to process the audio data 204 to detect one or more audio events 872. In the illustrative example, the event detector 810 is included in the audio unit 222, and the one or more audio events 872 correspond to Figure 2 872 of the audio event 272. In some implementations, the event detector 810 is configured to compare the sound characteristics of the audio data 204 with the audio event model to identify one or more audio events 872 based on matching (or substantially matching) one or more specific audio event models. In some implementations, the event detector 810 includes one or more classifiers configured to process the audio data 204 to determine the associated category from a plurality of categories supported by the one or more classifiers. In an example, the one or more classifiers operate in conjunction with the above-mentioned audio event model to determine the category (e.g., type, such as "dog barking", "glass breaking", "baby crying", etc.) of the sound represented in the audio data 204 and associated with the audio event 872. For example, the one or more classifiers may include a neural network that has been trained using labeled sound data to distinguish between sounds corresponding to various categories, and the neural network is configured to process the audio data 204 to determine the specific category of the sound represented by the audio data 204 (or determine the probability that the sound belongs to the category for each category).

[0140] The event detector 810 is configured to notify the semantic context 122 based on the detected audio event, which may correspond to an associated emotion, such as fear or surprise for "glass breaking", sympathy or frustration for "baby crying", etc. In other examples, the semantic context 122 associated with the detected audio event may correspond to other aspects, such as the location or environment of the user 108 (e.g., on a busy street, in an office, at a restaurant), which may be determined based on the detection of the audio event 272.

[0141] The event detector 810 outputs one or more event-based features 820 included in the feature data 124. In some examples, the one or more event-based features 820 include a label or other identifier of the one or more audio events 872, one or more expression characteristics or emotions associated with the one or more audio events 872, such as fear or surprise for "glass breaking", sympathy for "baby crying", etc. Including one or more event-based features 820 in the feature data 124 enables the facial data adjuster 130 to more accurately predict the facial expressions of the avatar 154, anticipate future facial expressions of the avatar 154, or a combination thereof.

[0142] Fig. 9 Another example of a component 900 that can be implemented in a system configured to generate adjusted facial data corresponding to an avatar facial expression, such as in device 102, is illustrated. Component 900 includes a context prediction network 910, a prediction override unit 930, an image unit 226, and a facial data adjuster 130. In a particular implementation, the image unit 226 and the facial data adjuster 130 operate substantially as described above. In a particular implementation, the context prediction network 910 and the prediction override unit 930, or both, are included in the feature data generator 120, such as in the audio unit 222.

[0143] According to one aspect, context prediction network 910 is configured to process at least a portion of the conversation represented in audio data 204 and use the context and tone of the conversation to predict emotions and which facial expressions are likely to occur, such as reference Figure 2264 of the conversation. In some implementations, the audio data 204 processed by the context prediction network 910 includes a portion of a single user of the conversation (e.g., the voice of the user 108 detected via one or more microphones 202). In other implementations, the audio data 204 processed by the context prediction network 910 also includes voices from one or more (or all) avatars and participants participating in the conversation. The context prediction network 910 is configured to output a predicted expression 920 in the context (e.g., an encoding or indication of a predicted facial expression, emotion, or behavior), one or more features associated with the predicted expression, or a combination thereof for the avatar 154. In a specific implementation, the context prediction network 910 includes a long short-term memory (LSTM) network that is configured to process the conversation and output the predicted expression 920 in the context.

[0144] According to one aspect, the predictive override unit 930 includes a comparator 932 configured to compare the predicted expression 920 in context with a user profile 934. As a non-limiting example, the user profile 934 may enumerate or indicate a range of permissible behaviors or characteristics of the avatar 154, or may enumerate or indicate a range of prohibited behaviors or characteristics of the avatar 154. In some implementations, the user profile 934 includes multiple parameter sets corresponding to different types of conversations, such as different sets of permissible behaviors or characteristics for business conversations, conversations with family, and conversations with friends. The predictive override unit 930 may be configured to be based on Figure 2 266, the social context 268, or both, and the comparator 932 can be configured to perform a comparison to determine whether the predicted expression 920 in the context complies with the selected parameter set. In some implementations, the user profile 934 can include one or more "personalities" selected by the user 108, which indicate the user's preferences for the behavior of the avatar 154 for one or more types of social situations or contexts, such as those previously referenced. Figure 2 described.

[0145] According to one aspect, in response to determining that the predicted expression 920 in the context “matches” the user profile 934 (e.g., conforms to its applicable parameters), the predictive override unit 930 generates an output 950 corresponding to the predicted expression 920 in the context. Otherwise, in response to determining that the predicted expression 920 in the context does not match the user profile 934, the predictive override unit 930 selects or generates an override expression 940 to replace the predicted expression 920 in the context, and generates an output 950 corresponding to the override expression 940. For purposes of illustration, as illustrative, non-limiting examples, the predictive override unit 930 may select an override expression 940 corresponding to concentration to replace the predicted expression 920 in the context corresponding to boredom, or may select an override expression 940 corresponding to a neutral or approving expression to replace the predicted expression 920 in the context corresponding to anger. In other examples, instead of changing the type of expression (e.g., from boredom to concentration), the predictive override unit 930 may change the magnitude of the expression. In a non-limiting illustrative example where an expression has an accompanying magnitude from 1 (barely noticeable) to 10 (extreme), the predictive override unit 930 may replace the “Magnitude 10 Bored” predicted expression (e.g., extremely bored) with an override expression 940 corresponding to the “Magnitude 1 Bored” expression (e.g., only slightly bored).

[0146] Thus, the avatar's behavior / characteristics may be changed to suit certain social situations by analyzing conversation and context or based on user preferences or settings.

[0147] Fig.10 Another example of a component 1000 that can be implemented in a system configured to generate adjusted facial data corresponding to an avatar facial expression, such as in device 102, is illustrated. Component 1000 includes context prediction network 910, prediction verifier 1030, image unit 226, and facial data adjuster 130. In a particular implementation, context prediction network 910, image unit 226, and facial data adjuster 130 operate substantially as described above. In a particular implementation, context prediction network 910, prediction verifier 1030, or both are included in feature data generator 120, such as in audio unit 222.

[0148] In contrast to the prediction override unit 930, the prediction validator 1030 is configured to replace the predicted expression 920 in the context with a corrected expression 1040 in response to determining that the predicted expression 920 in the context is a misprediction. For example, the user profile 934 may include one or more parameters that indicate which expressions are usually expressed by the user in general or in various specific contexts, which expressions the user does not express in general or in specific contexts, or a combination thereof based on enrollment data or the user's historical behavior. In response to the predicted expression 920 in the context not matching the user profile 934, the prediction validator 1030 determines that a misprediction has occurred and generates an output 1050 corresponding to the corrected expression 1040. Therefore, the prediction validator 1030 enables the generation of the avatar 154 with improved accuracy by correcting the misprediction of the user's expression.

[0149] Fig.11 This example shows that Fig. 9 The prediction overwrite unit 930 or Fig.10 The component 1100 implemented in the prediction validator 1030 of the embodiment of the present invention. The comparator 932 is coupled to receive the predicted expression 920 in the context and the user profile 934. In response to determining that the predicted expression 920 in the context matches the user profile 934, the comparator 932 provides the predicted expression 920 in the context as an output 1150. Otherwise, in response to determining that the predicted expression 920 in the context does not match the user profile 934, the comparator 932 provides a code 1130 corresponding to (e.g., included in) the predicted expression 920 in the context to the expression adjuster 1120.

[0150] The expression adjuster 1120 is configured to replace the code 1130 with a replacement code 1132 corresponding to the corrected expression. For example, the expression adjuster 1120 may include a data structure 1160 (such as a table 1162) that implements mapping and lookup operations involving various expressions and their corresponding codes. As illustrated, the code 1130 has a value corresponding to the "happy" expression (e.g., "NNNN"), and the expression adjuster 1120 replaces the value of the code 1130 with another value of the replacement code 1132 (e.g., "YYYY"). The replacement code 1132 corresponds to a replacement expression 1140 (e.g., an overwrite expression 940 or a corrected expression 1040) of "angry" provided as an output 1150 (e.g., output 950 or output 1050).

[0151] Thus, an expression override or expression correction may correspond to a type of dictionary comparison. For example, if the comparator 932 determines that the expression prediction is far from an expected or allowed expression prediction (e.g., does not "match" the user profile 934), the code for the expression prediction may be replaced by the code for a more appropriate expression (e.g., matching the user profile 934).

[0152] although Figures 9 to 11 A comparison to a user profile 934 is illustrated, but in other implementations, such as when separate user profiles are not supported or when a user has not established a separate profile, such a comparison may be made to a default set of parameters (eg, a default profile).

[0153] Fig.12 Another example of a component 1200 that can be implemented in a system configured to generate adjusted facial data corresponding to an avatar's facial expression, such as in device 102, is illustrated. Component 1200 includes a context-based future speech prediction network 1210, a representation generator 1230, an image unit 226, and a facial data adjuster 130. In a specific implementation, the image unit 226 and the facial data adjuster 130 operate substantially as described above. In a specific implementation, the context-based future speech prediction network 1210, the representation generator 1230, or both are included in the feature data generator 120, such as included in the audio unit 222.

[0154] The context-based future speech prediction network 1210 processes the audio data 204 (and optionally also processes the image data 208) to determine a predicted word 1220 in context. For example, in some implementations, the context-based future speech prediction network 1210 includes a long short-term memory (LSTM) type neural network that is configured to predict the most likely next word or distribution of words to be spoken by the user based on the context of the user's words identified in the audio data 204 (and in some implementations, further based on the image data 208). Optionally, audio event detection can be used to provide input to the context-based future speech prediction network 1210, such as a reference Fig.14 Further described.

[0155] The representation generator 1230 is configured to generate a representation 1250 of the predicted word 1220 in the context. In some implementations, the representation generator 1230 is configured to determine one or more phonemes or mel-spectrograms associated with the predicted word 1220 in the context, and generate the representation 1250 based on the one or more phonemes or mel-spectrograms. The representation 1250 (e.g., one or more phonemes or mel-spectrograms or encodings thereof) can be concatenated to or otherwise combined with the one or more image-based features 322 to generate the feature data 124.

[0156] Thus, the context-based future speech prediction network 1210 and representation generator 1230 enable prediction of what a word or sentence will be based on the context of the word being spoken, which is used to predict the facial image / texture of the avatar or ensure frame-by-frame compliance (e.g., the transition between "e" to "l") to ensure that the image of the avatar pronouncing the word transitions correctly over time.

[0157] Fig.13 Another example of a component 1300 that can be implemented in a system configured to generate adjusted facial data corresponding to an avatar's facial expression, such as in device 102, is illustrated. Component 1300 includes a context-based future speech prediction network 1310, a speech representation generator 1330, an image unit 226, and a facial data adjuster 130. In a specific implementation, the image unit 226 and the facial data adjuster 130 operate substantially as described above. In a specific implementation, the context-based future speech prediction network 1310, the speech representation generator 1330, or both are included in the feature data generator 120, such as included in the audio unit 222.

[0158] The context-based future speech prediction network 1310 processes the audio data 204 (and optionally also processes the image data 208) to determine the predicted speech in context 1320. For example, in some implementations, the context-based future speech prediction network 1310 includes a long short-term memory (LSTM) type neural network that is configured to predict the most likely speech or distribution of speech that will be spoken by the user based on the context of the user's words identified in the audio data 204 (and in some implementations, further based on the image data 208).

[0159] The speech representation generator 1330 is configured to generate a representation 1350 of the predicted speech 1320 in the context. In some implementations, the speech representation generator 1330 is configured to determine the representation 1350 as a “classical” representation (e.g., as in Figure 4 Mel-spectrograms, pitch, MFCC in ), one or more labels (e.g., as in Figure 5 ), one or more deep learning representations (e.g., as in Figure 6 ), or one or more other representations associated with the predicted speech 1320 in context. The representation 1350 can be concatenated to or otherwise combined with one or more image-based features 322 to generate feature data 124.

[0160] Fig.14Another example of a component 1400 that can be implemented in a system configured to generate adjusted facial data corresponding to an avatar's facial expression, such as in device 102, is illustrated. Component 1400 includes event detector 1402, context-based future speech prediction network 1410, representation generator 1430, image unit 226, and facial data adjuster 130. In a specific implementation, image unit 226 and facial data adjuster 130 operate substantially as described above. In a specific implementation, event detector 1402, context-based future speech prediction network 1410, representation generator 1430, or a combination thereof is included in feature data generator 120, such as included in audio unit 222.

[0161] Event detector 1402 is configured to process audio data 204 to determine event detection 1404. In certain implementations, event detector 1402 operates in a similar manner as described with reference to audio events 272, event detector 810, or both.

[0162] The context-based future speech prediction network 1410 processes the audio data 204 and the event detection 1404 (and optionally also processes the image data 208) to determine the prediction 1420. In some implementations, the context-based future speech prediction network 1410 corresponds to Fig.12 The context-based future speech prediction network 1210, and the prediction 1420 corresponds to the predicted word 1220 in the context. In other specific implementations, the context-based future speech prediction network 1410 corresponds to Fig.13 The context-based future speech prediction network 1310 is used, and the prediction 1420 corresponds to the predicted speech 1320 in the context.

[0163] Representation generator 1430 is configured to generate representation 1450 of prediction 1420. In some implementations, representation generator 1430 corresponds to representation generator 1230, and representation 1450 corresponds to representation 1250. In other implementations, representation generator 1430 corresponds to speech representation generator 1330, and representation 1450 corresponds to representation 1350.

[0164] By using event detection 1404 as input to context-based future speech prediction network 1410, predictions of future speech can be more accurate than predictions made without knowledge of the audio event. For example, if a sudden shattering of glass is detected in audio data 204, prediction 1420 can be informed by the additional knowledge that the user may be surprised, which may not be predictable based solely on the user's speech.

[0165] Fig.15An example 1500 of a particular implementation of the facial data adjuster 130 is depicted, illustrated as a deep learning architecture network including an encoder portion 1504 coupled to a decoder portion 1502. For example, as illustrative, non-limiting examples, the facial data adjuster 130 may correspond to a U-net or autoencoder type network.

[0166] The encoder portion 1504 is configured to process the facial data 132 and generate an output that is provided to the decoder portion 1502. The output of the encoder portion 1504 may be a reduced-dimensional representation of the facial data 132, and may be referred to as a code or latent vector.

[0167] The decoder portion 1502 is configured to process the output of the encoder portion 1504 in conjunction with the speech representation 1524 to generate the adjusted facial data 134. In some implementations, the speech representation 1524 corresponds to an audio representation (such as Figure 2 audio representation 224), one or more audio-based features (such as Figure 3 One or more audio-based features 320), audio-derived features (such as Figure 7 One or more audio and image based features 720 or Fig. 9 output 950). Figures 16 to 21 Examples of different specific implementations of how the output of the encoder portion 1504 is combined with the speech representation 1524 at the decoder portion 1502 for processing are illustrated in FIG.

[0168] Fig.16 An example 1600 is depicted where a skin representation 1624 (eg, the output of the encoder portion 1504) is concatenated with a speech representation 1524 to form a combined representation 1602. The combined representation 1602 is input to a neural network 1630. In a particular implementation, the neural network 1630 corresponds to the decoder portion 1502.

[0169] Fig.17 An example 1700 is depicted where a speech representation 1524 is processed at one or more neural network layers 1702 to generate an output 1712, and a skin representation 1624 is processed at one or more neural network layers 1704 to generate an output 1714. The outputs 1712 and 1714 are input to a neural network 1730, which may correspond to the decoder portion 1502. In a particular implementation, the output 1712 is concatenated with the output 1714 prior to being input to the neural network 1730.

[0170] Fig.18An example 1800 is depicted in which an encoder portion 1504 processes facial data 132 to generate a skin deep learning (DL) representation 1820 illustrated as code 1802. A concatenation unit 1804 concatenates the code 1802 with a speech representation 1524 and a facial part representation 1824, such as one or more image-based features 322, to generate concatenated input data 1830. For example, the concatenation unit 1804 may perform the concatenation according to the following equation:

[0171] D n =[A n , B n , C n ],

[0172] Where D n Indicates the cascaded input data 1830, A n Indicates code 1802, B n Indicates facial part representation 1824, and C n The speech representation 1524 is represented. The concatenated input data 1830 is processed by the decoder portion 1502 to generate the adjusted facial data 134.

[0173] Fig.19 An example 1900 is depicted where a fusion unit 1904 performs a latent space fusion operation of the code 1802, the speech representation 1524, and the facial part representation 1824 to generate a fused input 1930. For example, the fusion unit 1904 can perform the fusion according to one or more equations, such as a weighted sum, a Hadamard equation or transform, an element-wise product, etc. In a specific example, the fusion unit 1904 performs the fusion according to the following equation:

[0174] D n = αA n +βB n +γC n ,

[0175] Where D n represents the fusion input 1930, α, β and γ represent weighting factors, A n Indicates code 1802, B n Indicates facial part representation 1824, and C n The speech representation 1524 is represented. The fused input 1930 is processed by the decoder portion 1502 to generate the adjusted facial data 134.

[0176] Fig. 20 Depicted Fig.19The fusion unit 1904 of the example 2000 is replaced by a fusion neural network 2004. The fusion neural network 2004 is configured to perform the fusion of the code 1802, the speech representation 1524, and the facial part representation 1824 using network layers (such as one or more fully connected or convolutional layers) to generate a fused input 2030 for the decoder part 1502.

[0177] Fig.21 Another example 2100 of performing fusion of various codes (e.g., code 1802, speech representation 1524, and facial part representation 1824) at decoder portion 1502 is depicted. For example, decoder portion 1502 may process code 1802 at an input layer, followed by a sequence of layers that perform upconvolution. Speech representation 1524 and facial part representation 1824 may be fused at decoder portion 1502, such as provided as input at one or more of the upconvolution layers rather than at the input layer.

[0178] Fig. 22 An example of a system 2200 is depicted in which one or more motion sensors 210 are coupled to (e.g., integrated in) a head mounted device 2202 (such as an HMD) and configured to generate motion sensor data 212 included in the sensor data 106. For example, the one or more motion sensors 210 may include an inertial measurement unit (IMU), one or more other sensors configured to detect movement, acceleration, orientation, or a combination thereof. As illustrated, the motion sensor data 212 includes head tracker data 2210 indicating at least one of a head movement 2250 or a head orientation 2252 of the user 108.

[0179] Device 102 includes a Figure 2 The one or more processors 116 of the feature data generator 120 and the facial data adjuster 130 are implemented in a similar manner as described in . In a specific implementation, the one or more processors 116 are configured to determine the semantic context 122 based on comparing the motion 2240 represented in the motion sensor data 212 with at least one motion threshold 2242. For example, a motion 2240 (e.g., head movement 2250, head orientation 2252, or a combination thereof) that exceeds the motion threshold 2242 in a relatively short period of time may indicate a sudden reaction of the user to an external event, such as a startled or surprised reaction. As another example, the head movement of the user 108 may represent a gesture that conveys meaning. For the purpose of illustration, nodding up and down may indicate that the user 108 agrees or has a positive emotional state, shaking left and right may indicate disagreement or a negative emotional state, tilting the head to one side may indicate confusion, etc.

[0180] Thus, feature data generator 120 generates feature data 124 based on motion sensor data 212, and facial data adjuster 130 generates adjusted facial data 134 based on feature data 124. Adjusted facial data 134 may correspond to the avatar's facial expression based on semantic context 122 derived from motion sensor data 212 (and, in some implementations, not derived from any image data or audio data).

[0181] Optionally, the feature data generator 120 may also include an audio unit 222 configured to generate an audio representation 224 based on the audio data 204. In such implementations, the feature data 124 may include additional information derived from the audio data 204 and may therefore provide additional realism or accuracy to the generation of the avatar compared to using only the motion sensor data 212. Optionally, the system 2200 also includes one or more microphones 202, such as one or more microphones integrated in or attached to the head mounted device 2202.

[0182] Optionally, the feature data generator 120 may include an image unit 226 configured to generate a facial representation 228 based on the image data 208. In such implementations, the feature data 124 may include additional information derived from the image data 208 and may therefore provide additional authenticity or accuracy to the generation of the avatar compared to using only the motion sensor data 212. Optionally, the system 2200 also includes one or more cameras 206, such as integrated in or attached to the head mounted device 2202 and configured to generate Figure 2 The image data of multiple cameras 208A, 208B and 208C.

[0183] Additional synergies can be generated by using motion sensor data 212 in combination with one or both of audio data 204 or image data 208. For example, if user 108 makes a positive statement such as "That's a good idea" while shaking the user's head from side to side, the shaking motion alone can be interpreted as disagreement or negative emotion, while the user's voice alone can be interpreted as agreement or positive emotion. However, the combination of the user's voice and head movement can enable the device to more accurately determine that user 108 is expressing sarcasm. A similar synergy can be generated using a combination of image data 208 and motion sensor data 212. For example, a user showing a bright smile (e.g., a visual expression of happiness) while shaking the user's head from side to side (a gesture of disagreement with a negative emotion) can be more accurately determined as an expression of amused disbelief.

[0184] Fig.23An example of a system 2300 is depicted in which a feature data generator 120 is used to generate an audio output 2340 associated with an avatar 154. The system 2300 includes a specific implementation of a device 102 that includes a memory 112 coupled to one or more processors 116. The one or more processors 116 include a feature data generator 120, a facial data generator 230, a facial data adjuster 130, and an avatar generator 236 that operate in a similar manner as described above. For example, the feature data generator 120 is configured to process the sensor data 106 to generate the feature data 124, the facial data generator 230 may be configured to process the image data 208 corresponding to the face of the user to generate the facial data 132, and the facial data adjuster 130 and the avatar generator 236 are used together to generate a representation 152 of the avatar 154 based on the facial data 132 and the feature data 124. Optionally, the system 2300 includes one or more microphones 202 for capturing audio data 204 that may be included in the sensor data 106 , one or more cameras 206 for capturing image data 208 that may be included in the sensor data 106 , or a combination thereof.

[0185] The one or more processors 116 are configured to generate an audio output 2340 for the avatar 154 based on the sensor data 106. For example, the feature data generator 120 is configured to generate first output data 2320 representing speech. For example, in some implementations, the audio unit 222 is configured to process the audio data 204 to generate audio-based output data 2304 corresponding to the user speech represented in the audio data 204, such as reference Figure 24 to Figure 28 , Fig.31 and Fig.33 In some implementations, the image unit 226 is configured to process the image data 208 to generate image-based output data 2306 corresponding to the user's facial expressions (e.g., the shape, position, movement, etc. of the user's mouth, tongue, etc.) when the user speaks, such as reference Figure 29 to Figure 33 The audio-based output data 2304 , the image-based output data 2306 , or both are included in the first output data 2320 .

[0186] In some implementations, the first output data 2320 is processed by the voice converter 2310 to generate the second output data 2322, and the second output data 2322 corresponds to the converted output data representing the converted speech. For example, the voice converter 2310 can be configured to modify one or more aspects of the user's voice (e.g., accent, pitch, etc.) or replace the user's voice with a different voice corresponding to the avatar 154, as further described below. In other implementations, the voice converter 2310 is disabled, bypassed, or omitted from the device 102, and the second output data 2322 matches the first output data 2320.

[0187] The second output data 2322 is processed by the audio decoder 2330 to generate an audio output 2340 (e.g., pulse code modulated (PCM) audio data). In some implementations, the representation 152 of the avatar 154, the audio output 2340, or both can be transmitted to a second device (e.g., a headset sent to a user of the system 2300, a remote user's device, a server, etc.) for displaying the avatar 154, playing back the audio output 2340, or both. Optionally, the system 2300 includes a display device 150 configured to display the representation 152 of the avatar 154, one or more speakers 2302 configured to play the audio output 2340, or a combination thereof.

[0188] According to some specific implementations, such as reference Figure 24 to Figure 28 As further described, the first output data 2320, the second output data 2322, or both are based on the audio data 204 independent of any image data 208. For example, the audio data 204 may represent the voice of a user of the system 2300 (e.g., captured by one or more microphones 202), and the feature data generator 120 may be configured to process the audio data 204 to generate the first output data 2320 representing the user's voice, the second output data 2322 representing a modified version of the user's voice, or both. For example, the feature data generator 120 may generate the second output data 2322 as the user's voice in a voice different from the user's voice (e.g., a modified voice version of the user's voice corresponding to a different avatar or otherwise changing the user's voice). Alternatively or additionally, as non-limiting examples, second output data 2322 may be encoded to modify (e.g., enhance, reduce, or change) an accent in the user's voice to improve intelligibility to a listener, such as modifying the user's voice when the user is ill and wants the avatar to have a more robust voice, causing the avatar to speak in a different style than the user (e.g., calmer or steadier than the user's voice), or changing the language in which the avatar speaks.

[0189] Thus, as non-limiting examples, audio output 2340 may correspond to a modified version of the user's voice, such as when avatar 154 is a realistic representation of the user, or may correspond to a virtual voice of avatar 154 when avatar 154 corresponds to a fictional character or fantasy creature. Because generating the avatar's voice based on varying aspects of the user's voice may cause a misalignment between the avatar's facial movements and the avatar's voice, second output data 2322 (or information associated with second output data 2322) may also be included in feature data 124 to adjust the avatar's facial expressions to more closely match the avatar's voice.

[0190] According to some implementations, the first output data 2320, the second output data 2322, or both are based on the audio data 204 in combination with the image data 208. For example, as previously described, the image data 208 can help disambiguate the user's speech in the audio data 204, such as in a noisy or windy environment that results in low-quality capture of the user's speech by the one or more microphones 202. In some examples, the one or more processors 116 can determine a context-based predicted expression of the user's face and generate the audio output 2340 based at least in part on the context-based predicted expression. In a specific example, the image data 208 can be used in combination with the audio data 204 to perform voice activity detection based on determining when the user's mouth is predicted to be closed, as described in reference to FIG. Figure 31 to Figure 33 Further described.

[0191] According to some implementations, the first output data 2320, the second output data 2322, or both are based on the image data 208 independent of any audio data 204. For example, as an illustrative, non-limiting example, the system 2300 can operate in a lip-reading mode in which the audio output 2340 is generated based on the user's facial expressions and movements, such as in a very noisy environment or when one or more microphones 202 are disabled, or for privacy, such as when using public transportation or in a library, or if the user has a physical condition that prevents the user from speaking. Figure 29 to Figure 30 An example of generating audio output 2340 based on image data 208 is further described.

[0192] refer to Fig.24 , which may be used in a system configured to generate audio output for an avatar (such as in Fig.23 Component 2400 includes an audio network 310, a speech converter 2410, an output speech generator 2412, an image unit 226, and a facial data adjuster 130.

[0193] As previously described, the audio network corresponds to a deep learning neural network, such as an audio variational autoencoder, that can be implemented in the feature data generator 120. As an illustrative, non-limiting example, the audio network 310 is trained to identify characteristics of speech represented in the audio data 204 and determine an audio representation 324 that includes one or more of an expression condition, an audio phoneme, or a mel-spectrogram. The audio network 310 generates output data (illustrated as a first audio code 2420) representing the speech in the audio data 204. For illustration, the first audio code 2420 may correspond to a latent space representation of the speech represented in the audio data 204.

[0194] The speech converter 2410 is configured to perform latent space speech conversion based on the first audio code 2420 to generate a second audio code 2422. For illustration, the speech converter 2410 may correspond to one or more neural networks trained to process an input latent space representation of speech (e.g., the first audio code 2420) and generate an output latent space representation of modified speech (e.g., the second audio code 2422). The speech converter 2410 may be operable to modify the accent, the quality of the speech (e.g., robustness), change the language of the speech, make one or more other modifications, or a combination of the above.

[0195] The output speech generator 2412 is configured to process input data representing speech and generate an output speech signal, such as PCM data. For example, as an illustrative non-limiting example, the output speech generator 2412 may include a speech generator (e.g., a wavenet-type speech generator) or a vocoder-based speech synthesis system (e.g., a WORLD-type vocoder). As illustrated, the output speech generator 2412 is configured to process the second audio code 2422 to generate modified voice data 2440.

[0196] In a specific implementation, the audio network 310 and the speech converter 2410 are included in the feature data generator 120, and the output speech generator 2412 corresponds to the audio decoder 2330. In such an implementation, the audio network 310 may be included in Fig.23 In the audio unit 222, the first audio code 2420 corresponds to the audio-based output data 2304, the speech converter 2410 corresponds to the speech converter 2310, the second audio code 2422 corresponds to the second output data 2322, and the modified voice data 2440 corresponds to the audio output 2340.

[0197] In addition to being processed at the output speech generator 2412, the second audio code 2422 is also combined with one or more image-based features 322 output by the image unit 226 in the feature data 124. This combination of audio-based features and image-based features can be performed via concatenation, fusion, or one or more other techniques, such as previously described in Figures 18 to 21 The facial data adjuster 130 uses the feature data 124 to generate adjusted facial data 134 for the avatar.

[0198] Thus, the audio network 310 processes the audio data 204 to generate output data (first audio code 2420) representing speech in the audio data 204, and the speech converter 2410 performs speech conversion corresponding to the latent space speech conversion of the output data from the audio network 310 to generate converted output data corresponding to an audio code (second audio code 2422) representing the converted speech. A representation 152 of the avatar 154 is generated based on the converted output data, and the output speech generator 2412 generates an audio output (modified speech data 2440) for the avatar 154 with the converted speech.

[0199] Fig.25 An example of a method that can be used in a system configured to generate audio output for an avatar, such as in Fig.23 Component 2500 includes a speech signal processing unit 410, a speech converter 2510, an output speech generator 2512, an image unit 226, and a facial data adjuster 130.

[0200] As previously described, the speech signal processing unit 410 includes one or more components configured to process the audio data 204 and detect, generate, or otherwise determine characteristics of speech in the audio data 204 and determine an audio representation 424. As illustrative, non-limiting examples, the audio representation 424 includes one or more signal processed speech representations, such as MFCCs, MFCCs and pitch information, or spectrogram information (e.g., a regular spectrogram, a log-mel spectrogram, or one or more other types of spectrograms). The speech signal processing unit 410 generates output data (illustrated as a first speech representation output 2520) representing the speech in the audio data 204. For illustration, the first speech representation output 2520 may correspond to Figure 4 One or more audio-based features 420.

[0201] The speech converter 2510 is configured to perform speech conversion based on the first speech representation output 2520 to generate a second speech representation output 2522. The speech converter 2510 performs speech conversion in a speech representation domain associated with the speech representation outputs 2520 and 2522 (e.g., MFCC, MFCC and pitch information, spectrogram, etc.). The speech converter 2510 is operable to modify the accent, modify the speech quality (e.g., robustness), change the language of the speech, make one or more other modifications, or a combination of the above.

[0202] The output speech generator 2512 is configured to process input data representing speech and generate speech in a manner consistent with that for Fig.24 The output speech signal is generated in a manner similar to that described for the output speech generator 2412. As illustrated, the output speech generator 2512 is configured to process the second speech representation output 2522 to generate modified speech data 2540.

[0203] In a specific implementation, the speech signal processing unit 410 and the voice converter 2510 are included in the feature data generator 120, and the output speech generator 2512 corresponds to the audio decoder 2330. In such an implementation, the speech signal processing unit 410 may be included in Fig.23 In the audio unit 222, the first speech representation output 2520 corresponds to the audio-based output data 2304, the speech converter 2510 corresponds to the speech converter 2310, the second speech representation output 2522 corresponds to the second output data 2322, and the modified speech data 2540 corresponds to the audio output 2340.

[0204] The second speech representation output 2522 and the one or more image-based features 322 are combined in the feature data 124 and are used by the facial data adjuster 130 to match the image-based features 322 for the facial data. Fig.24 The described method is used in a similar manner.

[0205] Fig.26 An example of a method that can be used in a system configured to generate audio output for an avatar, such as in Fig.23 26. The component 2600 includes an ASR-based processing unit 510, a speech converter 2610, an output speech generator 2612, an image unit 226, and a facial data adjuster 130.

[0206] As previously described, the ASR-based processing unit 510 includes one or more components configured to process the audio data 204 and detect, generate, or otherwise determine characteristics of speech in the audio data 204 and determine an audio representation 524. As illustrative, non-limiting examples, the audio representation 524 includes one or more ASR-based speech representations or labels, such as one or more phonemes, diphones, or triphones, associated stress or prosody (e.g., duration, pitch), one or more words, or a combination thereof. The ASR-based processing unit 510 generates output data (illustrated as a first speech representation output 2620) representing the speech in the audio data 204. For illustration, the first speech representation output 2620 may correspond to Figure 5 One or more audio-based features 520.

[0207] The speech converter 2610 is configured to perform speech conversion based on the first speech representation output 2620 to generate a second speech representation output 2622. The speech converter 2610 performs speech conversion in a speech representation domain associated with the speech representation outputs 2620 and 2622 (e.g., phonemes, diphones or triphones, associated stress or prosody, one or more words, etc.). The speech converter 2610 is operable to modify the accent, speech quality (e.g., robustness), change the language of the speech, make one or more other modifications, or a combination of the above.

[0208] The output speech generator 2612 is configured to process input data representing speech and generate speech in a manner consistent with that for Fig.24 The output speech signal is generated in a manner similar to that described for the output speech generator 2412. As illustrated, the output speech generator 2612 is configured to process the second speech representation output 2622 to generate modified speech data 2640.

[0209] In a specific implementation, the ASR-based processing unit 510 and the speech converter 2610 are included in the feature data generator 120, and the output speech generator 2612 corresponds to the audio decoder 2330. In such an implementation, the ASR-based processing unit 510 may be included in Fig.23 In the audio unit 222, the first speech representation output 2620 corresponds to the audio-based output data 2304, the speech converter 2610 corresponds to the speech converter 2310, the second speech representation output 2622 corresponds to the second output data 2322, and the modified speech data 2640 corresponds to the audio output 2340.

[0210] The second speech representation output 2622 and the one or more image-based features 322 are combined in the feature data 124 and are used by the facial data adjuster 130 to match the image-based features 322 for the facial data. Fig.24 The described method is used in a similar manner.

[0211] Fig. 27 An example of a method that can be used in a system configured to generate audio output for an avatar, such as in Fig.23 Component 2700 is an example of a component implemented in the specific implementation of device 102 as illustrated in FIG. Component 2700 includes a deep learning model 610 based on self-supervised learning, a voice converter 2710, an output speech generator 2712, an image unit 226, and a facial data adjuster 130.

[0212] As previously described, the deep learning model 610 is configured to determine the audio representation 624. As illustrative, non-limiting examples, the audio representation 624 includes speech representations from one or more deep learning methods, such as self-supervised learning based on Wav2vec, VQ-Wav2vec, Wav2vec2.0, or Hubert implementations. The deep learning model 610 generates output data (illustrated as the first speech representation output 2720) representing the speech in the audio data 204. For illustration, the first speech representation output 2720 may correspond to Figure 6 One or more audio-based features 620.

[0213] The speech converter 2710 is configured to perform speech conversion based on the first speech representation output 2720 to generate a second speech representation output 2722. The speech converter 2710 performs speech conversion in a speech representation domain associated with the speech representation outputs 2720 and 2722. The speech converter 2710 is operable to modify the accent, the quality of the speech (e.g., robustness), change the language of the speech, make one or more other modifications, or a combination of the foregoing.

[0214] The output speech generator 2712 is configured to process input data representing speech and generate speech in a manner consistent with that for Fig.24 The output speech signal is generated in a manner similar to that described for the output speech generator 2412. As illustrated, the output speech generator 2712 is configured to process the second speech representation output 2722 to generate modified speech data 2740.

[0215] In a specific implementation, the deep learning model 610 and the speech converter 2710 are included in the feature data generator 120, and the output speech generator 2712 corresponds to the audio decoder 2330. In such an implementation, the deep learning model 610 may be included in Fig.23In the audio unit 222, the first speech representation output 2720 corresponds to the audio-based output data 2304, the speech converter 2710 corresponds to the speech converter 2310, the second speech representation output 2722 corresponds to the second output data 2322, and the modified speech data 2740 corresponds to the audio output 2340.

[0216] The second speech representation output 2722 and the one or more image-based features 322 are combined in the feature data 124 and are used by the facial data adjuster 130 to match the image-based features 322 for the facial data. Fig.24 The described method is used in a similar manner.

[0217] Fig.28 An alternative implementation of component 2800 is depicted, wherein Fig. 27 The functionality associated with the speech converter 2710 and the output speech generator 2712 is combined into a speech conversion unit 2812 that outputs the modified speech data 2740. The modified speech data 2740, instead of Fig. 27 The second speech representation output 2722 is included in the feature data 124 for use by the facial data adjuster 130.

[0218] Fig.29 An example of a method that can be used in a system configured to generate audio output for an avatar, such as in Fig.23 Component 2700 includes a character specific audio decoder 2930, an optional audio as text display unit 2980, an image unit 226, and a facial data adjuster 130.

[0219] The image unit 226 processes the image data 208 to generate an image code 2920, such as a latent vector generated at one or more neural networks (e.g., a facial part VAE) of the image unit 226. The image code 2920 contains information about the user's facial expression that is captured in the image data 208 and can be used to predict the user's voice independently of any input audio prompts (e.g., without receiving or processing audio data 204 that captures the user's voice). The image code 2920 may correspond to or may be different from one or more image-based features 322 included in the feature data 124 provided to the facial data adjuster 130.

[0220] The character-specific audio decoder 2930 is configured to process the image code 2920 to generate voice data 2940. For example, the character-specific audio decoder 2930 may include one or more neural networks that are trained to predict the user's voice based on information about the user's facial expressions received via the image code 2920. To illustrate, the character-specific audio decoder 2930 may receive a sequence of image codes 2920 corresponding to a sequence of images that capture the user's face as the user speaks and based on the expressions (e.g., the shape, position, and movement of the user's lips, tongue, etc.), and may generate voice data 2940 representing the user's predicted voice. The voice data 2940 may simulate voice characteristics associated with the avatar 154, such as a specific voice characteristic of the user, a modified version of the user's voice characteristics, or voice characteristics associated with a fictional character or fantasy creature in a virtual avatar implementation.

[0221] According to some aspects, voice data 2940 corresponds to an audio signal (eg, PCM data), such as Fig.23 The audio output 2340 of the speech data may be played via one or more speakers 2302 or sent to another device for playback. Additionally or alternatively, the speech data 2940 may be input to an audio display unit 2980 as text.

[0222] The audio as text display unit 2980 is configured to generate a text version of the speech represented in the speech data 2940 and output the text version for display, such as at the display device 150. In a specific implementation where the speech data 2940 includes a speech signal (such as PCM data), the audio as text display unit 2980 can perform ASR to generate a text version of the speech. In a specific implementation where the speech data 2940 includes another representation of speech, the audio as text display unit 2980 can perform a conversion to text based on the speech representation included in the speech data 2940.

[0223] Displaying a text version of the speech data 2940 provides a source of feedback to the user regarding how the user's facial expressions were interpreted to predict the user's voice. For example, as illustrative, non-limiting examples, based on feedback indicating one or more incorrect predictions, the user can adjust the speaking style (e.g., slow down, improve pronunciation, etc.), adjust the camera positioning to more accurately capture facial expressions, enter corrections to errors in the text to provide feedback that can be used to update the character-specific audio decoder 2930, or a combination of the above.

[0224] As an illustrative, non-limiting example, component 2900 enables a system (such as system 2300) to operate in a lip reading mode, such as in a very noisy environment or when one or more microphones 202 are disabled, or for privacy, such as when using public transportation or in a library, or if the user has a physical condition that prevents the user from speaking. In some implementations, one or more conditions, situations, or habits associated with the user can be determined to generate a personal profile, such as a reference Fig.30 In embodiments where the user is unable to speak audibly, such as due to a medical condition or illness, one or more aspects of the generation of speech data 2940 (such as speech characteristics generated by the character-specific audio decoder 2930) may be set to default or generic values, selected by the user (e.g., via selection of one or more set values ​​in a user profile), or a combination of the above.

[0225] In a particular implementation, the image code 2920 corresponds to the image-based output data 2306 , the character-specific audio decoder 2930 corresponds to the speech converter 2310 (or a combination of the speech converter 2310 and the audio decoder 2330 ), and the speech data 2940 corresponds to the second output data 2322 or audio output 2340 .

[0226] Fig.30 Another specific implementation of component 3000 is depicted, wherein the audio output is based on Fig.29 The voice data 2940 and further based on the user profile 3034. The component 3000 includes Fig.29 The image unit 226, the character-specific audio decoder 2930, the audio as text display unit 2980 and the facial data adjuster 130 in FIG. 1 and further include a prediction verifier 3030. In some specific implementations, the prediction verifier 3030 is included in Fig.23 In the feature data generator 120.

[0227] The predictive validator 3030 includes a comparator 3032 configured to determine whether one or more aspects of the speech data 2940 match a user profile 3034. For example, the user profile 3034 may include information corresponding to one or more conditions, contexts, or habits associated with a particular user. In response to determining that one or more aspects of the speech data 2940 fail to match the user profile 3034, the predictive validator 3030 generates corrected speech data 3042, such as by altering the speech data 2940 to include the corrected speech 3040. In response to determining that the speech data 2940 matches the user profile 3034, the predictive validator 3030 may output the speech data 2940 (unaltered) as the corrected speech data 3042.

[0228] According to some aspects, user profile 3034 includes information based on historical incorrect predictions that have been made for a particular user's speech in lip reading mode. According to some aspects, user profile 3034 corresponds to, includes, or otherwise provides information such as for Figures 9 to 11 934. In the illustrative example, user profile 3034 may include information about words or phrases that a particular user is unlikely to say or that the user has selected, speech characteristics (e.g., shouting, stuttering), etc. that should not be represented in the audio output for the user.

[0229] Fig.31 A specific implementation of component 3100 is depicted in which an audio output is generated based on both image data 208 and audio data 204. Component 3100 includes image unit 226, audio network 310, and facial data adjuster 130.

[0230] The image unit 226 and the audio network 310 each generate one or more codes (or alternatively, one or more other representations of the image data 208 and the audio data 204, respectively), which may correspond to the image-based output data 2306 and the audio-based output data 2304, respectively, and are illustrated as one or more audio / image codes 3110. The one or more audio / image codes 3110 may be processed to generate an image associated with the user and based on the user's voice represented in the audio data 204, the user's facial expression represented in the image data 208 (such as in Fig.30 As previously described in the context of predicting expressions, emotions, etc., using multimodal input (e.g., image data 208 and audio data 204) may be more accurate, more robust under changing conditions, or otherwise more effective than using input data from only one modality. In a specific example, one or more audio / image codes 3110 may be used to generate or modify avatar voice data, such as Fig.23 second output data 2322, audio output 2340, or both.

[0231] exist Fig.31In a specific example of the present invention, one or more audio / image codes 3110 are processed at a mouth closure detector 3150 configured to detect whether the user's mouth is closed. For example, the mouth closure detector 3150 may include a neural network configured to receive the one or more audio / image codes 3110 as input and generate an audio mute signal 3152 in response to predicting that the user's mouth is closed based on the image data 208 and the audio data 204. The audio mute signal 3152 can cause the system to mute the audio output for the avatar based on the prediction that the user's mouth is closed. In an illustrative, non-limiting example in which the audio data 204 includes the speech of one or more people other than the user (e.g., the speech of nearby people, speech broadcast during a video conferencing session, etc.) and the user is not speaking, determining by the mouth closure detector 3150 that the user's mouth is closed can prevent the audio output associated with the avatar 154 from erroneously including audio based on non-user speech. In some specific implementations, the mouth closure detector 3150 is included in Fig.23 in a feature data generator 120 , such as in a speech converter 2310 .

[0232] According to some aspects, the mouth closure detector 3150 corresponds to or is included in a voice activity detector (VAD) driven by audio and video. In some specific implementations, the VAD can also be configured to check whether other applications are being used that can indicate whether non-user voices may be present, such as video conferencing applications, audio or video playback applications, etc., which can further inform the VAD whether the voice in the audio data 204 is from the user.

[0233] In some implementations, the audio mute signal 3152 can also be used to prevent synthesis of a facial expression of the avatar 154 corresponding to the audio data 204. For example, the audio mute signal 3152 can be provided to the facial data adjuster 130, which can cause the avatar 154 to have a neutral facial expression while the user's mouth remains closed.

[0234] Fig.32 The image data 208 is configured to be generated independently of the audio data 204. Fig.31 Another example of a component 3200 of an audio mute signal 3152. As illustrated, the image unit 226 provides the image code 3210 to the mouth closure detector 3150. Fig.32 In the specific implementation depicted in , the mouth closure detector 3150 is configured to process the image code 3210 to determine whether to generate an audio mute signal 3152.

[0235] Fig.33 Depicted is a method configured to generate a context based at least in part on the context from the audio data 204. Fig.31Another example of a component 3300 for an audio silence signal 3152. The component 3300 includes a context prediction network 910 that processes the audio data 204 to generate a predicted expression in a context 920, such as previously referenced Fig. 9 and Fig.10 As described.

[0236] exist Fig.33 In the specific implementation depicted in , mouth closure detector 3150 is configured to process the predicted expression in context 920 in conjunction with image code 3210 to determine whether to generate an audio mute signal 3152. Thus, component 3300 enables a system (such as system 2300) to determine a context-based predicted expression of a user's face and generate an audio output based at least in part on the context-based predicted expression.

[0237] Fig.34 An example of a component 3400 that may be implemented in a system configured to generate facial expressions of a virtual avatar (such as a fantasy character or creature) is illustrated. The component 3400 includes a facial data generator 230 and a blended shape correction / personalization engine 3430 that is configured to process (e.g., deform) facial data 132 (e.g., a mesh of a user's face) based at least in part on feature data 124 to generate adjusted facial data that is processed by a binding unit 3436 to generate a representation 3408 of the virtual avatar. For example, the component 3400 may be implemented in a system configured to generate facial expressions of a virtual avatar (such as a fantasy character or creature). Figures 1 to 33 The facial data adjuster 130 and the avatar generator 236 may be implemented in any of the systems of FIG. 34 , such as replacing the facial data adjuster 130 and the avatar generator 236 with the blended shape correction / personalization engine 3430 and the binding unit 3436.

[0238] Fig.35 Device 102 is depicted as an implementation 3500 of an integrated circuit 3502 including a sensor-based avatar generator. For example, integrated circuit 3502 includes one or more processors 3516. One or more processors 3516 may correspond to one or more processors 116. One or more processors 3516 include a sensor-based avatar generator 3590. Sensor-based avatar generator 3590 includes feature data generator 120 and facial data adjuster 130, and may also optionally include facial data generator 230 and avatar generator 236; alternatively, as an illustrative, non-limiting example, sensor-based avatar generator 3590 may include feature data generator 120, facial data generator 230, blended shape correction / personalization engine 3430, and binding unit 3436. In some implementations, sensor-based avatar generator 3590 also includes audio decoder 2330.

[0239] The integrated circuit 3502 also includes a sensor input 3504 (such as one or more bus interfaces) to enable the sensor data 106 to be received for processing. The integrated circuit 3502 also includes a signal output 3506 (such as a bus interface) to enable the representation 152 of the avatar 154, the second output data 2322, the audio output 2340, or a combination thereof to be transmitted.

[0240] Integrated circuit 3502 implements sensor-based avatar face generation as a component in a system that includes one or more sensors, such as Fig.36 The mobile phone or tablet device shown, such as Fig.37 The headphones shown, Fig.38 Wearable electronic devices such as Fig.39 The voice-controlled speaker system shown, Fig.40 The camera shown, Fig.41 A virtual reality headset, a mixed reality headset, or an augmented reality headset, such as Fig.42 Augmented reality glasses or mixed reality glasses as shown, Fig.43 A set of in-ear devices as shown or Fig.44 or Fig.45 The vehicle shown.

[0241] As an illustrative, non-limiting example, Fig.36 A specific implementation 3600 is depicted in which the device 102 is a mobile device 3602, such as a phone or tablet device. The mobile device 3602 includes one or more microphones 202, one or more cameras 206, and a display screen 3604. A sensor-based avatar generator 3590 is integrated into the mobile device 3602 and is illustrated using dashed lines to indicate internal components that are generally not visible to a user of the mobile device 3602. In a specific example, the sensor-based avatar generator 3590 can be used to generate a representation 152 of an avatar 154 that can then be displayed at the display screen 3604 (e.g., in conjunction with one or more avatars representing one or more participants in an online activity), an audio output 2340 that can be broadcast at one or more speakers of the mobile device 3602, or a combination thereof.

[0242] Fig.37The device 102 is depicted as a specific implementation 3700 of a headphone device 3702. The headphone device 3702 includes a microphone 202, a camera 206A facing the left eye region, a camera 206B facing the right eye region, a camera 206C facing the mouth, and one or more motion sensors 210. A sensor-based avatar generator 3590 is integrated into the headphone device 3702. In a specific example, the sensor-based avatar generator 3590 can be used to generate feature data 124, adjusted facial data 134, a representation 152 of an avatar 154, an audio output 2340, or a combination thereof, which the headphone device 2702 can send to a second device (not shown) for further processing, displaying the avatar 154, or broadcasting the avatar's voice, or a combination thereof.

[0243] Fig.38 A specific implementation 3800 is depicted in which the device 102 is a wearable electronic device 3802 (illustrated as a "smart watch"). A sensor-based avatar generator 3590 and one or more sensors 104 (e.g., one or more microphones, cameras, motion sensors, or a combination thereof) are integrated into the wearable electronic device 3802. In a specific example, the sensor-based avatar generator 3590 can be used to generate feature data 124, adjusted facial data 134, a representation 152 of an avatar 154, an audio output 2340, or a combination thereof, which the wearable electronic device 3802 can send to a second device (not shown) for further processing, display of the avatar 154, or broadcasting of the avatar's voice, or a combination thereof. In a specific example, the sensor-based avatar generator 3590 can be used to generate a representation 152 of an avatar 154, which can then be displayed at a display screen 2804 (e.g., in conjunction with one or more avatars representing one or more participants in an online activity).

[0244] Fig.39 The device 102 is a specific implementation 3900 of a wireless speaker and voice activated device 3902. The wireless speaker and voice activated device 3902 may have wireless network connectivity and be configured to perform auxiliary operations. A sensor-based avatar generator 3590 and a plurality of sensors 104 (e.g., one or more microphones, cameras, motion sensors, or a combination thereof) are included in the wireless speaker and voice activated device 3902. The wireless speaker and voice activated device 3902 also includes a speaker 3904. In certain aspects, the speaker 3904 corresponds to Fig.23During operation, the sensor-based avatar generator 3590 may be used to generate a representation 152 of an avatar 154, an audio output 2340, or both based on the features of the user captured by the sensor 104, and may also determine whether a keyword is spoken by the user. In response to determining that a keyword is spoken, the wireless speaker and voice-activated device 3902 may perform an auxiliary operation, such as by executing an integrated auxiliary application. The auxiliary operation may include initiating or joining an online activity with one or more other participants, such as an online game or a virtual meeting, in which the user is represented by an avatar 154. For example, the wireless speaker and voice-activated device 3902 may transmit the representation 152 of the avatar 154, the audio output 2340, or both to another device (e.g., a game server), which may include the avatar 154 in a virtual setting shared by other participants. The auxiliary operation may also include adjusting the temperature, playing music, turning on lights, etc. For example, the auxiliary operation is performed in response to receiving a command after a keyword or key phrase (e.g., "Hello, assistant").

[0245] Fig.40 Device 102 is depicted as a specific implementation 4000 of a portable electronic device corresponding to camera device 4002. A sensor-based avatar generator 3590 and a plurality of sensors 104 (e.g., one or more microphones, cameras, motion sensors, or combinations thereof) are included in camera device 4002. During operation, sensor-based avatar generator 3590 may be used to generate feature data 124, adjusted facial data 134, a representation 152 of an avatar 154, audio output 2340, or a combination thereof, which camera device 4002 may send to a second device (not shown) for further processing, display of the avatar 154, or broadcast of the avatar's voice, or a combination thereof.

[0246] Fig.41The device 102 is depicted as including a specific implementation 4100 of a portable electronic device corresponding to an extended reality ("XR") head-mounted device 4102, such as a virtual reality ("VR"), augmented reality ("AR"), or mixed reality ("MR") headset device. A sensor-based avatar generator 3590, a plurality of sensors 104 (e.g., one or more microphones, cameras, motion sensors, or a combination thereof), or a combination thereof, are integrated into the XR head-mounted device 4102. The sensor-based avatar generator 3590 may be used to generate feature data 124, adjusted facial data 134, a representation 152 of an avatar 154, audio output 2340, or a combination thereof based on audio data, image data, motion sensor data, or a combination thereof received from the sensor 104 of the XR head mounted device 4102, and the XR head mounted device 4102 may send these data, representations, or a combination thereof to a second device (e.g., a remote server) for further processing, display the avatar 154 or broadcast the avatar's voice, distribute the avatar 154 to other participants in a virtual setting shared by other participants, or a combination thereof.

[0247] The XR head mounted device 4102 includes a visual interface device that is positioned in front of the user's eyes to enable the display of augmented reality or virtual reality images or scenes to the user while wearing the XR head mounted device 4102. In a particular example, the visual interface device is configured to display the user's avatar 154, one or more avatars associated with other participants in the shared virtual setting, or a combination thereof.

[0248] Fig.42 The device 102 is depicted as including an implementation 4200 of a portable electronic device corresponding to augmented reality or mixed reality glasses 4202. The glasses 4202 include a holographic projection unit 4204 configured to project visual data onto a surface of a lens 4206 or reflect the visual data from the surface of the lens 4206 onto the retina of the wearer. A sensor-based avatar generator 3590, a plurality of sensors 104 (e.g., one or more microphones, cameras, motion sensors, or a combination thereof), or a combination thereof are integrated into the glasses 4202. The sensor-based avatar generator 3590 can be used to generate feature data 124, adjusted facial data 134, a representation 152 of an avatar 154, audio output 2340, or a combination thereof based on audio data, image data, motion sensor data, or a combination thereof received from the sensor 104 of the glasses 4202, and the glasses 4202 can send these data, representations, or a combination thereof to a second device (e.g., a remote server) for further processing, display the avatar 154 or broadcast the avatar's voice, distribute the avatar 154 to other participants in a virtual setting shared by other participants, or a combination thereof.

[0249] In a specific example, the holographic projection unit 4204 is configured to display the avatar 154, one or more other avatars associated with other users or participants, or a combination thereof. For example, the avatar 154, one or more other avatars, or a combination thereof may be superimposed on the user's field of view at a specific location that coincides with the relative location of the user in the shared virtual environment superimposed on the user's field of view.

[0250] Fig.43 An implementation 4300 of a portable electronic device is depicted in which the device 102 includes a pair of earbuds 4306, the pair of earbuds including a first earbud 4302 and a second earbud 4304. Although earbuds are described, it should be understood that the disclosed technology may be applied to other in-ear or over-ear playback devices.

[0251] The first earbud 4302 includes: a first microphone 4320, such as a high signal-to-noise ratio microphone positioned to capture the voice of the wearer of the first earbud 4302; an array of one or more other microphones that are configured to detect ambient sounds and are spatially distributed to support beamforming, which are illustrated as microphones 4322A, 4322B, and 4322C; an "internal" microphone 4324 close to the wearer's ear canal (e.g., to assist in active noise cancellation); and a self-speech microphone 4326, such as a bone conduction microphone configured to convert sound vibrations of the wearer's ear bones or skull into audio signals.

[0252] In certain implementations, microphones 4320, 4322A, 4322B, and 4322C correspond to one or more microphones 202, and audio signals generated by microphones 4320, 4322A, 4322B, and 4322C are provided to sensor-based avatar generator 3590. Sensor-based avatar generator 3590 may be used to generate feature data 124, adjusted facial data 134, representation 152 of avatar 154, audio output 2340, or a combination thereof, which first earbud 4302 may send to a second device (not shown) for further processing, display of avatar 154, or broadcast of avatar's voice, or a combination thereof. In some implementations, sensor-based avatar generator 3590 may be further configured to process audio signals from one or more other microphones of first earbud 4302, such as internal microphone 4324, self-speech microphone 4326, or both.

[0253] The second earbud 4304 can be configured in a substantially similar manner as the first earbud 4302. In some implementations, the sensor-based avatar generator 3590 of the first earbud 4302 is further configured to receive one or more audio signals generated by one or more microphones of the second earbud 4304, such as via wireless transmission between the earbuds 4302, 4304 or via wired transmission in implementations where the earbuds 4302, 4304 are coupled via a transmission line. In other implementations, the second earbud 4304 also includes the sensor-based avatar generator 3590, so that the techniques described herein can be performed by a user wearing either of the earbuds 4302, 4304.

[0254] In some implementations, the earbuds 4302, 4304 are configured to automatically switch between various operating modes, such as a pass-through mode in which ambient sounds are played via the speaker 4330; a playback mode in which non-ambient sounds (e.g., streaming audio corresponding to a phone conversation, media playback, video games, etc.) are played back through the speaker 4330; and an audio zoom mode or beamforming mode in which one or more ambient sounds are emphasized and / or other ambient sounds are suppressed for playback at the speaker 4330. In other implementations, the earbuds 4302, 4304 may support fewer modes, or may support one or more other modes instead of or in addition to the described modes.

[0255] In one illustrative example, the earbuds 4302, 4304 may automatically switch from playback mode to pass-through mode in response to detecting the wearer's voice, and may automatically switch back to playback mode after the wearer has stopped speaking. In some examples, the earbuds 4302, 4304 may operate in two or more modes simultaneously, such as by performing audio zoom on a particular ambient sound (e.g., a dog barking), and playing the audio zoom sound superimposed on the sound being played while the wearer is listening to music (the volume may be reduced while the audio zoom sound is playing). In this example, the wearer may be alerted to the ambient sound associated with the audio event without stopping music playback.

[0256] Fig.44A specific implementation 4400 of the disclosed technology is depicted in a vehicle 4402, which is illustrated as a manned or unmanned aerial device (e.g., a private airplane, a surveillance drone). A sensor-based avatar generator 3590, one or more microphones 202, one or more cameras 206, one or more motion sensors 210, or a combination thereof are integrated into the vehicle 4402.

[0257] In some implementations where the vehicle 4402 is configured to transport a user, one or more of the microphone 202 and the camera 206 may be directed toward the user to capture audio data representing the user's voice and image data representing the user's face for use in generating an avatar of the user with enhanced accuracy or realism. One or more motion sensors 210 may be configured to capture motion data associated with the flight of the vehicle 4402, thereby enabling more accurate prediction of the user's facial expressions (or anticipated future expressions), such as surprise or fear in response to sudden or unexpected movements (e.g., erratic movements due to turbulence), joy or excitement in response to other movements (such as during a climb, descent, or bank maneuver, etc.).

[0258] In some implementations where the vehicle 4402 is configured as a surveillance drone, one or more of the microphone 202 and the camera 206 may be directed toward a particular person being monitored (e.g., a "user") to capture audio data representing the user's voice and image data representing the user's face for use in generating an avatar of the user with enhanced accuracy or realism. One or more motion sensors 210 may be configured to capture motion data associated with the flight of the vehicle 4402, which may be used as a proxy for the user's motion. For purposes of illustration, the vehicle 4402 may be configured to follow the user, and thus the speed of the vehicle 4402 may indicate the user's pace (e.g., stationary, casually walking, sprinting, etc.). In some examples, one or more of the motion sensors 210 may also or alternatively include a camera configured to track the user's body movements, which may provide context for the user's predicted or expected future expressions, such as a sudden turn of the user's head or body indicating that the user is startled, a reclining of the user's body on a chair or flat surface indicating that the user is relaxed, etc.

[0259] Fig.45 Another specific implementation 4500 of the disclosed technology is depicted in a vehicle 4502, which is illustrated as a car. A sensor-based avatar generator 3590, one or more microphones 202, one or more cameras 206, one or more motion sensors 210, or a combination thereof are integrated into the vehicle 4502.

[0260] One or more of microphone 202 and camera 206 may be directed toward a user (e.g., an operator or passenger of vehicle 4502) to capture audio data representing the user's voice and image data representing the user's face for use in generating an avatar of the user with enhanced accuracy or realism. One or more motion sensors 210 may be configured to capture motion data associated with the movement of vehicle 4502, thereby enabling more accurate prediction of the user's facial expressions (or anticipated future expressions), such as surprise or fear corresponding to sudden or unexpected movement (e.g., due to sudden braking, steering, or collision), joy or excitement in response to other movements (such as rapid acceleration or slalom-like movements, etc.).

[0261] In some implementations, the sensor-based avatar generator 3590 may be used to generate feature data 124, adjusted facial data 134, a representation 152 of an avatar 154, an audio output 2340, or a combination thereof, which the vehicle 4502 may send to a second device (not shown) for further processing, display of the avatar 154, or broadcast of the avatar's voice, or a combination thereof. In some implementations, the sensor-based avatar generator 3590 may be used to generate a representation 152 of an avatar 154 that may then be displayed at a display screen 4520 (e.g., in conjunction with one or more avatars representing one or more participants in an online activity), the avatar's voice that may then be broadcast at one or more speakers of the vehicle 4502, or both. For example, the vehicle 4502 may include a set of cameras 206 and microphones 202 and a display device (e.g., a seat back display screen) for each occupant of the vehicle 4502, and a game engine included in the vehicle 4502 may enable multiple occupants of the vehicle to interact in a shared virtual space via their respective avatars. In some embodiments, vehicle 4502 communicates wirelessly with one or more other servers or game engines to enable one or more occupants of vehicle 4502 to interact via their respective avatars with participants from other vehicles or other non-vehicle locations in the shared virtual environment.

[0262] refer to Fig.46 , a specific implementation of the method 4600 of avatar generation is shown. In certain aspects, one or more operations of the method 4600 are performed by the device 102 (such as by one or more processors 116).

[0263] The method 4600 includes processing sensor data at one or more processors to generate feature data at 4602. For example, the feature data generator 120 processes the sensor data 105 to generate the feature data 124.

[0264] The method 4600 also includes generating adjusted facial data at one or more processors based on the feature data, the adjusted facial data corresponding to the avatar facial expression based on the semantic context at 4604. For example, the facial data adjuster 130 generates the adjusted facial data 134 based on the feature data 124, and the adjusted facial data 134 corresponds to the avatar facial expression 156 based on the semantic context 122.

[0265] In some implementations, the sensor data includes audio data (e.g., audio data 204), and the semantic context is based on the meaning of speech (e.g., speech 258) represented in the audio data. In some implementations, the sensor data includes audio data, and the semantic context is based at least in part on audio events detected in the audio data (e.g., audio events 272). In some implementations, the sensor data includes motion sensor data (e.g., motion sensor data 212), and the semantic context is based on motion represented in the motion sensor data (e.g., motion 2240).

[0266] By generating adjusted facial data for an avatar based on feature data, the avatar can be generated with greater accuracy, enhanced realism, or both, and thus the user experience can be improved. In addition, the avatar generation can be performed with reduced latency, which improves the operation of the avatar generation device. Furthermore, the reduced latency also increases the perceived realism of the avatar, further enhancing the user experience.

[0267] Fig.46 The method may be implemented by a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a processing unit (such as a central processing unit (CPU)), a digital signal processing unit (DSP), a controller, another hardware device, a firmware device, or any combination thereof. As an example, Fig.46 The method may be performed by a processor executing instructions, such as reference Fig.48 as described.

[0268] In some implementations, a method of avatar generation includes processing sensor data at one or more processors to determine a semantic context associated with the sensor data. For example, feature data generator 120 processes sensor data 106 to determine semantic context 122. The method also includes generating adjusted facial data at one or more processors based on the determined semantic context and facial data, the adjusted facial data including an avatar facial expression corresponding to the semantic context. For example, facial data adjuster 130 generates adjusted facial data 134 based on facial data 132 corresponding to semantic context 122 and feature data 124, and the adjusted facial data 134 corresponds to avatar facial expression 156 based on semantic context 122.

[0269] refer to Fig.47 , a particular implementation of a method 4700 for avatar audio generation is shown. In certain aspects, one or more operations of the method 4700 are performed by the device 102 (such as by one or more processors 116).

[0270] Method 4700 includes processing, at one or more processors, image data corresponding to a face of a user to generate facial data at 4702. For example, facial data generator 230 processes image data 208 to generate facial data 132.

[0271] The method 4700 includes processing the sensor data at one or more processors to generate feature data at 4704. For example, the feature data generator 120 processes the sensor data 106 to generate the feature data 124.

[0272] Method 4700 includes generating, at one or more processors, a representation of an avatar based on the facial data and the feature data at 4706. For example, facial data adjuster 130 generates adjusted facial data 134 based on facial data 132 and feature data 124, and avatar generator 236 generates representation 152 of avatar 154 based on the adjusted facial data 134.

[0273] The method 4700 includes generating, at one or more processors, an audio output for the avatar based on the sensor data at 4708. For example, the feature data generator 120 generates the second output data 2322, which is processed by the audio decoder 2330 to generate the audio output 2340.

[0274] In some implementations, the sensor data includes audio data representing speech, such as audio data 204. In such implementations, method 4700 may include processing the audio data to generate output data representing speech (such as first output data 2320), and performing voice conversion of the output data to generate converted output data representing converted speech (such as second output data 2322). According to some aspects, a representation of the avatar is generated based on the converted output data, such as via a second audio code 2422 included in the feature data 124 provided to the facial data adjuster 130. In some implementations, method 4700 also includes processing the converted output data to generate an audio output, wherein the audio output corresponds to a modified voice version of the speech. For example, second output data 2322 is processed by audio decoder 2330 to generate audio output 2340.

[0275] In some implementations, the image code 2920 output from the image unit 226 and not based on the audio data 204 is generated based on the image data and independent of any audio data (such as Fig.29 In some implementations, the audio output is generated based on the user profile (such as Fig.30 User profile 3034) to generate audio output.

[0276] In some implementations, the sensor data includes image data and audio data, and an audio output is generated based on the image data and the audio data, such as Fig.23 Image-based output data 22306 and audio-based output data 2304. In some implementations, method 4700 includes determining a context-based predicted expression of a user's face, and generating an audio output based at least in part on the context-based predicted expression, such as a reference Fig.33 The context prediction network 910 and mouth closure detector 3150 are described.

[0277] As non-limiting examples, the audio output may correspond to a modified version of the user's voice, such as when the avatar is a realistic representation of the user, or may correspond to a virtual voice of the avatar when the avatar corresponds to a fictional character or fantasy creature. Because generating the avatar's voice based on varying aspects of the user's voice may cause misalignment between the avatar's facial movements and the avatar's voice, the output data (or information associated with the output data) may also be used (e.g., included in feature data 124) to adjust the avatar's facial expressions to more closely match the avatar's voice.

[0278] refer to Fig.48 , depicts a block diagram of a particular exemplary implementation of a device, and generally designated as 4800. In various implementations, device 4800 may have more Fig.48 More or fewer components may be illustrated. In an exemplary implementation, device 4800 may correspond to device 102. In an exemplary implementation, device 4800 may execute the reference Figures 1 to 47 One or more operations described.

[0279] In a particular implementation, the device 4800 includes a processor 4806 (e.g., a CPU). The device 4800 may include one or more additional processors 4810 (e.g., one or more DSPs). In certain aspects, the processor 116 corresponds to the processor 4806, the processor 4810, or a combination thereof. The processor 4810 may include a speech and music coder-decoder (CODEC) 4808, which includes a speech decoder ("vocoder") encoder 4836, a vocoder decoder 4838, a sensor-based avatar generator 3590, or a combination thereof.

[0280] Device 4800 may include memory 4886 and CODEC 4834. Memory 4886 may include instructions 4856 that may be executed by one or more additional processors 4810 (or processor 4806) to implement the functionality described with reference to sensor-based avatar generator 3590. In certain aspects, memory 4886 corresponds to memory 112 and instructions 4856 include instructions 114. Device 4800 may include a modem 4870 coupled to antenna 4852 via transceiver 4850. Modem 4870 may be configured to send signals to a second device (not shown). Depending on the particular implementation, modem 4870 may correspond to Figure 1 modem 140.

[0281] The device 4800 may include a display 4828 coupled to a display controller 4826. One or more speakers 2302 and one or more microphones 202 may be coupled to a CODEC 4834. The CODEC 4834 may include a digital-to-analog converter (DAC) 4802, an analog-to-digital converter (ADC) 4804, or both. In a particular implementation, the CODEC 4834 may receive an analog signal from the one or more microphones 202, convert the analog signal to a digital signal using the analog-to-digital converter 4804, and provide the digital signal to a voice and music codec 4808. The voice and music codec 4808 may process the digital signal, and the digital signal may be further processed by the sensor-based avatar generator 3590. In a particular implementation, the voice and music codec 4808 may provide the digital signal to the CODEC 4834. The CODEC 4834 may convert the digital signal to an analog signal using the digital-to-analog converter 4802 and may provide the analog signal to the one or more speakers 2302.

[0282] In a particular implementation, the device 4800 may be included in a system-in-package or system-on-chip device 4822. In a particular implementation, the memory 4886, the processor 4806, the processor 4810, the display controller 4826, the CODEC 4834, and the modem 4870 are included in the system-in-package or system-on-chip device 4822. In a particular implementation, the input device 4830, the one or more cameras 206, the one or more motion sensors 210, and the power supply 4844 are coupled to the system-in-package or system-on-chip device 4822. In addition, in a particular implementation, as shown in FIG. Fig.48As shown, the display 4828, input device 4830, one or more speakers 2302, one or more microphones 202, one or more cameras 206, one or more motion sensors 210, antenna 4852, and power supply 4844 are located outside the system-on-chip device 4822. In a particular implementation, each of the display 4828, input device 4830, one or more speakers 2302, one or more microphones 202, one or more cameras 206, one or more motion sensors 210, antenna 4852, and power supply 4844 can be coupled to a component of the system-on-chip device 4822, such as an interface or a controller.

[0283] Device 4800 may include a smart speaker, a speaker bar, a mobile communication device, a smart phone, a cellular phone, a laptop, a computer, a tablet device, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aerial vehicle, a home automation system, a voice-activated device, a wireless speaker and voice-activated device, a portable electronic device, an automobile, a vehicle, a computing device, a communication device, an Internet of Things (IoT) device, an extended reality (XR) device, a base station, a mobile device, or any combination thereof.

[0284] In conjunction with the described implementation, an apparatus includes a component for processing sensor data to generate feature data. For example, the component for processing sensor data to generate feature data may correspond to feature data generator 120, processor 116 or a component thereof, audio unit 222, image unit 226, motion unit 238, audio network 310, speech signal processing unit 410, ASR-based processing unit 510, deep learning model based on self-supervised learning 610, audio / image network 710, event detector 810 or 1402, context prediction network 910, prediction override unit 930, prediction validator 1030, context-based future speech prediction network 1210, 1310 or 1410, representation generator 1230 or 1430, speech representation generator 1330, processor 3706, processor 3710, one or more other circuits or components configured to process sensor data to generate feature data, or any combination thereof.

[0285] The apparatus also includes a component for generating adjusted facial data based on the feature data, the adjusted facial data corresponding to the avatar facial expression based on the semantic context. For example, the component for generating adjusted facial data may correspond to the processor 116, the facial data adjuster 130, the encoder portion 1504, the decoder portion 1502, the neural network 1630 or 1730, the neural network layer 1702 or 1704, the cascade unit 1804, the fusion unit 1904, the fused neural network 2004, the processor 3706, the processor 3710, one or more other circuits or components configured to generate adjusted facial data, or any combination thereof.

[0286] In conjunction with the described implementations, an apparatus includes means for processing image data corresponding to a user's face to generate facial data. For example, the means for processing image data corresponding to a user's face to generate facial data may correspond to feature data generator 120, processor 116 or a component thereof, facial data generator 230, processor 4806, processor 4810, one or more other circuits or components configured to process image data corresponding to a user's face to generate facial data, or any combination thereof.

[0287] In conjunction with the described implementations, an apparatus includes components for processing sensor data to generate feature data. For example, the components for processing sensor data to generate feature data may correspond to the feature data generator 120, the processor 116 or a component thereof, the audio unit 222, the image unit 226, the motion unit 238, the audio network 310, the speech signal processing unit 410, the ASR-based processing unit 510, the deep learning model based on self-supervised learning 610, the audio / image network 710, the event detector 810 or 1402, the context prediction network 910, the prediction override unit 930, the prediction verifier 1030, the context-based future speech prediction network 1210, 1310 or 1410, the representation generator 1230 or 1430, the speech representation generator 1330, the speech converter 2310, 2410, 2510, 2510, 2610 or 2710, the speech conversion unit 2812, the processor 4806, the processor 4810, one or more other circuits or components configured to process sensor data to generate feature data, or any combination thereof.

[0288] The apparatus also includes a component for generating a representation of the avatar based on the facial data and the feature data. For example, the component for generating the representation of the avatar may correspond to the processor 116, the facial data adjuster 130, the avatar generator 236, the encoder portion 1504, the decoder portion 1502, the neural network 1630 or 1730, the neural network layer 1702 or 1704, the cascade unit 1804, the fusion unit 1904, the fused neural network 2004, the hybrid shape correction / personalization engine 3430, the binding unit 3436, the processor 4806, the processor 4810, one or more other circuits or components configured to generate a representation of the avatar, or any combination thereof.

[0289] The apparatus also includes a component for generating an audio output for the avatar based on the sensor data. For example, the component for generating an audio output for the avatar based on the sensor data may correspond to the feature data generator 120, the processor 116 or a component thereof, the audio unit 222, the image unit 226, the audio network 310, the speech signal processing unit 410, the ASR-based processing unit 510, the deep learning model based on self-supervised learning 610, the audio / image network 710, the context prediction network 910, the prediction override unit 930, the prediction verifier 1030, the context-based future speech prediction network 1210, 1310 or 1410, the speech A voice converter 2310, 2410, 2510, 2510, 2610 or 2710, an audio decoder 2330, one or more speakers 2303, an output speech generator 2412, 2512, 2612 or 2712, a speech conversion unit 2810, a character-specific audio decoder 2930, a prediction validator 3030, a mouth closure detector 3150, a processor 4806, a processor 4810, one or more other circuits or components configured to generate an audio output for an avatar based on sensor data, or any combination thereof.

[0290] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device such as memory 4886) includes instructions (e.g., instructions 4856) that, when executed by one or more processors (e.g., one or more processors 4810 or processor 4806), cause the one or more processors to process sensor data (e.g., sensor data 106) to generate feature data (e.g., feature data 124). The instructions, when executed by the one or more processors, also cause the one or more processors to generate adjusted facial data (e.g., adjusted facial data 134) based on the feature data, the adjusted facial data corresponding to an avatar facial expression (e.g., avatar facial expression 156) based on a semantic context (e.g., semantic context 122).

[0291] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device such as memory 4886) includes instructions (e.g., instructions 4856) that, when executed by one or more processors (e.g., one or more processors 4810 or processor 4806), cause the one or more processors to process image data (e.g., image data 208) corresponding to a user's face to generate facial data (e.g., facial data 132). The instructions, when executed by the one or more processors, also cause the one or more processors to process sensor data (e.g., sensor data 106) to generate feature data (e.g., feature data 124). The instructions, when executed by the one or more processors, also cause the one or more processors to generate a representation of an avatar (e.g., representation 152 of avatar 154) based on the facial data and the feature data. The instructions, when executed by the one or more processors, also cause the one or more processors to generate an audio output for the avatar based on the sensor data (e.g., audio output 2340).

[0292] The present disclosure includes the following first group of embodiments.

[0293] According to embodiment 1, a device includes: a memory, the memory being configured to store instructions; and one or more processors, the one or more processors being configured to: process sensor data to generate feature data; and generate adjusted facial data based on the feature data, the adjusted facial data corresponding to an avatar facial expression based on a semantic context.

[0294] Embodiment 2 includes a device according to embodiment 1, wherein the one or more processors are further configured to: process image data corresponding to a person's face to generate facial data; further generate the adjusted facial data based on the facial data; and generate a representation of an avatar having the avatar's facial expression based on the adjusted facial data.

[0295] Embodiment 3 includes the apparatus of embodiment 1 or embodiment 2, wherein the sensor data includes audio data, and wherein the semantic context is based on the meaning of speech represented in the audio data.

[0296] Embodiment 4 includes the apparatus of embodiment 3, wherein the semantic context is based on the meaning of words detected in the speech.

[0297] Embodiment 5 includes the apparatus of embodiment 3 or embodiment 4, wherein the semantic context is based on the meaning of at least one phrase or sentence detected in the speech.

[0298] Embodiment 6 includes the apparatus of any one of Embodiments 3 to 5, wherein the speech comprises at least a portion of a conversation, and wherein the semantic context is based on characteristics of the conversation.

[0299] Embodiment 7 includes an apparatus according to embodiment 6, wherein the characteristic includes a type of relationship between participants in the conversation.

[0300] Embodiment 8 includes an apparatus according to embodiment 6 or embodiment 7, wherein the characteristic includes a social context of the conversation.

[0301] Embodiment 9 includes the apparatus of any one of Embodiments 1 to 8, wherein the sensor data includes audio data, and wherein the semantic context is based on emotions associated with speech represented in the audio data.

[0302] Embodiment 10 includes the apparatus of Embodiment 9, wherein the one or more processors are configured to process the audio data to predict the emotion.

[0303] Embodiment 11 includes an apparatus according to Embodiment 9 or Embodiment 10, wherein the adjusted facial data causes the avatar's facial expression to represent the emotion.

[0304] Embodiment 12 includes the apparatus of any one of Embodiments 1 to 11, wherein the semantic context is based on motion sensor data included in the sensor data.

[0305] Embodiment 13 includes the apparatus of Embodiment 12, wherein the one or more processors are configured to determine the semantic context based on comparing motion represented in the motion sensor data to at least one motion threshold.

[0306] Embodiment 14 includes an apparatus as in Embodiment 12 or Embodiment 13, wherein the motion sensor data includes head tracker data indicative of at least one of movement or orientation of a user's head.

[0307] Embodiment 15 includes a device according to Embodiment 12 or Embodiment 13, wherein the motion sensor data includes head tracker data indicating movement of the user's head.

[0308] Embodiment 16 includes a device according to Embodiment 12 or Embodiment 13, wherein the motion sensor data includes head tracker data indicating the orientation of the user's head.

[0309] Embodiment 17 includes the apparatus of any one of Embodiments 1 to 16, wherein the sensor data includes audio data, and wherein the semantic context is based at least in part on audio events detected in the audio data.

[0310] Embodiment 18 includes the apparatus of any one of Embodiments 1 to 17, wherein the one or more processors are configured to determine the avatar facial expression further based on a user profile.

[0311] Embodiment 19 includes the device of any one of Embodiments 1 to 18, further comprising one or more microphones configured to generate audio data included in the sensor data.

[0312] Embodiment 20 includes a device according to any one of Embodiments 1 to 19, wherein the device further includes one or more motion sensors, and the one or more motion sensors are configured to generate motion data included in the sensor data.

[0313] Embodiment 21 includes a device according to any one of Embodiments 1 to 20, wherein the device also includes one or more cameras configured to generate image data included in the sensor data.

[0314] Embodiment 22 includes a device according to any one of Embodiments 1 to 21, the device also comprising a display device configured to display a representation of the avatar with the avatar facial expression based on the adjusted facial data.

[0315] Embodiment 23 includes the apparatus of any one of Embodiments 1 to 22, further comprising a modem, wherein at least a portion of the sensor data is received from a second device via the modem.

[0316] Embodiment 24 includes a device according to any one of Embodiments 1 to 23, wherein the one or more processors are further configured to transmit a representation of the avatar having the avatar facial expression to a second device.

[0317] Embodiment 25 includes a device according to any one of Embodiments 1 to 24, wherein the one or more processors are integrated into an extended reality device.

[0318] According to embodiment 26, an avatar generation method includes: processing sensor data at one or more processors to generate feature data; and generating adjusted facial data based on the feature data at the one or more processors, wherein the adjusted facial data corresponds to the avatar's facial expression based on the semantic context.

[0319] Embodiment 27 includes the method of embodiment 26, wherein the sensor data includes audio data, and wherein the semantic context is based on the meaning of speech represented in the audio data.

[0320] Embodiment 28 includes the method of embodiment 27, wherein the semantic context is based on the meaning of words detected in the speech.

[0321] Embodiment 29 includes the method of embodiment 27 or embodiment 28, wherein the semantic context is based on the meaning of at least one phrase or sentence detected in the speech.

[0322] Embodiment 30 includes the method of any one of Embodiments 27 to 29, wherein the speech comprises at least a portion of a conversation, and wherein the semantic context is based on characteristics of the conversation.

[0323] Embodiment 31 comprises the method according to embodiment 30, wherein the characteristic comprises the type of relationship between the participants of the conversation.

[0324] Embodiment 32 comprises a method according to embodiment 30 or embodiment 31, wherein the characteristic comprises a social context of the conversation.

[0325] Embodiment 33 includes a method according to any one of Embodiments 26 to 32, wherein the sensor data includes audio data, and wherein the semantic context is based on emotions associated with speech represented in the audio data.

[0326] Embodiment 34 comprises the method according to embodiment 33, and the method also includes processing the audio data to predict the emotion.

[0327] Embodiment 35 includes the method of embodiment 33 or embodiment 34, wherein the adjusted facial data causes the avatar's facial expression to represent the emotion.

[0328] Embodiment 36 includes a method according to any one of Embodiments 26 to 35, wherein the sensor data includes audio data, and wherein the semantic context is based at least in part on audio events detected in the audio data.

[0329] Embodiment 37 includes the method of any one of Embodiments 26 to 36, wherein the sensor data includes motion sensor data, and wherein the semantic context is based on motion represented in the motion sensor data.

[0330] Embodiment 38 includes the method of Embodiment 37, wherein the semantic context is determined based on comparing the motion represented in the motion sensor data to at least one motion threshold.

[0331] Embodiment 39 includes the method of Embodiment 37 or Embodiment 38, wherein the motion sensor data includes head tracker data indicating at least one of movement or orientation of a user's head.

[0332] Embodiment 40 includes a method according to any one of embodiments 26 to 39, the method further comprising: processing image data corresponding to a user's face to generate facial data; further generating the adjusted facial data based on the facial data; and generating a representation of an avatar having the avatar's facial expression based on the adjusted facial data.

[0333] Embodiment 41 includes the method of any one of Embodiments 26 to 40, wherein the avatar facial expression is further determined based on a user profile.

[0334] Embodiment 42 includes the method of any one of Embodiments 26 to 41, further comprising receiving audio data included in the sensor data from one or more microphones.

[0335] Embodiment 43 includes the method of any one of Embodiments 26 to 42, further comprising receiving motion data included in the sensor data.

[0336] Embodiment 44 includes the method of any one of Embodiments 26 to 43, further comprising receiving image data included in the sensor data from one or more cameras.

[0337] Embodiment 45 includes the method of any one of Embodiments 26 to 44, further comprising displaying a representation of the avatar having the avatar facial expression based on the adjusted facial data.

[0338] Embodiment 46 includes the method of any one of Embodiments 26 to 45, further comprising receiving at least a portion of the sensor data from a second device.

[0339] Embodiment 47 includes the method of any one of Embodiments 26 to 46, further comprising transmitting a representation of the avatar having the avatar's facial expression to a second device.

[0340] Embodiment 48 includes a method according to any one of Embodiments 26 to 47, wherein the one or more processors are integrated into an extended reality device.

[0341] According to embodiment 49, a device includes: a memory configured to store instructions; and a processor configured to execute the instructions to perform the method according to any one of embodiments 26 to 48.

[0342] According to embodiment 50, a non-transitory computer-readable medium includes instructions, which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of embodiments 26 to 48.

[0343] According to embodiment 51, an apparatus includes components for performing the method according to any one of embodiments 26 to 48.

[0344] According to embodiment 52, a non-transitory computer-readable medium includes: instructions, which, when executed by one or more processors, cause the one or more processors to: process sensor data to generate feature data; and generate adjusted facial data based on the feature data, wherein the adjusted facial data corresponds to an avatar facial expression based on a semantic context.

[0345] According to embodiment 53, an apparatus includes a component for processing sensor data to generate feature data; and a component for generating adjusted facial data based on the feature data, wherein the adjusted facial data corresponds to an avatar facial expression based on a semantic context.

[0346] The present disclosure includes the following second group of embodiments.

[0347] According to embodiment 1, a device includes: a memory configured to store instructions; and one or more processors, wherein the one or more processors are configured to: process image data corresponding to a user's face to generate facial data; process sensor data to generate feature data; generate a representation of an avatar based on the facial data and the feature data; and generate an audio output for the avatar based on the sensor data.

[0348] Embodiment 2 comprises the apparatus according to embodiment 1, wherein the sensor data comprises audio data representing speech, and wherein the one or more processors are configured to: process the audio data to generate output data representing the speech; and perform speech conversion of the output data to generate converted output data representing the converted speech.

[0349] Embodiment 3 includes the apparatus of Embodiment 2, wherein the representation of the avatar is generated based on the converted output data.

[0350] Embodiment 4 includes the apparatus of Embodiment 2 or Embodiment 3, wherein the one or more processors are configured to process the converted output data to generate the audio output, the audio output corresponding to a modified voice version of the speech.

[0351] Embodiment 5 includes the apparatus of any one of Embodiments 2 to 4, wherein the output data corresponds to an audio code, and wherein the speech conversion corresponds to a latent spatial speech conversion.

[0352] Embodiment 6 includes the device of embodiment 1, wherein the sensor data includes the image data, and wherein the audio output is generated based on the image data.

[0353] Embodiment 7 includes the apparatus of embodiment 6, wherein the audio output is generated independently of any audio data.

[0354] Embodiment 8 includes the apparatus of any one of Embodiments 1 to 7, wherein the one or more processors are configured to generate the audio output further based on a user profile.

[0355] Embodiment 9 includes the apparatus of Embodiment 1, wherein the sensor data includes the image data and the audio data, and wherein the audio output is generated based on the image data and the audio data.

[0356] Embodiment 10 includes a device according to embodiment 9, wherein the one or more processors are configured to predict whether the user's mouth is closed based on the image data and the audio data, and mute the audio output based on predicting that the user's mouth is closed.

[0357] Embodiment 11 includes a device according to any one of embodiments 1 to 10, wherein the one or more processors are configured to: determine a context-based predicted expression of the user's face; and generate the audio output based at least in part on the context-based predicted expression.

[0358] Embodiment 12 includes the apparatus of any one of Embodiments 1-11, wherein the audio output corresponds to a modified version of the user's voice.

[0359] Embodiment 13 includes the apparatus of any one of Embodiments 1 to 11, wherein the audio output corresponds to a virtual voice of the avatar.

[0360] Embodiment 14 includes the device of any one of Embodiments 1 to 13, further comprising one or more microphones configured to generate audio data included in the sensor data.

[0361] Embodiment 15 includes a device according to any one of Embodiments 1 to 14, wherein the device further includes one or more cameras configured to generate the image data.

[0362] Embodiment 16 includes the device of any one of Embodiments 1-15, further comprising one or more speakers configured to play out the audio output.

[0363] Embodiment 17 includes the device of any one of Embodiments 1 to 16, further comprising a display device configured to display the representation of the avatar.

[0364] Embodiment 18 includes the device of any one of Embodiments 1 to 17, further comprising a modem, wherein the image data, one or more sets of sensor data, or both are received from a second device via the modem.

[0365] Embodiment 19 includes the device of any one of Embodiments 1 to 18, wherein the one or more processors are further configured to transmit the representation of the avatar, the audio output, or both to a second device.

[0366] Embodiment 20 includes a device according to any one of embodiments 1 to 19, wherein the one or more processors are integrated into an extended reality device.

[0367] According to embodiment 21, a method for avatar audio generation includes: processing image data corresponding to a user's face at one or more processors to generate facial data; processing sensor data at the one or more processors to generate feature data; generating a representation of an avatar based on the facial data and the feature data at the one or more processors; and generating an audio output for the avatar based on the sensor data at the one or more processors.

[0368] Embodiment 22 comprises a method according to embodiment 21, wherein the sensor data comprises audio data representing speech, the method further comprising: processing the audio data to generate output data representing the speech; and performing speech conversion of the output data to generate converted output data representing the converted speech.

[0369] Embodiment 23 includes the method of embodiment 22, wherein the representation of the avatar is generated based on the converted output data.

[0370] Embodiment 24 includes the method of Embodiment 22 or Embodiment 23, further comprising processing the converted output data to generate the audio output, the audio output corresponding to a modified voice version of the speech.

[0371] Embodiment 25 includes the method of embodiment 21, wherein the audio output is generated based on the image data and independently of any audio data.

[0372] Embodiment 26 includes the method of any one of Embodiments 21 to 25, wherein the audio output is generated further based on a user profile.

[0373] Embodiment 27 includes the method according to embodiment 21, wherein the sensor data includes the image data and the audio data, and wherein the audio output is generated based on the image data and the audio data.

[0374] Embodiment 28 includes the method according to embodiment 21, the method further comprising: determining a context-based predicted expression of the user's face; and generating the audio output based at least in part on the context-based predicted expression.

[0375] According to embodiment 29, a device includes: a memory configured to store instructions; and a processor configured to execute the instructions to perform the method according to any one of embodiments 21 to 28.

[0376] According to embodiment 30, a non-transitory computer-readable medium includes instructions, which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of embodiments 21 to 28.

[0377] According to embodiment 31, an apparatus includes components for performing the method according to any one of embodiments 21 to 28.

[0378] According to embodiment 32, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to: process image data corresponding to a user's face to generate facial data; process sensor data to generate feature data; generate a representation of an avatar based on the facial data and the feature data; and generate an audio output for the avatar based on the sensor data.

[0379] According to embodiment 33, a device includes: a component for processing image data corresponding to a user's face to generate facial data; a component for processing sensor data to generate feature data; a component for generating a representation of an avatar based on the facial data and the feature data; and a component for generating an audio output for the avatar based on the sensor data.

[0380] It will also be appreciated by the skilled person that the various illustrative logic blocks, configurations, modules, circuits, and algorithmic steps described in conjunction with the specific implementations disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or a combination of the two. Various illustrative components, blocks, configurations, modules, circuits, and steps have been generally described above in terms of their functions. Whether such functions are implemented as hardware or processor executable instructions depends on the specific application and the design constraints imposed on the overall system. The skilled person can implement the described functions in different ways for each specific application, and such specific implementation decisions will not be interpreted as causing a departure from the scope of the present disclosure.

[0381] The steps of the method or algorithm described in conjunction with the specific implementation disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integral with the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In an alternative, the processor and the storage medium may reside in a computing device or a user terminal as discrete components.

[0382] The previous description of the disclosed aspects is provided to enable those skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein, but should be granted the broadest scope that may be consistent with the principles and novel features as defined by the following claims.

Claims

1. A device, comprising: a memory configured to store instructions; as well as One or more processors configured to: processing image data corresponding to a face of a user to generate facial data; processing sensor data to generate feature data; generating a representation of an avatar based on the facial data and the feature data; and An audio output is generated for the avatar based on the sensor data.

2. The apparatus of claim 1 , wherein the sensor data comprises audio data representative of speech, and wherein the one or more processors are configured to: processing the audio data to generate output data representing the speech; and A voice conversion of the output data is performed to generate converted output data representing converted speech.

3. The apparatus of claim 2, wherein the representation of the avatar is generated based on the converted output data.

4. The device of claim 2, wherein the one or more processors are configured to process the converted output data to generate the audio output, the audio output corresponding to a modified voice version of the speech.

5. The apparatus of claim 2, wherein the output data corresponds to an audio code, and wherein the speech conversion corresponds to a latent spatial speech conversion.

6. The device of claim 1, wherein the sensor data comprises the image data, and wherein the audio output is generated based on the image data.

7. The apparatus of claim 6, wherein the audio output is generated independently of any audio data.

8. The device of claim 1, wherein the one or more processors are configured to generate the audio output further based on a user profile.

9. The device of claim 1, wherein the sensor data comprises the image data and the audio data, and wherein the audio output is generated based on the image data and the audio data.

10. The device of claim 9, wherein the one or more processors are configured to predict whether the user's mouth is closed based on the image data and the audio data, and to mute the audio output based on predicting that the user's mouth is closed.

11. The device of claim 1 , wherein the one or more processors are configured to: determining a context-based predicted expression of the user's face; and The audio output is generated based at least in part on the context-based predicted expression.

12. The apparatus of claim 1, wherein the audio output corresponds to a modified version of the user's voice.

13. The apparatus of claim 1, wherein the audio output corresponds to a virtual voice of the avatar.

14. The device of claim 1, further comprising one or more microphones configured to generate audio data included in the sensor data.

15. The device of claim 1, further comprising one or more cameras configured to generate the image data.

16. The device of claim 1, further comprising one or more speakers configured to play out the audio output.

17. The device of claim 1, further comprising a display device configured to display the representation of the avatar.

18. The device of claim 1, further comprising a modem, wherein the image data, one or more sets of sensor data, or both are received from a second device via the modem.

19. The device of claim 1, wherein the one or more processors are further configured to transmit the representation of the avatar, the audio output, or both to a second device.

20. The device of claim 1, wherein the one or more processors are integrated into an extended reality device.

21. A method for generating avatar audio, the method comprising: processing, at one or more processors, image data corresponding to a face of a user to generate facial data; processing the sensor data at the one or more processors to generate characteristic data; generating, at the one or more processors, a representation of an avatar based on the facial data and the feature data; as well as An audio output for the avatar is generated at the one or more processors based on the sensor data.

22. The method of claim 21, wherein the sensor data comprises audio data representing speech, the method further comprising: processing the audio data to generate output data representing the speech; as well as A voice conversion of the output data is performed to generate converted output data representing converted speech.

23. The method of claim 22, wherein the representation of the avatar is generated based on the converted output data.

24. The method of claim 22, further comprising processing the converted output data to generate the audio output, the audio output corresponding to a modified voice version of the speech.

25. The method of claim 21, wherein the audio output is generated based on the image data and independent of any audio data.

26. The method of claim 21, wherein the audio output is generated further based on a user profile.

27. The method of claim 21, wherein the sensor data comprises the image data and the audio data, and wherein the audio output is generated based on the image data and the audio data.

28. The method according to claim 21, further comprising: determining a context-based predicted expression of the user's face; as well as The audio output is generated based at least in part on the context-based predicted expression.

29. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: processing image data corresponding to a face of a user to generate facial data; processing sensor data to generate feature data; generating a representation of an avatar based on the facial data and the feature data; and An audio output is generated for the avatar based on the sensor data.

30. An apparatus, comprising: means for processing image data corresponding to a user's face to generate facial data; A component for processing sensor data to generate feature data; means for generating a representation of an avatar based on said facial data and said feature data; as well as Means for generating an audio output for the avatar based on the sensor data.

Images