Network system employing distributed generative modeling with jointly-trained neural network communications pathways

Abstract

A distributed generative modeling scheme employs a jointly-trained neural network path (118) implementing a generative model (112) at a UE (110) instead of wholly at a remote application server (106). An encoder neural network (120) at a base station (108) between the application server and the UE operates to, in effect, encode input prompt information (116) from the application server for efficient transmission to the UE. At the UE, a decoder neural network (122) is employed to, in effect, decode the encoded prompt information to recover a representation (126) of the original prompt information. The decoded prompt information is provided as an input to the generative model at the UE for processing, along with, in some instances, one or more additional inputs such as local sensor data (144), to generate output content (128) that is then processed by one or more UX components (114) associated with the UE.

Description

NETWORK SYSTEM EMPLOYING DISTRIBUTED GENERATIVE MODELING WITH JOINTLY-TRAINED NEURAL NETWORK COMMUNICATIONS PATHWAYSBACKGROUND

[0001] Generative modeling facilitates enriched user experience through the generation of high-detail, immersive content. Use cases for such generative modeling include augmented reality / virtual reality (ARA / R), holographic telepresence, natural language processing (NLP), and the like. However, in many instances the source of the generated content (e.g., an application server) and the consumer of the generated content (e.g., ARA / R goggles, personal computer, smartphone, or other user equipment) are separately implemented and connected via one or more network channels. While bandwidth and latency in such network channels are constantly improving, the high data communication rates and low latencies required for effective transmission of generative-model-created content between source and consumer will continue to pose significant challenges for networked implementations.BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The present disclosure is better understood, and its numerous features and advantages made apparent to those skilled in the art, by referencing the accompanying drawings. The use of the same reference symbols in different drawings indicates similar or identical items.

[0003] FIG. 1 is a diagram illustrating an example wireless system employing a jointly-trained neural network path with distributed generative modeling in accordance with some embodiments.

[0004] FIG. 2 is a diagram illustrating an example configuration of a user equipment of the wireless system of FIG. 1 in accordance with some embodiments.

[0005] FIG. 3 is a diagram illustrating an example configuration of a network component of the wireless system of FIG. 1 in accordance with some embodiments.

[0006] FIG. 4 is a diagram illustrating a machine learning module employing a neural network for use in a jointly-trained neural network path with distributed generative modeling in accordance with some embodiments.

[0007] FIG. 5 is a signal diagram illustrating an example method for remote generative prompting and local generative modeling in the wireless system of FIG. 1 in accordance with some embodiments.

[0008] FIG. 6 is a signal diagram illustrating an example method for local sensor feedback in the wireless system of FIG. 1 in accordance with some embodiments.

[0009] FIG. 7 is a flow diagram illustrating an example method for joint training of a candidate set of neural network architecture configurations for a distributed pathway in accordance with some embodiments.DETAILED DESCRIPTION

[0010] Conventionally, in a networked system utilizing generative modeling, a remote application server executes software that implements one or more generative models, such as a generative adversarial network (GAN) or a generative pre-trained transformer (GPT), to generate graphic, audio, and / or textual content based on a set of prompts or other inputs. The resulting generated content is transmitted to a user equipment (UE), which then processes the content for output to a user via one or more user experience (UX) devices, such as AR goggles, a panel display, a speaker, and the like. In many instances, the generated content is represented by an extensive amount of data, such as high-resolution still images or high-resolution video, the transmission of which may overwhelm the communication capabilities of the network channel(s) connecting the application server and the UE.

[0011] To more efficiently deliver generated content to the consuming device, in implementations a network utilizes a distributed generative modeling approach that employs a jointly-trained neural network pathway that implements a generative model at the UE instead of wholly at the remote application server. This neural network pathway includes at least an encoder neural network, a decoder neural network, and a generative model that have been jointly trained. The decoder neural network is implemented between the application server and the UE (e.g., at the base station and / or core network function serving to wirelessly connect the UE to the rest of the network) and operates to, in effect, encode input prompt information from the application server for efficient transmission to the UE. At the UE, the decoder neural network is employed to, in effect, decode the encoded prompt information to recovera lossless or lossy representation of the original prompt information. The decoded prompt information is provided as an input to the generative model at the UE for processing, along with, in some implementations, one or more additional inputs such as local sensor data, to generate output content that is then processed by one or more UX components associated with the UE. Thus, in this approach, rather than using a generative model at the application server to generate extensive data representing generated content to be consumed by the UE and then transmitting this data from the application server to the UE via a potentially-limited network channel, the network can determine input prompt information suitable to causing a UE- implemented generative model to generate the desired output content, and then use the jointly-trained encoder neural network at the network side and the decoder neural network at the UE side to efficiently transmit this prompt information (which often may be orders of magnitude smaller than the resulting generated content) to the UE for implementation at the local generative model.

[0012] Different UEs may have different capabilities for implementing generative models. Moreover, the ability of a given UE to implement a particular generative model may change over time due to a changing context for the UE (e.g., a changing battery reserve or a change in the current headroom for a thermal limit). Further, the network channel capabilities may differ between UEs, and also may vary over time for the same UE. Accordingly, in some implementations the system has access to a plurality of candidate neural network pathways having different combinations of neural network architectures for the encoder neural network, decoder neural network, and generative model to reflect different implementation circumstances. The UE thus can report its present capabilities, its present context, and / or the UE-observed present network conditions to the network. The network then uses this reported information to select a suitable neural network path from the candidate neural network pathways for implementation in the pathway between the application server and the UE. The network can then direct the base station or other network edge component to use the corresponding encoder neural network and direct the UE to use the corresponding decoder neural network and generative model from the selected neural network path.

[0013] In some implementations, the generative model further may be trained to concurrently receive local sensor data of the UE as input for processing by the localgenerative model along with the prompt information (or decoded representation thereof) supplied by the application server. To illustrate, in an implementation in which the generative model operates to generate video representative of an AR overlay that incorporates local objects, then the UE may provide sensor data from a position sensor pertaining to the position or motion of these local objects (e.g., video imagery of the local objects, geolocation information for the local objects, etc.) to the generative model for use in generating the next sequence of frames for the AR overlay. The UE and network also may feed this local sensor data (and / or user input) back to the application server for processing via a feedback pathway in a neural network path in which an encoder neural network at the UE, in effect, encodes the local sensor data and / or user input data and the output is transmitted to a base station or other edge component of the network. This edge component of the network implements a decoder neural network (which has been jointly-trained with the encoder neural network of the UE) to, in effect, decode the received input to recover the local sensor data, or a representation thereof.

[0014] For ease of illustration, implementations of systems and techniques directed to a network-based distributed generative modeling approach are described herein in an example context of the network as cellular system in which the core network component or edge component is a base station (BS) or similar network edge component and the UE is a cellular UE wirelessly connected to the BS or similar network edge component using cellular signaling. However, as also described below, these systems and techniques also may be employed in a non-cellular network, such as a network in which the UE is a user device with more-limited capabilities connected to another device with less-limited capabilities via a wired or wireless connection, such as via a universal serial bus (USB) connection, a Bluetooth (TM) or other wireless personal area network (PAN) connection, a Wireless Fidelity (WiFi) or other wireless local area network (WLAN) connection, and the like. As such, reference to a cellular network or cellular-specific components likewise applies to a non-cellular network or analogous non-cellular-specific components unless otherwise noted.

[0015] FIG. 1 illustrates an example wireless communications network 100 employing a distributed generative model scheme using jointly-trained neural networks in accordance with some embodiments. In the depicted example, the wirelesscommunication network 100 is a cellular network including a core network 102 coupled to one or more application servers 106 via one or more wide area networks (WANs) 104 or other packet data networks (PDNs), such as the Internet. The core network 102 includes a plurality of base stations or other network edge components, including the illustrated base station (BS) 108, to support wireless communication with one or more UEs, such as the illustrated UE 110, via radio frequency (RF) signaling using one or more applicable radio access technologies (RATs) as specified by one or more communications protocols or standards. As such, the BS 108 operates as the wireless interface between the UE 110 and various networks and services provided by the core network 102 and other networks, such as packet- switched (PS) data services, circuit-switched (CS) services, and the like. Conventionally and as used herein, communication of signaling from the BS 108 to the UE 110 is referred to as “downlink” or “DL” whereas communication of signaling from the UE 110 to the BS 108 is referred to as “uplink” or “UL.”

[0016] The BS 108 can employ any of a variety of RATs, such as operating as a NodeB (or base transceiver station (BTS)) for a Universal Mobile Telecommunications System (UMTS) RAT (also known as “3G”), operating as an enhanced NodeB (eNodeB) for a Third Generation Partnership Project (3GPP) Long Term Evolution (LTE) RAT, operating as a 5G node B (“gNB”) for a 3GPP Fifth Generation (5G) New Radio (NR) RAT, and the like. In this example cellular implementation, the UE 110, in turn, can implement any of a variety of electronic devices operable to communicate with the BS 108 via a suitable RAT, including, for example, a mobile cellular phone, a cellular-enabled tablet computer or laptop computer, a desktop computer, a cellular-enabled video game system, a server, a cellular-enabled appliance, a cellular-enabled automotive communications system, a cellular-enabled smartwatch or other wearable device, and the like.

[0017] To implement generative modeling functionality, a conventional cellular network or other conventional communications network would either employ a generative model entirely at an application server or other component of the network or entirely at the UE device. In such approaches, an input prompt is generated at, or local to, the same component implementing the generative model ti. In instances in which the generative model is implemented at a network component, considerable resources may be consumed in transmitting the resulting generated output to a UEfor further processing. For example, a generative model that operates to generate a series of video frames representative of, for example, a video teleconference would require the transmission of a considerable amount of video data over the network connection with the UE. Conversely, a conventional implementation of the generative model at the UE can lead to a likewise considerable amount of data in the form of input prompt information being inefficiently transmitted between a network component and the UE. Further, the implementation of a generative model at the UE in this manner results in a static implementation that may not be able to adapt to dynamic conditions associated with the UE, such as a change in available resources at the UE, a change in network conditions at the UE, and the like (referred to herein as a “change in context of the UE”).

[0018] Accordingly, in implementations, the network 100 employs a neural-network- based distributed generative modeling scheme in which the UE 110 operates to provide a local generative model while an application server or other network component operates to generate input prompt information to serve as input prompt(s) for use by the local generative model at the UE, and a jointly-trained set of neural networks, including at least an encoder neural network at the network and a decoder neural network at the UE, jointly operate to efficiently encode, transmit, and decode the input prompt information from the network component for use by the local generative model at the UE. To illustrate, as depicted in FIG. 1 , the UE 110 locally implements a generative model 112 that has been trained and otherwise configured to receive input prompt information representing one or more input prompts, and from this input prompt information generate an output that then is transferred to one or more user experience (UX) components 114 of the UE 110 (or associated therewith) for processing so as to facilitate a user experience. The generative model 112 can include any of a variety of generative models, or combinations thereof, including a generative adversarial network (GAN), a variational autoencoder, a Boltzmann machine, a large language model (LLM), a diffusion probabilistic model, a generative stochastic network (GSN), a variational autoencoder, a diffusion network, and the like. The one or more UX components 114 can include any of a variety of components that contribute to user experience for the UE 110, such as displays, speakers, haptic feedback generators, and the like. For example, the UE 110 may be a smartphone and the one or more UX components 114 may include a display and a speaker of the smartphone. As another example, the UE 110 may be an XRheadset and the one or more UX components 114 may include a near-eye AR display and / or integrated speaker. The one or more UX components 114 may be implemented at the UE 110 or connected to the UE 110 via a wired or wireless connection, such as a USB connection, a Bluetooth connection, or a WiFi Direct connection.

[0019] In some implementations, the application server 106 (or other network component) generates input prompt information 116, which is provided in some form to the UE 110 via wireless transmission from the BS 108 (or other edge network component). To facilitate efficient transfer of this input prompt information 116, the BS 108 and the UE 110 together implement a jointly-trained downlink pathway 118 in which the BS 108 (or other edge network component or core network component) implements a network encoder neural network 120 and the UE 110 implements a UE decoder neural network 122. The network encoder neural network 120 and the UE decoder neural network 122 may be implemented as any of a variety of suitable neural networks, such as DNNs or convolutional neural networks (CNNs), and are jointly trained (with or without the generative model 112) to facilitate transmission of the input prompt information 116 between the BS 108 and the UE 110. Thus, the network encoder neural network 120 processes the input prompt information 116 according to its trained configuration to generate an intermediate signal 124 that, in effect, is an encoded representation of the input prompt information 116. The BS 108 wirelessly transmits the intermediate signal 124 to the UE 110 as part of the cellular signaling between the BS 108 and the UE 110. At the UE 110, the UE decoder neural network 122 receives the intermediate signal 124 as an input and processes the intermediate signal 124 according to its trained configuration to generate a final signal 126 that, in effect, is a decoded or otherwise recovered representation of the input prompt information 116. As described in greater detail herein, in some instances an exact and full recovery of the original input prompt information 116 is not necessary for satisfactory operation of the generative model 112, and thus in such instances the intermediate signal 124 may be a lossless representation (that is, an inexact recovery of the input prompt information 116). This lossless representation may be a conventional lossless process, such as a reduction in resolution, frame rate, etc. In other implementations, this lossless representation may be achieved through, for example, a soft output or semantic representation. As the final signal 126 represents a partial or complete version of the input promptinformation 116 (or even a soft output representation or semantic meaning representation), the final signal 126 represents input prompt information having one or more input prompts for input to the generative model 112. The generative model 112 thus processes these one or more input prompts in accordance with its trained configuration to generate an output result 128 that is transferred to the one or more UX components 114 for further processing for facilitating a user experience at the UE 110.

[0020] To illustrate by way of one example, in implementations the distributed generative model scheme supports a videoconferencing use case in which the application server 106 generates input prompts that are intended to reflect, for example, facial expressions, posture changes, hand motions, and the like, and the generative model 112 at the UE 110 utilizes these input prompts along with current video of a corresponding participant (or an avatar) to generate a sequence of video images of the corresponding participant (or avatar) performing such movements / gestures. As another example, the application server 106 may include a gaming server supporting multiple UEs 110. The sensor information from one UE 110 (including, for example game controller input) is transmitted to the application server 106, which in turn generates input prompt information based on this sensor information to instigate the generative model 112 at a second UE 110 to generate gameplay and corresponding gaming video representative of the state of gameplay resulting from this sensor information as gaming input.

[0021] In addition to employing a neural-network-based downlink pathway 118, in some implementations the BS 108 and the UE 110 also together employ a jointly- trained uplink pathway 130 for facilitating efficient communication of information from the UE 110 to one or more components of the network, such as the application server 106. For the uplink pathway 130, the UE 110 implements a UE encoder neural network 132 and the BS 108 implements a network decoder neural network 134. The UE encoder neural network 132 and the network decoder neural network 134 may be implemented as any of a variety of suitable neural networks, such as DNNs or convolutional neural networks (CNNs), and are jointly trained to facilitate transmission of uplink information between the UE 110 and the BS 108. In embodiments, this uplink information can include feedback information for use by the application server 106 in controlling or adapting one or more subsequent iterations ofthe input prompt information 116 to the present context for the UE 1 10. This present context can include a situational context for the UE 110, such as the present pose or position of the UE 110, a present battery capacity, a present compute capability, a present network bandwidth capability or present network latency as observed by the UE 110, one or more present parameters of the one or more UX components 114 and / or of the one or more software applications that utilize the one or more UX components 114, and the like. Additionally or alternatively, this feedback information can include user input, such as user input provided via a game controller (for an instance in which the distributed generative approach is used for a remote gaming application) or user input providing feedback for correcting or improving the generative process.

[0022] In at least one embodiment, this situational context for the UE 110 is represented at least in part by sensor data 136 generated by a sensor set 144 of the UE 110. For example, the sensor set 144 can include a network interface that senses present network capability parameters, a battery sensor that measures present battery capacity, a software-based or hardware-based sensor that measures present compute capacity, such as remaining available memory or present processor utilization, a screen activation sensor to determine a present status of a display screen, thermal sensors to determine a present thermal status of one or more components of the UE 110, pose / position / orientation sensors such as an inertia management unit (IMU), gyroscope, global positioning system (GPS) sensor, a global satellite navigation system (GNSS) sensor, and / or cellular / wireless triangulation system to determine a present pose (position and / or orientation) of the UE 110, thermal sensors to detect thermal conditions of the UE 110, and the like.

[0023] An iteration of the sensor data 136 gathered from some or all of the sensors of the sensor set 144 can occur on a periodic basis (e.g., in accordance with a softwarebased timer), in response to an aperiodic trigger (e.g., in response to a UE Capability Enquiry Radio Resource Control (RRC) message from the BS 108), or a combination thereof. A present iteration of sensor data 136 generated by the sensor set 144 is provided as an input to the UE encoder neural network 132, which processes the present iteration of sensor data 136 according to its trained configuration to generate an intermediate signal 138 that is, in effect, an encoded representation of the sensor data 136. The UE 1 10 then transmits the intermediate signal 138 to the BS 108. Atthe BS 108, the intermediate signal 138 is provided as an input to the network decoder neural network 134, which processes the intermediate signal 138 according to its trained configuration to generate a final signal 140, which is a lossy or lossless representation of the present iteration of the sensor data 136, depending on implementation requirements and jointly-trained configuration of the neural networks 132 and 134 of the uplink pathway 130.

[0024] The final signal 140, representing the present iteration of sensor data 136, then may be transferred to the application server 106 or other component of the network for processing. In implementations, the application server 106 uses the present situational context of the UE 110 as represented by the recovered sensor data (in the final signal 140) to modify the next iteration of input prompt information 116 to be transmitted to the UE 110 via the downlink pathway 118. To illustrate, a software application executing at the UE 110 may be utilizing the generative model 112 to generate and render virtual reality (VR) content and the application server 106 may operate to generate input prompts used by the generative model 112 in generating this VR content. The sensor data 136 may include, for example, present pose information for the UE 110 as determined by one or more pose / positional sensors of the sensor set 144, and the application server 106 may use the present pose information in generating input prompts for inclusion in the next iteration of the input prompt information 116 that, when processed by the generative model 112, cause the generative model 112 to generate and render a sequence of video frames that accurately reflect a VR world from the perspective of a pose of the UE 110 in the VR world that corresponds to the present pose of the UE 110 in the real world.

[0025] In addition to, or instead of, providing sensor data, user input, and other feedback to the application server 106 for use in, for example, determining input prompts, in some implementations sensor data 142 from the sensor set 144 is provided as an input to the generative model 112 for processing along with the present iteration of the final signal 126 in generating the output result 128. The sensor data 142 thus may be the same as sensor data 136, or a different set of sensor data from a different subset of sensors from the sensor set 144 and / or sensor data captured at a different point in time. To illustrate using the example above in which the application server 106 uses the sensor data 136 to generate input prompts intended to direct the generative model 112 to generate VR scene video from acorresponding pose of the UE in the VR world, the sensor data 142 may include updated (i.e., more recent) pose information of the UE 110 which is processed by the generative model 112 to further refine the rendered pose of the UE in the VR world. As another example, the sensor data 142 may include a present graphics processing unit (GPU) capacity, which may be used by the generative model 112 to, for example, scale the resolution of the rendered VR video images accordingly.

[0026] In some implementations, the jointly-trained neural networks 120 and 122 of the downlink pathway 118 and / or the jointly-trained neural networks 132 and 134 of the uplink pathway 130 are “fixed”; that is, their particular architectural configurations (e.g., weights, connections, layers, etc.) do not change based on implementation or circumstances. However, in many instances the potential present context of the UE 110 may have a wide range. For example, one software application at the UE 110 may have one set of operational parameters with regard to the generative model 112, while another software application at the UE 110 may have a different set of operational parameters. Moreover, certain present context parameters, such as present processing capacity, present network channel conditions, and the like, may facilitate the use of certain architectural configurations for the neural networks 120, 122, 132, and 134 while making other architectural configurations less practicable. Accordingly, in other implementations, the architectural configurations for one or more of the neural networks 120, 122, 132, and 134, as well as for the generative model 112, may be selected for use based on the present context of the UE 110, the present context of the BS 110 or other network component, the present context of the application server 106, or a combination thereof. Thus, in some embodiments, to initiate this selection process, when a software application signals an intent to use the generative model 112, the UE 110 can transmit a request to the BS 108 and provide a representation of its present context to the BS 108 as, for example, a UE Capabilities Information RRC message or UE Assistive Information RRC message. The BS 108 (or other network component, such as a server in the core network) then may use the information from the request and the present context information for the UE 110 to select a suitable set of jointly-trained neural network architectures for the downlink pathway 118 and / or the uplink pathway 130 from a plurality of candidate sets of jointly-trained neural networks, and instruct the UE 110 to implement the corresponding neural architectures for the neural networks 122, 132, and for the generative model 112. Likewise, the BS 108 implements the corresponding neuralnetwork architectures for the neural networks 120 and 134. The BS 108 can signal the neural network architectures to be used by the UE 110 by transmitting data detailing the neural network architectures to be implemented (e.g., the weights, nodes, layers, etc.) or by sending index values or other identifiers used by the UE 110 to reference the selected neural network architectures from a repository of candidate neural network architectures accessible by the UE 110, as described in greater detail below.

[0027] FIGs. 2 and 3 illustrate example hardware configuration for the UE 110 and BS 108, respectively, in accordance with some embodiments. Note that the depicted hardware configurations represent the processing components and communication components most directly related to the distributed generative model scheme described herein and omit certain components well-understood to be frequently implemented in such electronic devices, such as displays, non-sensor peripherals, power supplies, and the like. Moreover, while a hardware configuration for the BS 108 is illustrated with reference to FIG. 3, it will be appreciated that a similar hardware figuration may be employed for another edge network component when another edge network component employs some or all of the functionality ascribed herein to the BS 108 with respect to the distributed generative modeling scheme.

[0028] Referring first to the hardware configuration of the UE 110 of FIG. 2, in implementations the UE 110 includes one or more antenna arrays 202, with each antenna array 202 having one or more antennas 203, and further includes an RF front end 204, one or more processors 206, one or more non-transitory computer- readable media 208, as well as the one or more UX components 114 and sensor set 144 described above. The RF front end 204 operates, in effect, as a physical (PHY) transceiver interface to conduct and process signaling between the one or more processors 206 and the antenna array 202 so as to facilitate various types of wireless communication. The antennas 203 can include an array of multiple antennas that are configured similar to or different from each other and can be tuned to one or more frequency bands associated with a corresponding RAT, such as a cellular RAT (e.g., a Third Generation Partnership Project (3GPP) Fourth Generation Long Term Evolution (4G LTE) RAT or a 3GPP Fifth Generation New Radio (5G NR) RAT), a WLAN RAT (e.g., an Institute of Electronic and Electrical Engineers (IEEE) 802.11- based RAT), a WPAN RAT (e.g., a Bluetooth (TM) RAT), and the like.

[0029] The one or more processors 206 can include, for example, one or more central processing units (CPUs), graphics processing units (GPUs), artificial intelligence (Al) accelerators or other application-specific integrated circuits (ASIC), and the like. To illustrate, the processors 206 can include an application processor (AP) utilized by the UE 110 to execute an operating system and various user-level software applications, as well as one or more processors utilized by modems or a baseband processor of the RF front end 204. The computer-readable media 208 can include any of a variety of media used by electronic devices to store data and / or executable instructions, such as random-access memory (RAM), read-only memory (ROM), caches, Flash memory, solid-state drive (SSD) or other mass-storage devices, and the like. For ease of illustration and brevity, the computer-readable media 208 is referred to herein as “memory 208” in view of frequent use of system memory or other memory to store data and instructions for execution by the processor 206, but it will be understood that reference to “memory 208” shall apply equally to other types of storage media unless otherwise noted.

[0030] As noted above, the UE 110 further may include a plurality of sensors, referred to herein as the sensor set 144, from which sensor data is obtained for one or both of transmission to the application server 106 or use by the generative model 112. Generally, the sensors of sensor set 144 capture sensor data representative of a present context of the UE 110, which may include positional / pose context, resource availability context, resource usage context, or resource capacity context, network channel status (e.g., bandwidth and / or latency), and the like. Accordingly, the sensor set 144 can include, for example, GPS sensors, GNSS sensors, IMU sensors, gyroscopes, tilt sensors or other inclinometers, ultrawideband (UWB)-based sensors. The sensor set 144 also may include imaging sensors, such as cameras for image capture by a user, cameras for facial detection, cameras for stereoscopy or visual odometry, light sensors for detection of objects in proximity to a feature of the UE 110, and the like. The sensor set 144 further can include user interface (Ul) sensors, such as touch screens, user-manipulable input / output devices (e.g., “buttons” or keyboards), or other touch / contact sensors, microphones or other voice sensors, thermal sensors (such as for detecting proximity to a user), and the like. Another example of sensors of the sensor set 144 can include thermal sensors for determining a thermal status of one or more components of the UE 110, battery capacity sensors for determining a present battery capacity of the UE 110, a networkinterface for determining a status of a network channel connecting the UE 110 to the BS 108, memory monitors to determine present memory availability, processor monitors to determine present processor capacity / utilization, and the like.

[0031] The UX components 114 include components of the UE 110 for providing an aspect of user experience to a user of the UE 110. This may include hardware components, such as displays, speakers, haptic feedback devices, touchscreens, and the like, as well as software components, such as the drivers for such hardware devices or the software applications that utilize the output of the generative model 112 to control the hardware UX components.

[0032] The one or more memories 208 of the UE 110 are used to store one or more sets of executable software instructions and associated data that manipulate the one or more processors 206 and other components of the UE 110 to perform the various functions described herein and attributed to the UE 110. The sets of executable software instructions include, for example, an operating system (OS) and various drivers (not shown), various software applications 210 (including at least one user application that utilizes the output of the generative model 112), a UE neural network manager 212 that implements one or more neural networks for the UE 110, such as the neural networks 122 and 132 (FIG. 1), and a generative model manager 214 that implements the generative model 112 (FIG. 1 ) of the UE 110, and which may be the same as, or different from, the UE neural network manager 212. The data stored in the one or more memories 208 includes, for example, a set of one or more candidate neural network architecture configurations 216 for implementation at the neural networks 122 and 132 and a set of one or more candidate generative model architectural configurations 218 for implementation at the generative model 112.

[0033] The one or more candidate neural network architecture configurations 216 include one or more data structures containing data and other information representative of a corresponding architecture and / or parameter configurations used by the UE neural network manager 212 to form a corresponding neural network of the UE 110, such as at one of the UE decoder neural network 122 or the UE encoder neural network 132. The information included in a neural network architectural configuration 216 includes, for example, parameters that specify a fully connected layer neural network architecture, a convolutional layer neural network architecture, a recurrent neural network layer, a number of connected hidden neural network layers,an input layer architecture, an output layer architecture, a number of nodes utilized by the neural network, coefficients (e.g., weights and biases) utilized by the neural network, kernel parameters, a number of filters utilized by the neural network, strides / pooling configurations utilized by the neural network, an activation function of each neural network layer, interconnections between neural network layers, neural network layers to skip, and so forth. Accordingly, the neural network architecture configuration 216 includes any combination of neural network formation configuration elements (e.g., architecture and / or parameter configurations) that can be used to create a neural network formation configuration (e.g., a combination of one or more neural network formation configuration elements) that defines and / or forms a DNN. The one or more candidate generative model architecture configurations 218 likewise include one or more data structures containing data and other information representative of a corresponding architecture and / or parameter configurations used by the generative model manager 214 to form a corresponding generative model 112 of the UE 110.

[0034] Turning to FIG. 3, an example hardware configuration of the BS 108 is shown in accordance with embodiments. However, it is noted that although the illustrated diagram represents an implementation of the BS 108 as a single network node (e.g., a 5G NR Node B, a WiFi access point, Bluetooth parent device, etc.), the functionality, and thus the hardware components, of the BS 108 instead may be distributed across multiple network nodes or devices and may be distributed in a manner to perform the functions described herein. As with the UE 110, the BS 108 includes at least one array 302 of one or more antennas 303, an RF front end 304, one or more processors 306 and one or more non-transitory computer-readable storage media 308 (as with the memory 208 of the UE 110, the computer-readable medium 308 is referred to herein as a “memory 308” for brevity). These components operate in a similar manner as described above with reference to corresponding components of the UE 110.

[0035] The one or more memories 308 of the BS 108 store one or more sets of executable software instructions and associated data that manipulate the one or more processors 206 and other components of the BS 108 to perform the various functions described herein and attributed to the BS 108. The sets of executable software instructions include, for example, an OS and various drivers (not shown),various software applications (not shown), a BS manager 310, and a BS neural network manager 312. The BS manager 310 configures the RF front end 204 for communication with the UE 110, as well as communication with a core network, such as the core network 102, and the application server 106 via a core network interface 314. The BS neural network manager 312 implements one or more neural networks for the BS 108, such as the neural networks 120 and 134 of the downlink pathway 118 and uplink pathway 130, respectively (FIG. 1).

[0036] The data stored in the one or more memories 308 of the BS 108 includes one or more neural network architecture configurations 316 that represent one or more data structures containing data and other information representative of a corresponding architecture and / or parameter configurations used by the BS neural network manager 312 to form a corresponding neural network of the BS 108. Similar to the neural network architectural configuration 216 of the UE 110, the information included in a neural network architectural configuration 316 includes, for example, parameters that specify a fully connected layer neural network architecture, a convolutional layer neural network architecture, a recurrent neural network layer, a number of connected hidden neural network layers, an input layer architecture, an output layer architecture, a number of nodes utilized by the neural network, coefficients utilized by the neural network, kernel parameters, a number of filters utilized by the neural network, strides / pooling configurations utilized by the neural network, an activation function of each neural network layer, interconnections between neural network layers, neural network layers to skip, and so forth. Accordingly, the neural network architecture configuration 316 includes any combination of neural network formation configuration elements that can be used to create a neural network formation configuration that defines and / or forms a DNN or other neural network.

[0037] FIG. 4 illustrates an example machine learning (ML) module 400 for implementing a neural network in accordance with some embodiments. As described herein, one or both of the BS 108 and the UE 110 implement one or more neural networks in the downlink pathway 118 and the uplink pathway 130 and the UE 110 further implements the generative model 112. The ML module 400 therefore illustrates an example module for implementing one or more of these neural networks. For example, the ML module 400 may represent a DNN implemented asone of the neural networks 120, 122, 132, or 134 of FIG. 1 . The ML module 400 may also represent one of the DNNs or other neural networks implemented as part of the generative model 112. For example, a GAN utilizes a pair of unstable DNNs: a generator and a discriminator. As such, in implementations in which the generative model 112 is a GAN, one instance of the ML module 400 may be used to implement the generator DNN and another instance of the ML module may be used to implement the discriminator DNN.

[0038] In the depicted example, the ML module 400 implements at least one deep neural network (DNN) 402 with groups of connected nodes (e.g., neurons and / or perceptrons) that are organized into three or more layers. The nodes between layers are configurable in a variety of ways, such as a partially-connected configuration where a first subset of nodes in a first layer are connected with a second subset of nodes in a second layer, a fully-connected configuration where each node in a first layer is connected to each node in a second layer, etc. A neuron processes input data to produce a continuous output value, such as any real number between 0 and 1 . In some cases, the output value indicates how close the input data is to a desired category. A perceptron performs linear classifications on the input data, such as a binary classification. The nodes, whether neurons or perceptrons, can use a variety of algorithms to generate output information based upon adaptive learning. Using the DNN 402, the ML module 400 performs a variety of different types of analysis, including single linear regression, multiple linear regression, logistic regression, step- wise regression, binary classification, multiclass classification, multivariate adaptive regression splines, locally estimated scatterplot smoothing, and so forth.

[0039] In some implementations, the ML module 400 adaptively learns based on supervised learning. In supervised learning, the ML module 400 receives various types of input data as training data. The ML module 400 processes the training data to learn how to map the input to a desired output. During a training procedure, the ML module 400 uses labeled or known data as an input to the DNN 402. The DNN 402 analyzes the input using the nodes and generates a corresponding output. The ML module 400 compares the corresponding output to truth data and adapts the algorithms implemented by the nodes to improve the accuracy of the output data. Afterward, the DNN 402 applies the adapted algorithms to unlabeled input data to generate corresponding output data. The ML module 400 uses one or both ofstatistical analyses and adaptive learning to map an input to an output. For instance, the ML module 400 uses characteristics learned from training data to correlate an unknown input to an output that is statistically likely within a threshold range or value. This allows the ML module 400 to receive complex input and identify a corresponding output. Some implementations train the ML module 400 on characteristics of communications transmitted over a wireless communication system (e.g., time / frequency interleaving, time / frequency deinterleaving, convolutional encoding, convolutional decoding, power levels, channel equalization, inter-symbol interference, quadrature amplitude modulation / demodulation, frequency-division multiplexing / de- multiplexing, transmission channel characteristics). This allows the trained ML module 400 to receive samples of a signal as an input, such as samples of a downlink signal received at a UE, and recover information from the downlink signal, such as the binary data embedded in the downlink signal.

[0040] In the depicted example, the DNN 402 includes an input layer 404, an output layer 406, and one or more hidden layers 408 positioned between the input layer 404 and the output layer 406. Each layer has an arbitrary number of nodes, where the number of nodes between layers can be the same or different. That is, the input layer 404 can have the same number and / or a different number of nodes as output layer 406, the output layer 406 can have the same number and / or a different number of nodes than the one or more hidden layer 408, and so forth.

[0041] Node 410 corresponds to one of several nodes included in input layer 404, wherein the nodes perform separate, independent computations. As further described, a node receives input data and processes the input data using one or more algorithms to produce output data. Typically, the algorithms include weights and / or coefficients that change based on adaptive learning. Thus, the weights and / or coefficients reflect information learned by the neural network. Each node can, in some cases, determine whether to pass the processed input data to one or more next nodes. To illustrate, after processing input data, node 410 can determine whether to pass the processed input data to one or both of node 412 and node 414 of hidden layer 408. Alternatively or additionally, node 410 passes the processed input data to nodes based upon a layer connection architecture. This process can repeat throughout multiple layers until the DNN 402 generates an output using the nodes (e.g., node 416) of output layer 406.

[0042] A neural network can also employ a variety of architectures that determine what nodes within the neural network are connected, how data is advanced and / or retained in the neural network, what weights and coefficients are used to process the input data, how the data is processed, and so forth. These various factors collectively describe a neural network architecture configuration, such as the neural network architecture configurations 216, 218, and 316 briefly described above. To illustrate, a recurrent neural network, such as a long short-term memory (LSTM) neural network, forms cycles between node connections to retain information from a previous portion of an input data sequence. The recurrent neural network then uses the retained information for a subsequent portion of the input data sequence. As another example, a feed-forward neural network passes information to forward connections without forming cycles to retain information. While described in the context of node connections, it is to be appreciated that a neural network architecture configuration can include a variety of parameter configurations that influence how the DNN 402 or other neural network processes input data.

[0043] A neural network architecture configuration of a neural network can be characterized by various architecture and / or parameter configurations. To illustrate, consider an example in which the DNN 402 implements a convolutional neural network (CNN). Generally, a convolutional neural network corresponds to a type of DNN in which the layers process data using convolutional operations to filter the input data. Accordingly, the CNN architecture configuration can be characterized by, for example, pooling parameter(s), kernel parameter(s), weights, and / or layer parameter(s).

[0044] A pooling parameter corresponds to a parameter that specifies pooling layers within the convolutional neural network that reduce the dimensions of the input data. To illustrate, a pooling layer can combine the output of nodes at a first layer into a node input at a second layer. Alternatively or additionally, the pooling parameter specifies how and where in the layers of data processing the neural network pools data. A pooling parameter that indicates “max pooling,” for instance, configures the neural network to pool by selecting a maximum value from the grouping of data generated by the nodes of a first layer, and uses the maximum value as the input into the single node of a second layer. A pooling parameter that indicates “average pooling” configures the neural network to generate an average value from thegrouping of data generated by the nodes of the first layer and use the average value as the input to the single node of the second layer.

[0045] A kernel parameter indicates a filter size (e.g., a width and a height) to use in processing input data. Alternatively or additionally, the kernel parameter specifies a type of kernel method used in filtering and processing the input data. A support vector machine, for instance, corresponds to a kernel method that uses regression analysis to identify and / or classify data. Other types of kernel methods include Gaussian processes, canonical correlation analysis, spectral clustering methods, and so forth. Accordingly, the kernel parameter can indicate a filter size and / or a type of kernel method to apply in the neural network.

[0046] Weight parameters specify weights and biases used by the algorithms within the nodes to classify input data. In some implementations, the weights and biases are learned parameter configurations, such as parameter configurations generated from training data.

[0047] A layer parameter specifies layer connections and / or layer types, such as a fully-connected layer type that indicates to connect every node in a first layer (e.g., output layer 406) to every node in a second layer (e.g., hidden layer 408), a partially- connected layer type that indicates which nodes in the first layer to disconnect from the second layer, an activation layer type that indicates which filters and / or layers to activate within the neural network, and so forth. Alternatively or additionally, the layer parameter specifies types of node layers, such as a normalization layer type, a convolutional layer type, a pooling layer type, and the like.

[0048] While described in the context of pooling parameters, kernel parameters, weight parameters, and layer parameters, it will be appreciated that other parameter configurations can be used to form a DNN consistent with the guidelines provided herein. Accordingly, a neural network architecture configuration can include any suitable type of configuration parameter that can be applied to a DNN that influences how the DNN processes input data to generate output data.

[0049] In some embodiments, the configuration of the ML module 400 is based on a present situational context of the UE 110, the BS 108, or both, as well as requirements or other parameters of one or more software applications 210 of the UEthat utilize the output of the generative model 112. To illustrate, consider two jointly- trained instances of the ML module 400; one trained to efficiently compress or otherwise encode input prompt information for wireless transmission and the other trained to efficiently decompress or otherwise decode encoded input prompt information from the first instance for input to the generative model 112. The manner in which the software application 210 is to utilize the output of the generative model 112 generated from this input, as well as the quality-of-service (QoS) requirements or quality-of-experience (QoE) requirements of the software application 210 (such as required minimum bandwidth, required maximum latency, required maximum error rate) may inform the training of the two paired instances of the ML module 400. For example, when the output of the generative model 112 is audio output and the software application 210 has relatively low QoS requirements, the paired ML modules 400 may be trained so as to accept a lower throughput or a higher error rate or lower audio resolution in favor of reduced processing resource requirements at one or both of the UE 110 or the BS 108. Conversely, when the output of the generative model 112 is high-resolution video and the software application 210 has relatively high QoE requirements, the paired ML modules 400 may be trained so to a higher throughput and accuracy standard at the expense of higher processor resource utilization.

[0050] Accordingly, in some embodiments, the device implementing the ML module 400 generates and stores different neural network architecture configurations for different potential situational contexts, such as different combinations of network conditions, available resource configurations, QoS / QoE requirements, software application types, and the like. To this end, paired instances of the ML module 400 can be trained for each combination of interest, and the training may occur offline when no active communication exchanges are occurring, or online during active communication exchanges. The training of such paired instances of the ML module 400 is described in great detail below with reference to FIG. 8.

[0051] FIG. 5 illustrates a signal diagram 500 illustrating an example operation of the network 100 for implementing a distributed generative modeling scheme in accordance with some embodiments. Operation of the distributed generative modeling scheme typically is initiated with a local application trigger 502 at the UE 110, which can include, for example, the start-up or other initiation of a software application 210 (FIG. 2) that will use the output of the generative model 112,execution of a particular subroutine, library, call function, application programming interface (API), or the like at the UE 110, and so forth. In response, at action 504 the UE neural network manager 212 determines a present context of the UE 110 as it pertains to configuring the downlink pathway 118, the uplink pathway 130, and / or the generative model 112. This present context can be represented at least in part based on a present iteration of the sensor data 136 obtained from the sensor set 144, and which may represent, for example, present network channel conditions, such as available bandwidth or observed latency, UE-observed signal-to-noise ratio (SNR), signal-to-interference plus noise ratio (SINR), reference signal received power (RSRP), reference signal received quality (RSRQ), and the like. The sensor data 136 further may represent a present context of the operating state of the UE 110 itself, such as available processing resources or processing resource utilization, available memory resources or memory resource utilization, present battery capacity, as well. The sensor data 136 also may include data representing the relationship between the UE 110 and its physical environment, including pose data, position data, radar data, lidar data, and the like. Further, the present context can include an identification of the type of software application 210 that instigated the local application trigger 502 (e.g., video, audio, VR, XR, gaming, telepresence, etc.), as well as the requirements of the software application 210 with respect to the output to be generated by the generative model 112, such as QoS requirements or QoE requirements.

[0052] The UE 110 then wirelessly transmits a request 506 for instantiation of a generative model to the BS 108, with this request 506 including or being associated with data representing the determined present context of the UE 110. In one embodiment, one or both of the request 506 or the data representing the present context of the UE 110 are transmitted to the BS 108 as part of a UE Capability Information RRC message or a UE Assistance Information RRC message. For example, in one embodiment, the UE 110 transmits the request 506 to the BS 108 and, in response, the BS 108 replies with a UE Capability Enquiry RRC message to the UE 110. In response to the UE Capability Enquiry RRC message, the UE 110 determines the data representative of the present context and transmits this data to the BS 108 as a UE Capability Information RRC message. In other embodiments, the request 506 and the data representing the present context of the UE are transmitted together as a UE Capability Information RRC message. At action 508,the BS 108 in turn may determine its own present context, such as BS-observed network channel conditions, such as BS-observed bandwidth, latency, error rate, SNR, SI NR, RSRP, and / or RSRQ. The BS present context further can include present resource availability or utilization at the BS 108, and the like.

[0053] As described in greater detail below with reference to FIG. 7, in implementations various combinations of parameter values for parameters of a UE context and / or a BS context may be identified and then a corresponding tuple of neural network architecture configurations for each of the generative model 112, the network encoder neural network 120, and the UE decoder neural network 122 are jointly trained for each identified combination of parameter values to generate a corresponding candidate neural network pathway. For example, for a set of parameters for a UE context in which a particular type of generative model is specified (e.g., a type of output generated by the generative model), a QoE expectation for the output is specified, a particular set of network channel conditions are specified, and a particular set of UE resource availabilities is specified, a corresponding candidate network encoder neural network architecture configuration, a candidate UE decoder neural network architecture configuration, and a candidate generative model neural network architecture configuration tuple can be jointly trained for the downlink pathway 118 and generative model 112 under simulated conditions that meet these specified parameters, so as to generate one candidate neural network pathway for the downlink pathway 118 and the generative model 112. For a different combination in which a different type of generative model is specified, different QoE expectation is specified, a different set of network channel conditions are specified, and / or a different set of UE resource availabilities is specified, a different candidate network encoder neural network architecture configuration, candidate UE decoder neural network architecture configuration, and candidate generative model neural network architecture configuration tuple can be jointly trained for this different set of specified parameters to generate another candidate neural network pathway, and so forth. This same process may be applied for determining one or more jointly-trained candidate sets of neural network architecture configurations for the uplink pathway 130. Thus, as a result of different neural network training iterations for different combinations of context parameters (for the UE 110, the BS 108, and or the network channel connecting the two), the network 100 may generate sets of tuples of candidate jointly-trained neural networkarchitectural configurations for the downlink pathway 118 and the generative model 112 and / or the uplink pathway 130.

[0054] The context parameters selected for use in joint training of a corresponding candidate set of neural network architectures can include any variety and combination of context parameters. For example, the selected context parameters can include hardware capabilities, indicating certain processing, memory, and / or storage capabilities, such as maximum instruction throughput, maximum memory access speed, current instruction throughput, current memory access speed, current utilization of a particular hardware resource, and the like. The selected context parameters also can include one or more QoS or QoE requirements, such as QoS requirements pertaining to jitter, throughput, latency, data loss, and the like. Power and / or thermal parameters likewise may be considered. This may include, for example, whether the LIE 110 is connected to a non-battery power source or to a battery power source, and if the latter, the amount of remaining battery power. Another example may be some parameter pertaining to the thermal state of the UE 110 overall or for one or more components of the UE 110, such as a current skin temperature and its relationship to a specified threshold, or a thermal state of one or more processing components of the UE 110 in view of one or more corresponding thresholds. The selected context parameters also may include parameters pertaining to the software application(s) 210 that will consume the content generated by the generative model 112 and / or the type of content to be generated, such as the manner in which the software application will consume the content (e.g., audio output, video output, still image output, text output, etc.), the prioritization of the software application 210, and the like. Parameters pertaining to the present conditions of the network channel(s) connecting the UE 110 and the BS 108, such as the aforementioned observed bandwidth, latency, error rate, SNR, SINR, RSRP, and / or RSRQ, may be selected. Still further, parameters pertaining to the relationship of the UE 110 to the physical environment, such as location and / or pose, the presence or absence of interferer objects, and the like, may be utilized as training parameters.

[0055] The candidate neural network architectural configurations determined through such joint training may be made available for implementation at the BS 108 and the UE 110 in any of a variety of ways. In some embodiments, the network 100 mayutilize a centralized repository of these candidate neural network architectural configurations, such as at a component of the core network 102 or at the application server 106, and the BS 108 or UE 110 may access an identified neural network architectural configuration for implementation as a corresponding neural network by requesting the identified neural network architectural configuration using a corresponding index or other identifier or by being supplied the identified neural network architectural configuration as a result of selection of the identified neural network architectural configuration based on analysis of one or both of the present context of the UE 110 or the present context of the BS 108. In other implementations, a local cache of some or all of the trained sets of neural network architectural configurations may be stored at each of the BS 108 and the UE 110, and the BS 108 and UE 110 each may access a corresponding neural network architectural configuration for implementation from the corresponding local cache based on an index or other identifier associated with the corresponding neural network architectural configuration.

[0056] In some embodiments, the application server 106 operates to select the neural network architectural configurations to be implemented for the downlink pathway 118, the uplink pathway 130, and / or the generative model 112. In such situations, at action 510 the BS 108 forwards the request 506 to the application server 106, along with the data representing the present context of the UE 110 and / or the present context of the BS 108. In other embodiments, the BS 108 (or other network edge component) operates to select the neural network architectural configurations for the downlink pathway 118, the uplink pathway 130, and / or the generative model 112. In either approach, at action 512 either the BS 108 or the application server 106 uses the present context of the UE 110 and / or the present context of the BS 108 to select suitable jointly-trained pairs of encoder / decoder neural network architecture configurations for implementation from the corresponding sets of candidate jointly- trained network architecture configurations, as well as to select a suitable neural network architecture configuration from a corresponding set of candidate neural network architecture configurations for implementation as the generative model 112. The BS 108 / application server 106 can make this selection algorithmically using the present contexts of the UE 110 and / or BS 108 as inputs, via a look-up table (LUT) or other similar selection structure using these same inputs, or the selection itself mayutilize a trained neural network that takes the present context(s) of the UE 110 and / or BS 108 as inputs.

[0057] The selection of a particular candidate neural network pathway determines the neural network architectural configurations for the network encoder neural network 120 and the UE decoder neural network 122 of the downlink pathway 118 and the generative model 112, and if the uplink pathway 130 is implemented, for the UE encoder neural network 132 and the network decoder neural network 134 as well. Accordingly, at action 514 the component selecting the candidate neural network pathway sends an indication of the selected neural network architecture configurations to the BS 108 (if the BS 108 is not the component doing the selection) and to the UE 110, with the BS 108 receiving an indication of the neural network architecture configurations to implement for the neural networks 120 and 134 and the UE 110 receiving an indication of the neural network architecture configurations to implement for the neural networks 112, 122, and 132. As noted above, these indications may include identifiers used to index into a local or global repository of such neural network architecture configurations, the actual weights, connections, and other parameters of the neural networks themselves, or a combination thereof. In response to receiving the indication (or in response to making the selection from the candidate neural network pathways), at action 516 the BS 108 implements the indicated neural network architectural configuration at the network encoder neural network 120 and, if the uplink pathway 130 is in use, at the network decoder neural network 134 using corresponding instances of, for example, the ML module 400 of FIG. 4. Likewise, at action 518 the U 110 implements the indicated neural network architectural configurations at the UE decoder neural network 122 and the generative model 112 and, if the uplink pathway 130 is in use, at the UE encoder neural network 132 using corresponding instances of the ML module 400.

[0058] With the generative model 112 and pathways 118 and 130 configured and initialized, the distributed generative modeling scheme is ready for providing generated content at the UE 110. Accordingly, at action 520 the application server 106 generates an initial set of one or more input prompts based on various factors, such as application type of software applications 210, the type of content to be generated, content generation parameters, and the like, and provides the initial set of one or more input prompts as an initial instance of the input prompt information 116for transmission to the BS 108. For example, if the distributed generative modeling scheme is implemented for generating AR graphical content for display at the UE 110, the one or more input prompts may include descriptors of the AR graphical content to be generated. At action 522, the network encoder neural network 120 of the BS 108 receives the input prompt information 116 as an input and processes this input according to its trained architectural configuration to generate an initial instance of the intermediate signal 124, which, in effect, is an encoded representation of the input prompt information 116. At action 524, the BS 108 transmits the intermediate signal 124 for receipt by the UE 110.

[0059] At action 526, the UE decoder neural network 122 receives the intermediate signal 124 as an input and processes this input according to its trained architectural configuration to generate a corresponding instance of the final signal 126. Concurrently, at action 528 the sensor set 144 generates an instance of the sensor data 142. Due at least in part to the nature of the joint training of the encoder neural network 120, the decoder neural network 122, and the generative model 112, and to the generative nature of the generative model 112, in implementations the final signal 126 need not be a lossless reconstruction of the input prompt information 116 or of the intermediate signal 124. Accordingly, when the UE decoder neural network 122 processes the intermediate signal 124 as an input to generate the final signal 126, in some implementations certain processes utilized to confirm 100% recovery accuracy may be bypassed, such as by bypassing or disregarding a cyclic redundancy check (CRC) process, a hybrid automatic repeat request (HARQ) process, a radio link control (RLC) acknowledgement (ACK) process, and the like. Moreover, in addition to, or rather than, being a hard output, the UE decoder neural network 122 can be trained to provide the final signal 126 as a soft output, such as a certain probability distribution as a function of a softmax output. As another example, the UE decoder neural network 122 can be trained to provide the final signal 126 as a semantic representation or meaning of the intermediate signal 124. For example, for an instance in which the intermediate signal 124 is, in effect, an encoded representation of a voice packet, rather than being trained to recover the voice packet in its exact original form, the UE decoder neural network 122 instead could use previous voice packets, a previous voice scenario or context and the encoded representation of the voice signal to generate the desired voice sequence, even with some error as a result.

[0060] At action 530, the generative model 112 receives the final signal 126 and the sensor data 142 as inputs and processes these inputs according to its architectural configuration to generate an instance of the output result 128, which represents generated UX content based on the one or more input prompts transmitted in some form via the downlink pathway 118 and based on the sensor data 142 from the sensor set 144, which provides an indication of a current context of the UE 110 (e.g., relative to the surrounding physical environment). This generated output result 128 is then transferred at action 532 to the one or more UX components 114 for processing so as to facilitate a user experience at the UE 110, such as via display output, audio output, or the like, depending on the type of content generated. In some embodiments, at least one of the one or more UX components 114 is wirelessly connected to the UE 110, such as via a Bluetooth (TM) wireless connection or a WiFi Direct (TM) wireless connection.

[0061] Referring again to action 528, as explained above, sensor data generated by the sensor set 144 local to the UE 110 may be utilized by the generative model 112 of the UE 110 in generating output content. Moreover, the same sensor data or a different set of sensor data from the sensor set 144 likewise may be utilized by the application server 106 in generating the next instance of input prompt information 116. FIG. 6 depicts a signal diagram 600 that illustrates an example of this use of local sensor data as feedback for configuring a following instance of input prompt information 116 in accordance with implementations. The following description assumes an implementation in which the sensor data fed back to the application server 106 is the same sensor data used by the generative model 112 (that is, sensor data 136 and sensor data 142 are the same), but this same approach may be employed using a different set of sensor data from the same or different set of sensors using the guidelines provided below.

[0062] With the obtainment of the present instance of the sensor data 142 from the sensor set 144 at action 528 (FIG. 5), the UE 110 also can feed this sensor data back to the application server 106 via the uplink pathway 130. Accordingly, at action 602 the sensor data is input to the UE encoder neural network 132, which processes the sensor data according to its trained architecture configuration to generate an instance of the intermediate signal 138, which, in effect, is an encoded representation of the input sensor data. At action 604 the UE 110 wirelessly transmits the intermediatesignal 138 to the BS 108. At action 606, the intermediate signal 138 is provided as an input to the network decoder neural network 134, which processes this input according to its trained architectural configuration to generate a corresponding instance of the final signal 140, which, in effect, represents a decoded representation of the effectively encoded representation of the sensor data obtained at the UE 110. At action 608, the final signal 140 is transmitted from the BS 108 to the application server, and at action 610 the application server 106 processes the recovered sensor data represented by the final signal 140 as part of the process of generating a next instance of the input prompt information 116 representing one or more input prompts.

[0063] To illustrate, the distributed generative modeling scheme of the network 100 may be employed to support a videoconferencing application (one example of software application 210), and the sensor data from the sensor set 144 may include data from a radar sensor, lidar sensor, or other depth sensor that identifies the position and dimensions of objects in the immediate area surrounding the UE 110, and the application server 106 may utilize this information to construct input prompts intended to trigger generation of AR overlays for these objects. As another example, the sensor data may include pose information from a pose sensor for the UE 110, which in turn is used by the application server 106 to construct input prompts that trigger generation of video imagery that corresponds to the present perspective view of the UE 110. The generated input prompts then may be used to generate, transmit, and process a corresponding instance of the input prompt information 116 using the distributed generative modeling scheme provided by the jointly-trained downlink pathway 118 and the generative model 112 in another iteration of actions 520 through 532 of the signal diagram 500 of FIG. 5.

[0064] As explained above, in some embodiments the distributed generative modeling scheme of the network 100 utilizes a select one of a plurality of candidate neural network pathways for a jointly-trained downlink pathway 118 and generative model 112 for the distributed generative modeling scheme used to efficiently distribute the effort of generative modeling between the application server 106 and the UE 110. In this implementation, each candidate neural network pathway includes a corresponding neural network architecture configuration for each of the network encoder neural network 120, the UE decoder neural network 122, and the generative model 112, the combination of which has been jointly trained in view of acorresponding set of context parameters representing specific context parameters for the UE 110 and / or the BS 108, such as power parameters, resource availability parameters, thermal parameters, software type parameters, QoS / QoE parameters, and the like.

[0065] FIG. 7 illustrates an example method 700 for such joint training of candidate neural network architectural configurations for the downlink pathway 118 and the generative model 112 in accordance with implementations. Although described with reference to downlink pathway 118, the same or similar process may be employed for joint training of the neural networks 132 and 134 of the uplink pathway 130 using the guidelines provided herein. An iteration of method 700 starts with a training system identifying the context parameters to be associated with the corresponding candidate neural network pathway to be jointly trained for this iteration at block 702. To illustrate, for this iteration, the corresponding parameter set could include: a certain range of processing resource utilization at the UE 110 combined with a certain QoS / QoE specification, a certain battery remaining range for the UE 110, a certain thermal range for the UE 110, a range of network channel conditions observed by the UE 110, a range of network channel conditions observed by the BS 108, and a range of UEs presently being served by the BS 108. At block 704, the training system initializes training instances of each of the network encoder neural network 120, the UE decoder neural network 122, and the generative model 112 based on the context parameters identified at block 702.

[0066] At block 706, the training system obtains a batch of training data sets that reflect the corresponding context parameter set, such as a batch of training data sets including a plurality of training data instances, each training data instance including known input prompt information and corresponding known and suitable resulting generated content. At block 708, the training system processes each training data set in the chain of encoder neural network 120, decoder neural network 122, and generative model 112. This can include inputting the known input prompt information from the training data set into the training instance of the encoder neural network 120, transmitting the resulting output over a network channel that has constraints representing the channel characteristic parameters identified at block 702 (or simulating the transmission of the resulting output over an emulated version of such a network channel having similar constraints), and processing the received result at thetraining instance of the decoder neural network 122 at an actual or simulated UE having constraints consistent with the UE context parameters as those identified at block 702 (e.g., same available processing resources as specified, same remaining battery level as specified, etc.). The resulting output is then processed at the training instance of the generative model 112 at an actual or simulated UE having the same identified UE context parameters, which results in a generated output content.

[0067] After all training data sets of the batch have been processed in this matter, at block 710 the training system determines the performance of the candidate neural network pathway in processing the batch of training data sets. For example, this can include calculating a joint loss function using performance metrics associated with one or more of the generated output content for each training data set (e.g., how closely does the generated output content reflect the expected output content), performance metrics for the real or simulated operation of each of the neural networks (e.g., how closely does the operation meet a specified QoS / QoE goal, maintain an acceptable battery drain rate, utilize processing resources within an acceptable range), etc. At block 712, the training system determines whether this analyzed performance indicates that the candidate neural network pathway has been sufficiently trained. For example, whether the one or more joint loss functions indicate errors within corresponding acceptable ranges. If the training system decides that the candidate neural network pathway has been sufficiently trained, at block 714 the resulting trained architecture configuration of each neural network in the neural network pathway is extracted and stored for subsequent use as neural network architecture configurations for the neural networks of the corresponding candidate neural network pathway. This can include, for each of the encoder neural network 120, decoder neural network 122, and generative model 112 being trained, the extraction and storage of, for example, parameters that specify a fully connected layer neural network architecture, a convolutional layer neural network architecture, a recurrent neural network layer, a number of connected hidden neural network layers, an input layer architecture, an output layer architecture, a number of nodes utilized by the neural network, coefficients (e.g., weights and biases) utilized by the neural network, kernel parameters, a number of filters utilized by the neural network, strides / pooling configurations utilized by the neural network, an activation function of each neural network layer, interconnections between neural network layers, neural network layers to skip, and so forth.

[0068] Otherwise, if the training system determines that the candidate neural network pathway has not been sufficiently trained at block 712, then at block 716 the training system updates the architectural configurations for the neural networks in the candidate neural network pathway being trained by, for example, updating the weights and other parameters of the training instances of the neural networks 120, 122, and 112 using gradient back-propagation based techniques based on the aforementioned loss function(s). After updating, the method 700 returns to block 706 in which another batch of training data sets is obtained and another iteration of the training process is performed with this next batch of training data.

[0069] In some embodiments, certain aspects of the techniques described above may be implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software can include the instructions and certain data that, when executed by the one or more processors, manipulate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium can include, for example, a magnetic or optical disk storage device, solid state storage devices such as Flash memory, a cache, random access memory (RAM) or other non-volatile memory device or devices, and the like. The executable instructions stored on the non-transitory computer-readable storage medium may be in source code, assembly language code, object code, or other instruction format that is interpreted or otherwise executable by one or more processors.

[0070] A computer-readable storage medium may include any storage medium, or combination of storage media, accessible by a computer system during use to provide instructions and / or data to the computer system. Such storage media can include, but is not limited to, optical media (e.g., compact disc (CD), digital versatile disc (DVD), Blu-Ray disc), magnetic media (e.g., floppy disc, magnetic tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or Flash memory), or microelectromechanical systems (MEMS)-based storage media. The computer- readable storage medium may be embedded in the computing system (e.g., system RAM or ROM), fixedly attached to the computing system (e.g., a magnetic harddrive), removably attached to the computing system (e.g., an optical disc or Universal Serial Bus (USB)-based Flash memory), or coupled to the computer system via a wired or wireless network (e.g., network accessible storage (NAS)).

[0071] Note that not all of the activities or elements described above in the general description are required, that a portion of a specific activity or device may not be required, and that one or more further activities may be performed, or elements included, in addition to those described. Still further, the order in which activities are listed are not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, one of ordinary skill in the art appreciates that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present disclosure.

[0072] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and any feature(s) that may cause any benefit, advantage, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature of any or all the claims. Moreover, the particular embodiments disclosed above are illustrative only, as the disclosed subject matter may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the claims below. It is therefore evident that the particular embodiments disclosed above may be altered or modified and all such variations are considered within the scope of the disclosed subject matter. Accordingly, the protection sought herein is as set forth in the claims below.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method, in a user equipment, comprising: receiving, at a network interface, a first signal representative of input prompt information; processing the first signal at a first neural network of the user equipment to generate a second signal; generating output content based on processing of the second signal at a generative model at the user equipment; and transferring the output content to a user experience component of the user equipment.

2. The method of claim 1 , wherein generating the output content is further based on processing sensor data from one or more sensors of the user equipment at the generative model.

3. The method of claim 2, wherein the one or more sensors comprise: a position sensor, a pose sensor, a camera, a microphone, a thermal sensor, a battery sensor, a network interface, a processing resource utilization sensor, a radar sensor, or a lidar sensor.

4. The method of claim 2 or 3, further comprising: processing the sensor data at a second neural network to generate a third signal; and transmitting the third signal to a network component connected to the user equipment via a network channel.

5. The method of claim 4, wherein the network channel is a wireless network channel.

6. The method of claim 4, wherein the second neural network is jointly trained with a third neural network of the network component.

7. The method of any of the preceding claims, further comprising: receiving, from a network component, an indication of the first neural network responsive to transmitting context information from the user equipment to the network component, the context information representing a present context of the user equipment; and implementing the first neural network at the user equipment responsive to receiving the indication.

8. The method of claim 7, wherein transmitting context information comprises transmitting the context information as at least one of a UE Capability Information Radio Resource Control message or UE Assistance Information Radio Resource Control message.

9. The method of claim 7, wherein the context information includes at least one of: present capabilities of the user equipment; an application type of an application that is to process the output content; a quality-of-service parameter; a quality-of-experience parameter; a present location of the user equipment; or a network condition of a network channel between the network component and the user equipment.

10. The method of any of claims 7 to 9, wherein the indication of the first neural network comprises at least one of: an identifier of one of a plurality of candidate neural networks accessible by the user equipment; or data representing a neural network architectural configuration of the first neural network.11 . The method of any preceding claim, wherein the first neural network is jointly trained with a second neural network at a network component.

12. The method of claim 11 , wherein the first neural network is jointly trained with the second neural network and the generative model.

13. The method of any of claims 11 or 12, wherein the first neural network and the second neural network are deep neural networks (DNNs).

14. The method of any of the preceding claims, wherein the generative model comprises at least one of: a generative adversarial network; a generative pre-trained transformer; a variational autoencoder; or a diffusion network.

15. The method of any preceding claim, wherein the network interface is a wireless network interface.

16. The method of any preceding claim, wherein the user experience component is wirelessly connected to the user equipment.

17. A user equipment comprising: a radio frequency interface; at least one processor coupled to the radio frequency interface; and a non-transitory computer-readable medium storing a set of instructions, the set of instructions configured to manipulate one or both of the at least one processor or the radio frequency interface to perform the method of any of claims 1 to 17.

18. A computer-implemented method comprising: receiving, at a network component, input prompt information from an application server; processing the input prompt information at a first neural network of the network component to generate a first signal for use as one or more input prompts for a generative model at a user equipment connected to the network component via a network channel; and transmitting the first signal for receipt by the user equipment.

19. The method of claim 18, further comprising:receiving a second signal from the user equipment, the second signal representative of sensor data captured by one or more sensors of the user equipment; processing the second signal at a second neural network of the network component to generate a first output; and providing the first output to the application server.

20. The method of claim 19, wherein: the second neural network is jointly trained with a third neural network at the user equipment.21 . The method of claim 18, wherein the first neural network is jointly trained with a second neural network of the user equipment.

22. The method of claim 21 , further comprising: receiving, from the user equipment, context information representing a present context of the user equipment; and providing, to the user equipment, an indication of the second neural network based on the context information.

23. The method of claim 22, wherein the indication of the second neural network comprises at least one of: an identifier of one of a plurality of candidate neural networks accessible by the user equipment; or data representing a neural network architectural configuration of the second neural network.

24. The method of any of claims 21 to 23, wherein the first neural network and the second neural network comprise deep neural networks (DNNs).

25. The method of any of claims 21 to 24, wherein the first neural network is jointly trained with the second neural network and the generative model of the user equipment.

26. The method of any of claims 18 to 25, wherein transmitting the first signal for receipt by the user equipment comprises wirelessly transmitting the first signal for receipt by the user equipment.

27. A network component comprising: a radio frequency interface; at least one processor coupled to the radio frequency interface; and a non-transitory computer-readable medium storing a set of instructions, the set of instructions configured to manipulate one or both of the at least one processor or the radio frequency interface to perform the method of any of claims 16 to 23.

28. A method in a cellular system, comprising: configuring a network component to implement a first neural network and a user equipment to use a second neural network and a generative model based on one or both of a present context of the user equipment or a network condition of a network channel connecting the network component and the user equipment; generating, at an application server, data representative of prompt information for use by a generative model implemented at a user equipment; processing the data at the first neural network of the network component to generate a first signal and transmitting the first signal to the user equipment; processing the first signal at the second neural network of the user equipment to generate a second signal; processing the second signal at the generative model of the user equipment to generate an output content; and processing the output content at the user equipment for facilitating a user experience.

29. The method of claim 28, further comprising: processing sensor data at the generative model concurrent with processing the second signal to generate the output content, the sensor data captured by one or more sensors of the user equipment.

30. The method of claim 29, further comprising: processing the sensor data at a third neural network of the user equipment to generate a third signal and transmitting the third signal to the network component; processing the third signal at a fourth neural network of the network component to generate a fourth signal; and providing the fourth signal to the application server.31 . A system comprising: a network component and a user equipment to perform the method of any of claims 28 to 30.

Images