A wireless system that employs an end-to-end neural network configuration for data streaming

Abstract

The systems and techniques provide for joint training and implementation of an end-to-end chain (116) of neural networks (120, 124, 128, 130, 132, 134) along at least partially the nodes of a wireless transmission path used to transmit a data stream (106) between a data source device (102-1) and at least one data sink device (102-2). The source side neural networks (120, 128) of the chain may perform one or both of data encoding and channel encoding of outgoing data blocks, and the sink side neural networks (124, 134) of the chain may conversely perform one or both of channel decoding and data decoding, providing efficient end-to-end transmission of the data stream without requiring separate design, testing, and implementation of discrete processing for each encoding and decoding stage, while also facilitating adjustment of the end-to-end neural network chain processing to various operating parameters.

Description

Background Art

[0001] Background In a wireless system, the transmission of a data stream between a data source device and a data sink device typically involves a transmission path having a series of nodes, each node performing one or more processes on the signal transmission representing the data of the stream. For example, for a video call being streamed between a first user equipment (UE) and a second UE, the video captured by the first UE is decomposed into a stream of data blocks. Each data block is compressed using video compression processing, and the resulting compressed data block is then channel encoded to generate an output signal suitable for radio frequency (RF) transmission to an infrastructure component directed to the first UE, such as a base station, a wireless local area network (WLAN) access point. The infrastructure component directed to the first UE performs a decoding (and demodulation) process on the received RF signal to obtain the underlying data, which is then packetized or processed for network transmission to an infrastructure component directed to the second UE via one or more network components. Each infrastructure component along the transmission path similarly depacketizes the incoming signal, performs various processes on the resulting data, and then repacketizes the result for further network transmission. The infrastructure component directed to the second UE then performs additional processing on this received signal, including channel encoding of the signal, and the resulting output can be transmitted to the second UE via RF signal transmission. The second UE decodes (and further demodulates) the incoming signal to obtain the underlying compressed data block, and then decompresses the compressed data block to obtain the original uncompressed data block (or its loss representation depending on the compression / decompression process). The recovered data block can then be processed at the second UE to display the video content represented by the data block.

[0002] The sequence of this process from data generation in the first UE to data consumption in the second UE is typically implemented using module design techniques. Each process is effectively "handcrafted" individually by one or more designers. The relative complexity of each process typically translates into the corresponding complexity in the design, testing, and implementation of the hard-coded implementation form of the process. Therefore, it can become impractical to design and implement highly efficient and robust processing to highly adapt to changing conditions and reduce transmission latency. Also, the supervision of various devices in the transmission path may be split among multiple entities, and thus it becomes difficult to ensure that each device can execute its corresponding processing in a manner that is fully compatible with the capabilities of downstream devices or in an optimal manner for low-latency end-to-end transmission of the data stream. SUMMARY OF THE INVENTION

[0003] SUMMARY OF EMBODIMENTS In one aspect, a method in a data source device includes receiving, as an input to a transmission-side neural network of the data source device, a first data block of a data stream; generating, in the transmission-side neural network, a first output representing a data-encoded and channel-encoded version of the first data block based on the first data block; and controlling, based on the first output, a radio frequency (RF) antenna interface of the data source device to transmit a first RF signal representing the data-encoded and channel-encoded version of the first data block.

[0004] In another aspect, a method implemented on a computer in a data sink device includes receiving, at an RF antenna interface of the data sink device, a first RF signal representing a data-encoded and channel-encoded version of a first data block of a data stream; providing, as an input to a receiving neural network of the data sink device, a first input representing the first RF signal; generating, in the receiving neural network, a first recovered data block representing a recovered, channel-decoded, and data-decoded version of the first data block; and providing the first recovered data block for processing in one or more software applications of the data sink device.

[0005] In another aspect, a method implemented on a computer in an infrastructure component of a network infrastructure includes configuring a data source device to implement a first neural network architecture configuration for a transmitting neural network of the data source device, the transmitting neural network being configured to generate, for each input data block of a data stream generated at the data source device, an output corresponding to transmission by a radio frequency (RF) antenna interface of the data source device, the corresponding output representing a data-encoded and channel-encoded version of the input data block, and the method includes configuring a data sink device to implement a second neural network architecture configuration for a receiving neural network of the data sink device, the receiving neural network being configured to generate, for each input from an RF antenna interface of the data sink device, a corresponding data block for providing to one or more software applications of the data sink device, the corresponding data block representing a recovered, channel-decoded, and data-decoded version of the corresponding data block of the data stream.

[0006] The device may include a network interface, at least one processor coupled to the network interface, and a memory storing executable instructions that operate the at least one processor to execute any of the foregoing manners.

[0007] The device may include an RF antenna interface, at least one processor coupled to the RF antenna interface, and a memory storing executable instructions that operate the at least one processor to execute any of the foregoing manners.

[0008] By reference to the accompanying drawings, those skilled in the art will better understand the present disclosure and its many functions and advantages will become apparent. The use of the same reference numerals in separate drawings indicates similar or identical items.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

[0010] Detailed Description Figures 1-12 illustrate systems and techniques for the joint training and implementation of an end-to-end chain of a neural network along nodes of at least a partially wireless transmission path used to transmit a data stream between a data source device and one or more data sink devices. While providing efficient end-to-end transmission of the data stream without the need for individual design, testing, and implementation of discrete processing for each stage of encoding and decoding, the source-side neural network of the chain can perform one or both of data encoding and channel encoding of the transmitted data block, and the sink-side neural network of the chain can, conversely, perform one or both of channel decoding and data decoding, in order to facilitate the adjustment of end-to-end neural network chain processing for various operating parameters such as RF operating environment parameters represented in the captured sensor data, changes in the capabilities of one or more nodes along the transmission path.

[0011] FIG. 1 shows a wireless communication system 100 that employs an end-to-end neural network configuration for data streaming according to some embodiments. The system 100 includes a data source device 102-1 connected to one or more data sink devices 102-2 via a network infrastructure 104 that consists of various infrastructure components such as a base station (BS), a wireless local area network (WLAN) access point (AP), a core network, a non-core network (e.g., the Internet), and an application server. In operation, the data source device 102-1 generates an outgoing data stream 106-1, which is received and processed by the network infrastructure 104 to generate a corresponding incoming data stream 106-2 that is a lossy or lossless representation of the data content of the outgoing data stream 106-1 depending on the implementation, and this incoming data stream 106-2 is received, processed, and consumed by the data sink device 102-2. In a two-way communication scenario such as when the system 100 is providing a voice or video call service between devices 102-1 and 102-2, it should be noted that the roles of these devices are reversed such that device 102-2 acts as the data source device for the data streamed from device 102-2 to device 102-1, and device 102-1 acts as the data sink device for this streamed data. Therefore, it is understood that "source" and "sink" are not fixed references to specific devices in system 100 but are relative with respect to the directed flow of the corresponding data stream.

[0012] The network infrastructure 104 includes components of a wireless communication system 100 that operates to transfer streaming data from a data source device 102-1 to a data sink device 102-2. Such components include, for example, a source-oriented infrastructure component 108-1 that communicates wirelessly or wired with the data source device 102-1, a sink-oriented infrastructure component 108-2 that communicates wirelessly or wired with the data sink device 102-2, and a network, server, and other components that facilitate communication between the infrastructure components 108-1 and 108-2. These intermediate infrastructure components may include, for example, a core network 1 (CN1) 110-1 associated with the source-oriented infrastructure component 108-1, a core network 2 (CN2) 110-2 associated with the sink-oriented infrastructure component 108-2, and one or more non-core networks 112 that connect CN1 110-1 and CN2 110-2. And also, the one or more non-core networks 112 may include or be coupled to one or more application servers or other infrastructure components that provide support for services on which the data stream is based. Note that in the case where the source-oriented infrastructure component 108-1 and the sink-oriented infrastructure component 108-2 (collectively "device-oriented infrastructure components 108") are supported by the same network operator, CN1 110-1 and CN2 110-2 may be the same core network.

[0013] In a specific example of FIG. 1, data source device 102-1 and data sink device 102-1 are UEs (such as smartphones, tablet computers, laptop computers, desktop computers, networked systems in vehicles, etc.) or other non-infrastructure devices (such as wireless repeaters) wirelessly connected to a mobile communication base station or other infrastructure components, the infrastructure component 108-1 towards the source is a mobile communication base station (BS), and the infrastructure component 108-2 towards the sink is a WLAN AP. Therefore, data source device 102-1, data sink device 102-2, infrastructure component 108-1 towards the source, and infrastructure component 108-2 towards the sink are also referred to herein as UE102-1, UE102-2, BS108-1, and AP108-2, respectively. UE102-1 and BS108-1 are connected using a radio access technology (RAT) for mobile communication such as the 3rd Generation Partnership Project (3GPP (registered trademark)), 4th Generation Long Term Evolution (4G LTE) RAT or 3GPP 5th Generation New Radio (5G NR) RAT. UE102-2 and AP108-2 are connected using a WLAN RAT such as an Institute of Electrical and Electronics Engineers (IEEE) 802.11-based RAT (i.e., "WiFi" RAT). And also, BS108-1 and AP108-2 are connected via one or more non-core networks 112 such as via the Internet via their respective core networks 110-1 and 110-2. For ease of explanation, various aspects of the present disclosure will be described below with reference to this specific example implementation. However, the technology described herein is not limited to this implementation and can be adopted in any of various network configurations. For example, both infrastructure components towards the device may be mobile communication base stations, or both infrastructure components towards the device may be WLAN access points.In addition, on top of that, one device can be connected to the infrastructure components directed to its corresponding device via a wired network. For example, the data source device 102-1 can be connected to CN2 110-2 via the Internet and can include a cloud-based video game server that remotely executes an instance of a video game application, and the resulting video content and audio content are streamed to the data sink device 102-2 via the Internet, CN2 110-2, and AP108-2. The radio RAT used to connect one of the devices 102 to the infrastructure components directed to its corresponding device can include, in addition to the RAT for mobile communication or the WLAN RAT, a RAT such as a Bluetooth (registered trademark)-based RAT or other wireless personal area network (WPAN) RAT.

[0014] Data streaming from a data source device to a data sink device in a conventional wireless communication system relies on a series of processing blocks such as source encoding (e.g., data compression), channel encoding, channel decoding, and destination decoding (e.g., data decompression), which are designed, tested, and implemented relatively separately from each other. This convention and individual design approach for each process result in high complexity and a significant lack of resilience to changes in situation parameters. Instead of taking a manual approach for each of the individual processes, the wireless communication system 100 adopts an end-to-end neural network approach that provides rapid development and deployment, flexibility, and QoE optimization. In at least one embodiment, the end-to-end transmission path between the data source device 102-1 that generates a data stream and the data sink device 102-2 that consumes the data stream implements a chain 116 of DNNs or other neural networks across the nodes of the end-to-end transmission path, which is trained to provide substantially equivalent processing to a conventional series of processes, eliminating the need for special design and testing for that series of processes.

[0015] As an example, the transmitting neural network 120 in the data source device 102-1 can be trained to receive a series of data blocks of a data stream from the data source module 122 and generate, for each input data block, an output of a result representing an equivalent of the data-encoded and channel-encoded version of the data block. Conversely, the receiving neural network 124 of the data sink device 102-2 can be trained to receive a series of input signals, each representing a data-encoded and channel-encoded data block of a data stream, and generate a recovered, channel-decoded and destination-decoded version of the corresponding data block to supply to a data sink module 126 that consumes or processes this data block. That is, the transmitting neural network 120 of the data source device 102-1 can be configured, by joint training, to provide an equivalent of a conventional data encoding process involving a conventional channel encoding process for RF transmission preparation. On the other hand, the receiving neural network 124 of the data sink device 102-2 can be configured, by joint training, to provide an equivalent of a conventional channel decoding process involving a data decoding process for generating a recovered data block. In a similar way, some or all of the receiving-side / transmitting-side neural networks of the infrastructure components of the network infrastructure 104 between the data source device 102-1 and the data sink device 102-2 can be trained to receive and decode (including demodulating) incoming RF signals for further processing by the wireless infrastructure, or to encode (including modulating) data from the stream for further transmission along the transmission path. For example, BS108-1 can include an RX neural network 128 that channel-decodes and destination-decodes an RF signal transmission received from the data source device 102-1 and supplies the resulting output to CN1 110-1, and CN1 110-1 can likewise employ one or more neural networks 130 to process a signal transmission representing the streamed content for transmission to CN2 110-2.Thus, CN2 110-2 may employ one or more neural networks 132 to further process this signal transmission for transmission to AP110-2. And also, AP110-2 may employ a TX neural network 134 to supply an output that is the equivalent of the source-encoded and channel-encoded representation of the information received from CN2 110-2 for RF transmission to the data sink device 102-2. The data sink device 102-2, as described above, may employ a sink RX neural network 124 to process this signal transmission to recover the representation of the original data supplied to the data sink module 126. The result is an end-to-end chain of neural networks that are trained together to provide efficient processing of data blocks of the data stream at various nodes of the transmission path without the need for extensive design, testing, and implementation of manual processing at each node.

[0016] In at least one embodiment, the network infrastructure 104 includes an infrastructure management component 136 (or simply "management component 136") that serves to manage the overall operation of the end-to-end chain 116, and the infrastructure management component 136 includes one or more of steps such as monitoring the collaborative training of the neural network of the end-to-end chain 116, managing the selection of a specific neural network architecture configuration for one or more of the nodes of the end-to-end chain 116, receiving and processing an ability update for the selection of the neural network configuration, and receiving and processing feedback for the training or selection of the neural network. The management component 136 may include one of the servers of the CN 110 or other networked components, a server or other component connected to the non-core network 112, etc. For example, in a case where both infrastructure components 108 for devices are coupled to the same one core network and thus the end-to-end chain 116 is controlled by one network operator, the management component 136 may be implemented in one of the infrastructure components for devices or a server of the shared core network. In other cases where the transmission path spans multiple core networks and thus multiple network operators, the management component 136 may include, for example, a third-party server or other intermediate component located in the non-core network 112. Moreover, the functionality attributed to the management component 136 herein may be distributed across multiple infrastructure components, such as between the servers of separate core networks 110, between the servers of the core network 110 and the infrastructure components 108 for corresponding devices, etc.

[0017] As will be described in more detail below, the management component 136 can select a particular neural network architecture to be employed at a particular location in the end-to-end neural network chain 116, based at least in part on the current capabilities of the components implementing the corresponding neural network, the current capabilities of other components in the transmission chain, or a combination thereof. These capabilities can include, for example, sensor capabilities, processing resource capabilities, battery / power capabilities, RF antenna capabilities, as well as data generation capabilities of the source of the data stream (e.g., image resolution capabilities) or data consumption capabilities of the consumer part of the data stream (e.g., display resolution of the display component used to display the image content of the video stream) of one or more accessories of one or both of the data source device or the data sink device. For this purpose, in some embodiments, the management component 136 can manage the joint training of different combinations of neural network architecture configurations for different combinations of capabilities. Thus, the management component 136 can obtain capability information from the data source device, the data sink device, and one or more intermediate nodes in the transmission path. The management component 136 selects, from this capability information, a neural network architecture configuration for each component in the path, based at least in part on the indicated corresponding capabilities, thus resulting in an overall configuration of the end-to-end neural network chain 116 between the data source device 102-1 and the data sink device 102-2 in a manner that better adapts to the capabilities of the nodes in the chain.

[0018] Furthermore, in some embodiments, when a data stream is transmitted and processed, the data sink device 102-2 supplies to the management component 136 feedback in the form of objective quality metrics (i.e., objective feedback independent of user input) or subjective quality metrics supplied by the user (i.e., subjective feedback based on user input), and the management component 136 utilizes this feedback to further improve or change the neural network architecture configuration of one or more neural networks in the end-to-end neural network chain 116. This may include, for example, the management component 136 improving the weights used in the current architecture configuration or determining a new, separate DNN architecture to adopt. Similarly, when a node in the end-to-end chain 116 experiences a change in capabilities, such as a decrease in available battery power, addition or removal of accessories, or a change in the signal-to-noise ratio in the RF antenna interface, the management component 136 can switch the neural network architecture configuration employed by that node to a new or modified configuration to adapt to the changed capabilities.

[0019] Using an end-to-end chain of a neural network that is managed and trained together, rather than a separately designed processing block that may not be specially designed for optimal compatibility between a data source device and a data sink device, facilitates the overall data streaming process based on the trained cooperation. This not only improves flexibility, but in certain situations, can provide faster processing at each node, thus reducing the latency in the transmission of each data block of the data stream compared to a modular design approach. This reduction in latency is particularly relevant to the improvement of the quality of experience (QoE) for real-time data streams such as the audio stream of a voice call between two UEs, the audio stream or video stream of a video call between two UEs, or the rendered video stream that is generated by a cloud-based video game server and streamed to be displayed on a UE.

[0020] FIG. 2 shows an exemplary hardware configuration for either the data source device 102-1 and the data sink device 102-2 (collectively or individually, "device 102") according to some embodiments. Note that the represented hardware configuration represents the processing components and communication components most directly related to the neural network-based processing described herein, and certain components that are well understood to be frequently implemented in such electronic devices are omitted.

[0021] In the illustrated configuration, device 102 includes one or more antenna arrays 202, where each antenna array 202 has one or more antennas 203, and further includes an RF antenna interface 204, one or more processors 206, and one or more non-transitory computer-readable media 208. The RF antenna interface 204 operates substantially as a physical (PHY) transceiver interface that transmits and processes signals between the one or more processors 206 and the antenna array 202 to facilitate various types of wireless communication. The antennas 203 may include an array of multiple antennas of similar or different configurations to each other and may be tuned to one or more frequency bands associated with the corresponding RAT. The one or more processors 206 may include, for example, one or more central processing units (CPUs), graphics processing units (GPUs), artificial intelligence (AI) accelerators, or other application-specific integrated circuits (ASICs). As an example, the processor 206 may include an application processor (AP) utilized by the device 102 to execute an operating system and various user-level software applications, as well as one or more processors utilized by the modem or baseband processor of the RF antenna interface 204. The computer-readable media 208 may include any of various media such as random access memory (RAM), read-only memory (ROM), cache, flash memory, solid-state drive (SSD), or other mass storage devices that are used by an electronic device to store data and / or executable instructions. Since system memory or other memory is frequently used by the processor 206 to store data and execution instructions, for ease and brevity of explanation, the computer-readable media 208 is referred to herein as "memory 208", but it should be understood that references to "memory 208" apply equally to other types of storage media unless otherwise noted.

[0022] In at least one embodiment, device 102 further includes a plurality of sensors, herein referred to as sensor set 210, at least some of which are utilized in the neural network-based approach described herein. Generally, the sensors of sensor set 210 include sensors that sense some aspects of the environment of device 102 or the usage of the device by the user, and these sensors have the ability to sense parameters that at least somewhat affect or reflect the RF propagation path or RF transmit / receive performance that device 102 has with respect to infrastructure component 108 directed to the corresponding device. The sensors of sensor set 210 can include one or more sensors for object detection, such as radar sensors, lidar sensors, imaging sensors, structured light-based depth sensors, etc. Sensor set 210 can also include one or more sensors for determining the position or orientation of device 102, such as satellite positioning sensors like GPS sensors, global navigation satellite system (GNSS) sensors, inertial measurement unit (IMU) sensors, visual odometry sensors, gyroscopes, tilt sensors or other inclinometers, ultra-wideband (UWB)-based sensors, etc. Other examples of the types of sensors of sensor set 210 can include imaging sensors such as cameras for user image capture, cameras for face detection, cameras for stereoscopy or visual odometry, and light sensors for detecting objects in proximity to the features of the device.

[0023] Device 102 may further include one or more batteries 212 or other portable power sources, and one or more user interface (UI) components 214 such as a touch screen, user-operable input / output devices (such as "buttons" or a keyboard), or other touch / contact sensors, a microphone or other audio sensors for capturing audio content, an image sensor for capturing video content, a thermal sensor (such as for detecting proximity to the user), etc. Further, device 102 may include a display panel 216 for displaying video content and one or more speakers for outputting audio content. Further, in at least one embodiment, device 102 may include one or more wired or wireless accessories 220 related to the generation of streamed content or for consuming content received in a stream. For example, when operating as a data sink device 102-2, device 102 may include a wired or wireless audio headset accessory used to output the audio content of a received audio stream, a wired or wireless head-mounted display (HMD) used to display the video content of a received video stream, etc. Conversely, when operating as a data source device 102-1, device 102 may include accessories in the form of, for example, a wired or wireless microphone used to capture audio content, a wired or wireless video camera for capturing video content, a printer or other peripheral device controlled based on a data stream, etc.

[0024] One or more memories 208 of device 102 are used to store one or more sets of executable software instructions and corresponding data that operate one or more processors 206 and other components of device 102 to perform the various functions described herein and attributed to device 102. The set of executable software instructions includes, for example, an operating system (OS) and various drivers (not shown), and various software applications. The set of executable software instructions further includes a neural network management module 222, a capability management module 224, and a feedback management module 226. The neural network management module 222 implements one or more neural networks for device 102, as will be described in detail below. The capability management module 224 monitors device 102 for changes in the capabilities of device 102, including changes in RF and processing capabilities, availability or capabilities of accessories, etc., and manages the reporting of such capabilities and changes in capabilities to management component 136. When device 102 is operating as data sink device 102-2, the feedback management module 226 operates to obtain either or both objective feedback regarding the processing of the received data stream (such as one or more standardized quality of service (QoS) or quality of experience (QoE) metrics) or subjective feedback from the user regarding the processing and / or utilization of the received data stream, and provides a representation of this feedback to management component 136. These operations will be described in more detail below.

[0025] To facilitate the operation of device 102 described herein, one or more memories 208 of device 102 can further store data related to these operations. This data can include, for example, device data 228 and one or more neural network architecture configurations 230. Device data 228 can represent, for example, user data, multimedia data, beamforming codebooks, software application configuration information, and the like. Device data 228 can further include capability information regarding one or more sensors of sensor set 210, including, for example, the presence or absence of a particular sensor or sensor type for device 102, and can present one or more representations of corresponding capabilities, such as the range and resolution of a lidar sensor or radar sensor, and the image resolution and color depth of an imaging camera, for those sensors. The capability information can further include information regarding the capabilities of one or more accessories 220 associated with device 102, such as the screen resolution, color gamut, or frame rate of a display accessory, and the frequency response, sample rate, and number of channels of an audio headset accessory.

[0026] One or more data structures included in one or more neural network architecture configurations 230 contain data and other information representing corresponding architecture and / or parameter configurations that are used by neural network management module 222 to form the corresponding neural network of device 102. The information included in neural network architecture configuration 230 includes, for example, fully connected layer neural network architecture, convolutional layer neural network architecture, recurrent neural network layer, multiple connected hidden neural network layers, input layer architecture, output layer architecture, multiple nodes utilized by the neural network, coefficients (such as weights and biases) utilized by the neural network, kernel parameters, multiple filters utilized by the neural network, stride / pooling configurations utilized by the neural network, activation function of each neural network layer, interconnections between neural network layers, neural network layers to skip, etc. Thus, neural network architecture configuration 230 can be used to define a DNN and / or generate a configuration for forming an NN (e.g., a combination of components of one or more NN formation configurations) that includes any combination of components of NN formation (e.g., architecture and / or parameter configurations).

[0027] Further, when operating as the data source device 102-1, the device 102 includes the data source module 122, when operating as the data sink device 102-2, it includes the data sink module 126, or when operating as both the data source device for the outgoing data stream and the data sink device for the incoming data stream, it includes both modules. One or both of the data source module 122 and the data sink module 126 can be implemented as software applications such as the data source application 232 or the data sink application 234 stored in one or more memories 208 of the device 102, respectively. In other embodiments, one or both of the data source module 122 and the data sink module 126 are implemented as hardware modules such as ASICs or programmable logic devices (PLDs), and in still other embodiments, one or both of the modules 122 and 126 are implemented as a combination of one or more hardware modules and one or more software applications.

[0028] Figure 3 shows an exemplary hardware configuration of an infrastructure component 108 directed to a device, such as an infrastructure component 108-1 directed to a source or an infrastructure component 108-2 directed to a sink, according to some embodiments. It should be noted that the represented hardware configuration represents the processing components and communication components most directly related to the neural network-based processing described herein, and certain components that are well understood to be frequently implemented in such electronic devices are omitted. Further, Figure 3 shows an infrastructure component 108 directed to a device as a single network node (e.g., a 5G NR Node B or a WiFi AP), but it is noted that the functionality of the infrastructure component 108 directed to a device, and thus the hardware components, may instead be distributed across multiple infrastructure components or nodes and may be distributed to perform the functions described herein.

[0029] The infrastructure component 108 for the device, like the device 102, includes at least one array 302 of one or more antennas 303, an RF antenna interface 304, as well as one or more processors 306 and one or more non-transitory computer-readable storage media 308 (the computer-readable medium 308 is referred to as "memory 308" in this specification for brevity, similar to the memory 208 of the device 102). The infrastructure component 108 for the device further includes a sensor set 310 having one or more sensors that provide sensor data that can be used in the NN-based sensor and transceiver fusion method described herein. Similar to the sensor set 210 of the device 102, the sensor set 310 of the infrastructure component 108 for the device can include, for example, object detection sensors and imaging sensors, and in cases where the infrastructure component 108 for the device is movable (such as when implemented in a vehicle or a drone), can include one or more sensors for detecting position or orientation. These components operate in a manner similar to those described above with reference to the corresponding components of the device 102.

[0030] One or more sets of executable software instructions and corresponding data stored in one or more memories 308 of infrastructure component 108 for a device operate one or more processors 306 and other components of infrastructure component 108 for a device to perform various functions described herein that are attributed to infrastructure component 108 for a device. The set of executable software instructions includes, for example, an operating system (OS) and various drivers (not shown), various software applications (not shown), a component management module 312, and a neural network management module 314. The component management module 312 configures the RF antenna interface 304 for communication with the device 102 and for communication with a core network, such as one of the core networks 110. The neural network management module 314 implements one or more neural networks for infrastructure component 108 for a device, such as the neural networks employed in the TX and RX processing paths as described herein.

[0031] In at least one embodiment, the software stored in memory 308 further includes one or more of training module 316 and capacity management module 318. Training module 316 is operative to train one or more neural networks implemented in infrastructure component 108 or device 102 directed to the device using one or more sets of input data. This training can be performed for various purposes such as the processing of communications transmitted across the wireless communication system. The training can include training the neural network offline (i.e., without actively participating in the communication processing) and / or online (i.e., while actively participating in the communication processing). Still further, the training can be individual or separate such that each neural network is trained individually on its own dataset without the results being communicated or affecting the DNN training at the opposite end of the transmission path, or the training can be joint training such that the neural networks in the data stream transmission path are trained together on the same or complementary datasets. Capacity management module 318 of infrastructure component 108 directed to the device monitors changes in the capacity of infrastructure component 108, such as changes in RF or processing capacity, and manages the reporting of such capacity and capacity changes to management component 136, similar to capacity management module 224 of device 102.

[0032] Data stored in one or more memories 308 of the infrastructure component 108 for the device includes, for example, component data 320 and one or more neural network architecture configurations 322. The component data 320 represents, for example, network scheduling data, wireless communication resource management data, beamforming codebooks, software application configuration information, etc. The component data 320 further includes sensor capability information regarding one or more sensors of the sensor set 310, including the presence or absence of a particular sensor or sensor type and one or more representations of their corresponding capabilities regarding the sensors present. The one or more data structures included in the one or more neural network architecture configurations 322 contain data and other information representing the corresponding architecture and / or parameter configuration used by the neural network management module 314 to form the corresponding neural network of the infrastructure component 108 for the device. The information included in the neural network architecture configuration 322 is similar to the neural network architecture configuration 230 of the device 102 and includes, for example, fully connected layer neural network architecture, convolutional layer neural network architecture, recurrent neural network layer, multiple hidden neural network layers connected, input layer architecture, output layer architecture, multiple nodes utilized by the neural network, coefficients utilized by the neural network, kernel parameters, multiple filters utilized by the neural network, stride / pooling configuration utilized by the neural network, activation function of each neural network layer, interconnection between neural network layers, neural network layers to skip, etc. parameters that specify. Thus, the neural network architecture configuration 322 includes any combination of components of the NN formation that can be used to define a DNN or other neural network and / or generate a NN formation configuration that forms it.In some embodiments, the infrastructure component 108 for the device further includes a core network interface 324 configured by the component management module 312 to exchange user plane, control plane, and other information with core network functions and / or entities.

[0033] FIG. 4 shows an exemplary hardware configuration of the management component 136 according to some embodiments. Note that the presented hardware configuration represents the processing components and communication components most directly related to the neural network-based processing described herein, and certain components that are well understood to be frequently implemented in such electronic devices are omitted. Further, although the hardware configuration is shown as being disposed in one component, the functionality of the management component 136, and thus the hardware components, may alternatively be distributed across multiple infrastructure components or nodes and may be distributed to perform the functions described herein.

[0034] As described above, the management component 136 may be implemented in any combination of components, such as various components within the network infrastructure 104, or infrastructure components 108 for base stations, access points, or other devices, servers or other components of the core network 110, application servers or other components in a non-core network 112 such as a private network or an Internet network. For ease of explanation, the management component 136 is described herein with reference to an exemplary implementation as a server or other component in one of the core networks 110.

[0035] As shown, the management component 136 includes one or more network interfaces 402 (e.g., Ethernet® interfaces) for coupling to one or more networks of the system 100, one or more processors 404 coupled to the one or more network interfaces 402, and one or more non-transitory computer-readable storage media 406 (referred to herein as "memory 406" for brevity) coupled to the one or more processors 404. One or more sets of executable software instructions and corresponding data stored in the one or more memories 406 operate the one or more processors 404 and other components of the management component 136 to perform the various functions described herein and attributed to the management component 136. The set of executable software instructions includes, for example, an OS, various drivers (not shown), and one or more network applications that support a data stream transmitted from the data source device 102-1 to the data sink device 102-2. For example, in an implementation of Voice over Internet Protocol (VoIP) where voice content captured by the data source device 102-1 is transmitted as a corresponding data stream to the data sink device 102-2, the supporting software application can include one or more IP Multimedia Subsystem (IMS) applications configured to facilitate, for example, the initiation, maintenance, and termination of a corresponding VoIP connection.

[0036] The software stored in the one or more memories 406 may further include a training module 412 that operates to manage the collaborative training of neural networks that use one or more sets of training data 416, which may be employed throughout the neural network chain 116. The training may include training the neural networks offline (i.e., without actively participating in the communication process) and / or online (i.e., while actively participating in the communication process). Still further, the training may be individual or separate such that each neural network is trained individually on its own training data set without the results being communicated to or affecting the DNN training at the opposite end of the transmission path, or the training may be collaborative training in which the neural networks in the data stream transmission path are trained together on the same or complementary data sets. Other data stored in the one or more memories 406 includes, for example, chain data 418 and one or more neural network architecture configurations 420. The chain data 418 represents, for example, the current capability information of some or all of the infrastructure components 108, core network 110, and devices 102 in the transmission path of the supported data stream, the identifier of the neural network architecture configuration implemented at each of these nodes or an indication of the parameters of the neural network architecture configuration implemented at the node, and feedback information as a streaming of data progression received from the data sink device 102-2.

[0037] In an implementation where the management component 136 operates in one of the core networks 110 and is thus in the transmission path of the data stream, the memory 406 can utilize the neural network management module 410 to implement one or more neural networks, such as the neural networks employed in one or both of the TX processing path and the RX processing path as described herein, to facilitate the transmission of the data stream. Similar to the neural network architecture configuration of the device 102 and the infrastructure component 108 directed to the device, one or more neural network architecture configurations 420 contain one or more data structures that represent the corresponding architecture and / or parameter configurations used by the neural network manager 410 to form the corresponding neural network of the management component 136 and other information. This information may include, for example, a fully connected layer neural network architecture, a convolutional layer neural network architecture, a recurrent neural network layer, a plurality of connected hidden neural network layers, an input layer architecture, an output layer architecture, a plurality of nodes utilized by the neural network, coefficients utilized by the neural network, kernel parameters, a plurality of filters utilized by the neural network, a stride / pooling configuration utilized by the neural network, an activation function of each neural network layer, an interconnection between neural network layers, parameters specifying neural network layers to skip, etc. Thus, the neural network architecture configuration 420 includes any combination of components of the NN formation that can be used to define and / or form the configuration of the NN formation that generates the DNN or other neural network.

[0038] FIG. 5 shows an exemplary machine learning (ML) module 500 for implementing a neural network according to some embodiments. As described herein, one or more of device 102, infrastructure components 108 directed to the device, and core network 110 implement one or more DNNs or other neural networks in one or both of a TX processing path or an RX processing path for processing wireless communication inputs and outputs. Accordingly, ML module 500 shows an exemplary module for implementing one or more of these neural networks.

[0039] In the example shown, ML module 500 implements at least one deep neural network (DNN) 502 having a group of nodes (e.g., neurons and / or perceptrons) organized and connected in three or more layers. The nodes between the layers can be configured in various ways, such as a partially connected configuration where a first subset of nodes in the first layer is connected to a second subset of nodes in the second layer, a fully connected configuration where each node in the first layer is connected to each node in the second layer, and the like. Neurons process input data to generate a continuous output value, such as some real number between 0 and 1. In some cases, the output value indicates how close the input data is to a desired category. A perceptron performs a linear classification, such as binary classification, on the input data. Nodes, whether neurons or perceptrons, can use various algorithms to generate output information based on adaptive learning. ML module 500 uses DNN 502 to perform various different types of analysis, including simple regression analysis, multiple regression analysis, logistic regression analysis, sequential regression analysis, binary classification, multi-class classification, multivariate adaptive regression splines, local estimated scatterplot smoothing, and the like.

[0040] In some implementations, the ML module 500 adaptively learns based on supervised learning. In supervised learning, the ML module 500 receives various types of input data as training data. The ML module 500 processes the training data to learn how to map the input to the desired output. As an example, the ML module 500 receives digital samples of a signal as input data and learns how to map the signal samples to binary data that reflects the information embedded in the signal. As another example, the ML module 500 receives binary data as input data and learns how to map the binary data to digital samples of a signal in which the binary data is embedded. Additionally, as another example, as will be described in more detail below, when used in the TX mode, the ML module 500 receives a transmission information block and learns how to generate an output that substantially represents both the data-encoded (e.g., compressed) representation and the channel-encoded representation in the information block in order to form an output suitable for wireless transmission by the RF antenna interface. Conversely, when implemented in the RX mode, the ML module 500 receives an input that substantially represents the data-encoded and channel-encoded representation of the information block, processes the input, and can be trained to generate an output that substantially represents the data-decoded and channel-decoded representation of the input, and thus the recovered representation of the information data. As will be further described below, the training in one or both of the TX mode or the RX mode may further include training using sensor data, capability information, accessory information, etc. as inputs.

[0041] During the training procedure, the ML module 500 uses labeled or known data as input to the DNN 502. The DNN 502 analyzes the input using nodes and generates corresponding outputs. The ML module 500 compares the corresponding outputs with the true data to adapt the algorithms implemented by the nodes in order to improve the accuracy of the output data. Then, the DNN 502 applies the adapted algorithms to unlabeled input data to generate corresponding output data. The ML module 500 maps the input to the output using one or both of statistical analysis and adaptive learning. For example, the ML module 500 uses characteristics learned from training data to statistically associate unknown inputs with likely outputs within a threshold range or range of values. This enables the ML module 500 to receive complex inputs and identify corresponding outputs. As described above, some implementations train the ML module 500 with respect to the characteristics of communications transmitted through a wireless communication system (e.g., time / frequency interleaving, time / frequency deinterleaving, convolutional encoding, convolutional decoding, power levels, channel equalization, intersymbol interference, quadrature amplitude modulation / demodulation, frequency division multiplexing / demultiplexing, transmit channel characteristics) and simultaneously with respect to the characteristics of the data encoding / decoding schemes employed in such a system. Then, the trained ML module 500 can receive samples of a signal as input and recover information from the signal, such as the binary data incorporated in the signal.

[0042] In the example shown, the DNN 502 includes an input layer 504, an output layer 506, and one or more hidden layers 508 disposed between the input layer 504 and the output layer 506. Each layer has an arbitrary number of nodes, and the number of nodes between layers can be the same or different. That is, the input layer 504 can have the same number or a different number of nodes as the output layer 506, the output layer 506 can have the same number or a different number of nodes as one or more hidden layers 508, and so on.

[0043] Node 510 corresponds to one of several nodes included in the input layer 504, and this node performs separate individual computations. As further described, a node receives input data, processes the input data using one or more algorithms, and yields output data. Typically, the algorithms include weights and / or coefficients that vary based on adaptive learning. Thus, the weights and / or coefficients reflect the information learned by the neural network. Each node can, in some cases, determine whether to pass the processed input data to one or more next nodes. As an example, node 510 can determine whether to pass the processed input data to one or both of nodes 512 and 514 in the hidden layer 508 after processing the input data. Alternatively, or in addition, node 510 passes the processed input data to nodes based on a layer connection architecture. This process can be repeated across multiple layers until the DNN 502 generates an output using a node (e.g., node 516) in the output layer 506.

[0044] A neural network can adopt various architectures to determine the connected nodes inside the neural network, the way data evolves and / or is retained in the neural network, the weights and coefficients used to process the input data, the way the data is processed, and so on. These various coefficients, taken together, describe a neural network architecture configuration such as the neural network architecture configuration outlined above. As an example, recurrent neural networks such as long short-term memory (LSTM) neural networks form a cycle between node connections to retain information from previous parts of the input data series. The recurrent neural network then uses the retained information for subsequent parts of the input data series. As another example, a feedforward neural network passes information forward without forming a cycle for retaining information. Although described in terms of node connections, it should be understood that the neural network architecture configuration can include various parameter configurations that affect the way a DNN502 or other neural network processes input data.

[0045] The neural network architecture configuration of a neural network can be characterized by various architectures and / or parameter configurations. As an example, consider an example where DNN502 implements a convolutional neural network (CNN). Generally, a convolutional neural network corresponds to a type of DNN where layers use a convolution operation to process data and filter the input data. Thus, the CNN architecture configuration can be characterized by, for example, pooling parameter(s), kernel parameter(s), weights, and / or layer parameter(s).

[0046] Pooling parameters correspond to parameters that specify the pooling layer inside a convolutional neural network and reduce the dimensionality of the input data. As an example, the pooling layer can combine the output of nodes in the first layer with the node inputs in the second layer. Alternatively, or in addition, the pooling parameters specify how and where the neural network pools data in the data processing layer. For example, with a pooling parameter that instructs "max pooling", the neural network is configured to pool by selecting the maximum value from a group of data generated by nodes in the first layer and using this maximum value as the input to a single node in the second layer. With a pooling parameter that instructs "average pooling", the neural network is configured to generate an average value from a group of data generated by nodes in the first layer and use this average value as the input to a single node in the second layer.

[0047] Kernel parameters indicate the filter size (e.g., width and height) used to process input data. Alternatively, or in addition, kernel parameters specify the type of kernel method used for filtering and processing input data. Support vector machines, for example, correspond to kernel methods that use regression analysis to identify and / or classify data. Other types of kernel methods include Gaussian processes, canonical correlation analysis, spectral clustering methods, and the like. Thus, kernel parameters can indicate the filter size and / or the type of kernel method to be applied in a neural network. Weight parameters specify the weights and biases used to classify input data by an algorithm inside a node. In some implementations, the weights and biases are learned parameter configurations, such as parameter configurations generated from training data. Layer parameters specify layer connections and / or layer types, such as a fully connected layer type that indicates connecting all nodes of a first layer (e.g., output layer 506) to all nodes of a second layer (e.g., hidden layer 508), a partially connected layer type that indicates nodes of the first layer to be disconnected from the second layer, and an activation layer type that indicates filters and / or layers to be activated inside a neural network. Alternatively, or in addition, layer parameters specify the type of node layer, such as a normalization layer type, a convolutional layer type, a pooling layer type, and the like.

[0048] Although described in the context of pooling parameters, kernel parameters, weight parameters, and layer parameters, it will be understood that other parameter configurations may be used to form a DNN consistent with the teachings provided herein. Thus, a neural network architecture configuration can include any suitable type of configuration parameter applicable to a DNN that affects how the DNN processes input data to generate output data.

[0049] In some embodiments, the configuration of the ML module 500 is further based on the current operating environment. As an example, consider an ML module trained to generate binary data from digital samples of signals. The RF signal propagation environment often changes the characteristics of signals moving through the physical environment. The RF signal propagation environment often varies and affects how the environment changes the signals. The first RF signal propagation environment changes the signal in, for example, a first way, and the second RF signal propagation environment changes the signal in a way different from the first way. These differences affect the accuracy of the output results generated by the ML module 500. For example, a DNN 502 configured to process communications transmitted in a first RF signal propagation environment may produce errors or limit performance when processing communications transmitted in a second RF signal propagation environment. Certain sensors in the sensor set of the components implementing the DNN 502 may supply sensor data representing one or more aspects of the current RF signal propagation environment. The foregoing examples may include a lidar sensor, a radar sensor, or other object detection sensors for determining the presence or absence of interfering objects inside the LOS propagation path, a UI sensor for determining the presence and / or position of the user's body relative to the components, and the like. However, it will be understood that effective specific sensor capabilities may depend on a particular device 102 or infrastructure component 108 specific to the device. For example, BS 108-1 may have lidar or radar capabilities and thus the ability to detect nearby objects, while WiFi AP 108-2 may not have lidar or radar capabilities. As another example, a smartphone (one embodiment of device 102) may have a light sensor that can be used to sense whether the smartphone is in the user's pocket or bag, while a laptop computer (another embodiment of device 102) may not have this ability. Therefore, in some embodiments, the specific configuration implemented in the ML module 500 may at least partially depend on the specific sensor configuration of the device implementing the ML module 500.

[0050] The configuration of the ML module 500 may also be based on the capabilities of the node implementing the ML module 500, the capabilities of one or more nodes upstream or downstream of the node implementing the ML module 500, or a combination thereof. For example, in an implementation form of video streaming, the display panel implemented in the data sink device 102-2 may have any of various different capability parameters such as resolution, frame rate, color gamut, etc., and thus, the ML module 500 in each of the data source device 102-1 and the data sink device 102-2 may be separately trained for some or all of the changes in some or all of these capability parameters. As another example, the battery power of the data source device 102-1 may be limited, and thus, the ML modules 500 of both the data source device 102-1 and the data sink device 102-2 may be trained based on the battery power as an input to, for example, facilitate the adoption of a data encoding / channel encoding (or channel modulation) scheme that is more suitable for low power consumption by the ML modules 500 at both ends.

[0051] Accordingly, in some embodiments, the node implementing the ML module 500 generates and stores various neural network architecture configurations for various combinations such as capability parameters, RF environment parameters, etc. For example, the device may have one or more neural network architecture configurations to use when the device's imaging camera is available, and the data sink device 102-2 may utilize a 5-channel audio system and may have a separate set of one or more neural network architecture configurations to use when the device's imaging camera is not available, and the data sink device 102-2 may utilize a 2-channel audio system.

[0052] For this purpose, the management component 136 trains the ML module 500 of the neural network chain 116 using any combination of neural network management modules and training modules at each node in the chain 116. The training may be performed offline when no active communication exchange is occurring and may be performed online during an active communication exchange. For example, the management component 136 may mathematically generate training data and access a file storing the training data to obtain real communication data. The management component 136 then extracts and stores various learned neural network architecture configurations for future use. Some implementations store input characteristics along with each neural network architecture configuration, whereby the input characteristics describe one or both of the characteristics of the RF signal propagation environment and the ability configuration corresponding to each neural network architecture configuration. In an implementation, the neural network manager selects a neural network architecture configuration by aligning the current RF signal propagation environment and the current operating environment with the input characteristics in a situation where the current operating environment includes indications of the capabilities of one or more nodes along the chain 116, such as sensor capabilities, RF capabilities, streaming accessory capabilities, processing capabilities, etc.

[0053] As described above, network devices that communicate wirelessly, such as device 102 and infrastructure component 108 for corresponding devices, can be configured to process wireless communication exchanges using one or more DNNs in each networked device, where each DNN replaces one or more functions (e.g., uplink processing, downlink processing, uplink encoding processing, downlink decoding processing, etc.) that have been conventionally implemented by one or more hard-coded blocks or fixed design blocks. Additionally, each DNN can further incorporate current sensor data from one or more sensors in the sensor set of the networked device and / or capability data from some or all of the nodes along chain 116 in order to substantially change or adapt its operation according to the current operating environment.

[0054] For this purpose, FIG. 6 shows an exemplary operating environment 600 of the implementation form of the DNN in the end-to-end neural network chain 116 between the data source device 102-1 and the data sink device 102-2. In the example shown, the neural network management module 222 of the data source device 102-1 implements a source transmission side (TX) processing module 602, and the neural network management module 222 of the data sink device 102-2 implements a sink reception side (RX) processing module 604. In the neural network chain between the source and the sink, the infrastructure component 108-1 directed towards the source implements an RX processing module 606 directed towards the source, and the infrastructure component 108-2 directed towards the sink implements a TX processing module 604 directed towards the sink. In some embodiments, the transmission path from the infrastructure component 108-1 directed towards the source to the infrastructure component 108-2 directed towards the sink may utilize conventional processing and signal transmission technologies, and thus the corresponding ML processing module may be unnecessary. However, in other embodiments, one or more of the transmission links between the infrastructure component 108 directed towards the device and the corresponding CN110, or between each CN110, may utilize ML processing modules on the TX side and the RX side of the transmission link. Therefore, the infrastructure component 108-1 directed towards the source may further implement a TX processing module (not shown for ease of explanation) for transmission to the CN1 110-1, and the CN1 110-1 may implement an RX processing module for receiving the transmission from the TX processing module of the infrastructure component 108-1 directed towards the source and a TX processing module for the corresponding transmission to the CN2 110-2 through one or more networks 112 (one or both of these processing modules are collectively represented by the CN1 processing module 610 in FIG. 6).Similarly, CN2 110-2 may implement an RX processing module for receiving transmissions from CN1 110-1 and a TX processing module for transmitting to infrastructure component 108-2 towards the sink (one or both of these processing modules being collectively represented by CN1 processing module 612), and also, infrastructure component 108-2 towards the sink may implement an RX processing module (not shown for ease of explanation) for receiving and processing transmissions from CN2 110-2. In at least one embodiment, each of these processing modules implements one or more DNNs by a corresponding ML module implementation such as those described above with reference to one or more DNNs 502 of the ML module 500 of FIG. 5.

[0055] The source TX processing module 602 of the data source device 102-1 and the source RX processing module 606 of the infrastructure component 108-1 directed towards the source interact with each other to support the wireless communication path 614 between the data source device 102-1 and the infrastructure component 108-1 directed towards the source. One or more conventional communication components among the corresponding processing modules and / or the infrastructure component 108-1 directed towards the source, CN1 110-1, CN2 110-2, and the infrastructure component 108-2 directed towards the sink similarly interact to support a series of communication paths between the infrastructure component 108-1 directed towards the source and the infrastructure component 108-2 directed towards the sink. Also, the TX processing module 608 directed towards the sink of the infrastructure component 108-2 directed towards the sink and the sink RX processing module 604 of the data sink device 102-2 interact with each other to support the wireless communication path 622 between the infrastructure component 108-2 directed towards the sink and the data sink device 102-2. Therefore, the series of communication paths 614, 616, 618, 620, and 622 represent the transmission paths for communicating signals representing the data of the data stream from the data source module 122 to the data sink module 126.

[0056] One or more DNNs of the source TX processing module 602 receive, as inputs, the transmitted data block 624 of the data stream generated by the data source module 122, as well as other inputs such as the sensor data 626 from the sensor set 210 and the transmit-side RF state information 628 representing the current parameters of the transmit side of the RF antenna interface 204, and are trained to generate corresponding outputs from these inputs for transmission as RF signals via the RF analog stage of the RF antenna interface 204 of the data source device 102-1. Specifically, in some embodiments, one or more DNNs of the source TX processing module 602 are trained to provide a process that substantially results in a data-encoded (e.g., compressed) and channel-encoded (including modulation and error codes or redundancy) representation of the data block 624 that is ready for digital-to-analog conversion and RF transmission. That is, rather than employing individual discrete processing blocks to perform the data encoding process followed by the initial RF encoding process, the TX processing module 602 simultaneously provides the equivalent of such processes and is trained to generate corresponding signals that are source-encoded and channel-encoded and substantially ready for RF transmission, based at least in part on other current data such as the sensor data 626 and the transmit RF state information 628.

[0057] The output generated by the source TX processing module 602 is wirelessly transmitted by the data source device 102-1 and is first processed by the RF antenna interface 304 of the infrastructure component 108-1 directed towards the source. One or more DNNs of the RX processing module 606 directed towards the source receive, as inputs, the output of the RF antenna interface 304, along with one or more other inputs such as sensor data 630 from the sensor set 310 and receive-side RF state information 632 representing the current parameters on the receiving side of the RF antenna interface 304, and are trained to generate corresponding outputs for transmission to the CN1 110-1 from these inputs. The processing performed by the RX processing module 606 directed towards the source may include, for example, steps of channel decoding the input signal to generate a digital representation of the data-encoded version of the transmitted data block 624. Additionally, in other embodiments, the processing may include steps of decoding the data itself to generate a decoded representation of the data block 624, which representation may later be re-encoded for downstream transmission.

[0058] Next, the output of the results of the RX processing module 606 directed towards the source can be transmitted from the infrastructure component 108-1 directed towards the source, through the CN1 110-1, CN2 110-2 and the corresponding communication paths 616, 618, and 620, to the infrastructure device 108-2 directed towards the sink. In some embodiments, these paths are implemented by conventional transmission techniques such as the packetization of the output into IP-based packets and the transmission of the resulting IP-based packets using conventional IP technology. In other embodiments, some or all of these paths are supported by a TX processing module employing one or more DNNs on the transmission side of the communication path and a corresponding RX processing module employing one or more DNNs on the receiving side of the communication path. For example, the CN1 processing module 610 of the CN1 110-1 can represent a TX processing module having one or more DNNs, and the DNN is trained to receive, as an input, a representation of a signal representing the data block 624 received at the CN1 110-2, along with other inputs such as the network state information 634 of the network 112 and other priority information 636 indicating QoS, QoE, or the priority of the data stream, QoS, or QoE parameters, and to generate a corresponding output from these inputs. This output is then packetized and transmitted through the network 112 to the CN2 110-2, where one or more DNNs included in the RX processing module implemented as the CN2 processing module 612 at the CN2 110-2 receive this output as an input, along with other inputs such as the network state information 638 of the network 112 and the priority information 636, and generate an output that can be further transmitted downstream in the chain of ML processing modules.

[0059] In the infrastructure component 108-2 towards the sink, one or more DNNs of the TX processing module 608 towards the sink receive, as inputs, the representation of the data block 624 received from the upstream node, as well as one or more other inputs such as the sensor data 640 from the sensor set 310 and the transmission-side RF state information 642 representing the current parameters on the transmission side of the RF antenna interface 304 of the infrastructure component 108-2. From these inputs, they are configured to generate corresponding outputs for transmission as RF signals via the RF analog stage of the RF antenna interface 304 of the infrastructure component 108-2 towards the sink. Specifically, in some embodiments, one or more DNNs of the TX processing module 608 towards the sink are trained to provide a process that substantially results in a channel-encoded (including modulation) representation of the data block 624 that is ready for digital-to-analog conversion and RF transmission. Moreover, in some embodiments, one or more DNNs are also trained to substantially simultaneously result in a data encoding of the representation of the data block 624, which may be in the same format as the data encoding employed by the source TX processing module 602 or in a different or complementary format.

[0060] The output generated by the TX processing module 608 towards the sink is wirelessly transmitted by the infrastructure component 108-2 towards the sink and is first processed by the RF antenna interface 204 of the data sink device 102-2. One or more DNNs of the sink RX processing module 604 receive, as input, the resultant output from the RF antenna interface 204, along with one or more other inputs such as sensor data 644 from the sensor set 210 of the data sink device 102-2 or receive-side RF state information 648 representing the current parameters on the receiving side of the RF antenna interface 204, and are trained to generate an incoming data block 646 representing the original transmitted data block 624 generated at the data source device 102-1 from these inputs. Specifically, one or more DNNs are trained to provide a process that substantially effects channel decoding (including demodulation) of the output from the analog-to-digital stage of the RF antenna interface 204 and destination decoding (e.g., data decompression) of the channel-decoded result to obtain a channel-decoded and data-decoded representation of the original data of the data block 624. That is, rather than employing individual discrete processing blocks to perform channel decoding processing followed by data decoding processing, the sink RX processing module 604 simultaneously provides an equivalent of such processing and is trained to generate, as output, a recovered version of the transmitted data block 624 based at least in part on other current data such as sensor data 644 and receive-side RF state information 648. This recovery process can be lossless such that the recovered version is an exact replica of the original data, depending on the implementation and training. In other implementations, the DNNs along the chain may be trained to employ loss processing, for efficiency or to reduce complexity, and thus the recovered version of the original data block, represented by the incoming data block 646, may be a lossy version of the data of the transmitted data block 624. In any case, the incoming data block 646 can then be supplied to the data sink module 126 for consumption or further processing.

[0061] Further, as described herein, in some embodiments, the management component 136 uses the current capabilities of one or more of the device 102, infrastructure component 108, CN 110, or other nodes in the transmission path as a criterion for selecting a particular DNN architecture configuration employed by a given processing module at a corresponding node. For example, based on the resolution capabilities of the display panel of the data sink device 102-2, the management component 136 may be driven to select a DNN for the source TX processing module 602 and a DNN for the sink RX processing module 604, where the data encoding and decoding processes are specially trained for the corresponding resolution capabilities. However, in other embodiments, one or more of the DNNs along the transmission path do not use the current capabilities to select the DNN architecture configuration, or in addition thereto, may utilize the capability information of one or more nodes (collectively identified as capability data 650 in FIG. 6) as an input for controlling the operation of the DNN itself. For example, capability data indicating a low battery level supplied by the data source device 102-1 may be input to the source TX processing module 602, and thus, in the DNN(s) of the source TX processing module 602, it may be specified to prepare a data encoding format that consumes less power when generating the resulting output. On the other hand, when this same data is input to the sink RX processing module 604, in the DNN(s) of the sink RX processing module 604, it is specified to prepare a complementary data decoding process for recovering the original data.

[0062] The implementation form of a DNN or other neural network that is trained together for some or all of the nodes in the transmission path between the data source device 102-1 and the data sink device 102-2 provides flexibility in design and facilitates efficient updates compared to the conventional block-by-block design and testing methods. At the same time, various nodes in the transmission path can also quickly adapt the processing of outgoing and incoming signals to the current operating parameters. However, before a DNN can be deployed and operate, it is usually trained or configured to provide appropriate outputs for a given set of one or more inputs. For this purpose, FIG. 7 shows an exemplary method 700 for growing one or more jointly trained DNN architecture configurations as options for various operating environments for various nodes along the chain or transmission path between devices 102, according to some embodiments. It should be noted that the order of operations described with reference to FIG. 7 is for illustrative purposes only, and the operations may be executed in a different order. Furthermore, in the illustrated method, one or more operations may be omitted, or one or more additional operations may be included. Although FIG. 7 shows an offline training method using one or more test nodes, it should be further noted that a similar method can also be implemented for online training using one or more nodes during active operation.

[0063] As described above, the operation of the DNN employed in some or all of the devices 102, infrastructure components 108, CN 110, or other nodes in the DNN chain (e.g., chain 116 in FIG. 1) for the data stream transmission path can be based on specific capabilities and the current operating parameters of the nodes employing the corresponding DNNs of one or more upstream or downstream nodes, or a combination thereof. These capability parameters and operating parameters can include, for example, the type of sensors used to sense the RF transmission environment of the node, the capabilities of such sensors, the power capacity of one or more nodes, the availability status of one or more accessories used to generate, consume, or process data from the data stream, and the like. Since the DNN utilizes such information to define its operation, in many cases, the specific DNN configuration implemented at one of the nodes is based on the specific capabilities and operating parameters currently employed at that node or at upstream or downstream nodes, that is, the specific DNN configuration implemented reflects the capability information and current operating parameters currently indicated by one or more nodes in the data stream transmission path.

[0064] Accordingly, method 700 begins with block 702 that determines the expected capabilities (including expected operating parameters or parameter ranges) of one or more test nodes of the test transmission path, where the test nodes will include a test data source device, a test data sink device, infrastructure components towards the source, infrastructure components towards the sink (which may be the same components as the infrastructure components towards the source), and one or more core networks that connect the infrastructure components towards the source and the infrastructure components towards the sink (when a DNN or other NN is implemented in the core network(s) for transmitting the corresponding data stream). It is assumed that for the following, the training module 412 of the management component 136 manages joint training, and thus the capability information regarding the nodes in the DNN chain is known to the training module 412 (e.g., by a database storing this information or other locally stored data structures). However, since the management component 136 likely has no prior knowledge of the capabilities of any given UE, the test source device and the test sink device supply to the management component 136 an indication of their respective capabilities such as an indication of the types of sensors available in the test device, an indication of various parameters of these sensors (e.g., the imaging resolution and image data format of an imaging camera, the satellite positioning type and format of a satellite-based position sensor, etc.), the accessories available in the device and applicable parameters (e.g., the number of voice channels). For example, the test device can supply this indication of capabilities as part of a UECapabilityInformation radio resource control (RRC) message supplied by the UE, usually in response to a UECapabilityEnquiry RRC message transmitted by the BS in accordance with at least the 4G LTE standard and the 5G NR standard. Alternatively, the test UE can supply an indication of sensor capabilities as communication on an individual side channel or control channel.Furthermore, in some embodiments, the capabilities of the test device can be stored in a local or remote database available to the management component 136, and thus, the management component 136 can query this database based on some form of the identifier of the test device, such as the International Mobile Subscriber Identity (IMSI) value associated with the test device.

[0065] In block 704, the training module 412 selects a specific ability configuration for training the DNNs of the DNN chain using the capabilities of the applicable nodes identified in the DNN chain. In some embodiments, the training module 412 may attempt to train all permutations of the available capabilities. However, in implementations where nodes are likely to have a relatively large number of diverse capabilities, this effort may be infeasible. Thus, in other embodiments, the training module 412 selects from only a representative set of limited, possible ability configurations. As an example, lidar information from various lidar modules manufactured by the same company may be relatively invariant, and thus, for example, if the source / sink device can implement any of a plurality of lidar sensors from its manufacturer, the training module 412 may choose to omit some lidar sensors from the sensor configuration being trained. As another example, a data sink device may have a headset accessory with any of a variety of audio sample rate capabilities, but the training module 412 may choose to eliminate all but the few most frequently used audio sample rate options. Additionally, the capabilities of certain components, such as the core network 110, may not be particularly relevant to effective training, and thus, the training module 412 may ignore the reported capabilities of the test core network in the DNN chain or may assume a single default or minimum set of capabilities for these particular nodes. In yet other embodiments, there may be a defined set of chain ability configurations that the training module 412 can select for training, and thus, the training module 412 selects a chain ability configuration from this defined set (and also avoids selecting chain ability configurations that rely on capabilities not commonly supported by the associated nodes).

[0066] When a chain ability configuration for training is selected, at block 706, training module 412 identifies one or more sets of training data to use when training the DNNs of the DNN chain together based on the selected chain ability configuration. That is, the one or more sets of training data include or represent data that can be supplied as input to the corresponding DNNs in online operation, and thus data suitable for training the DNNs. As an example, this training data can include a stream of test data blocks of a data stream (e.g., video data blocks of a test video stream, audio data blocks of a test audio stream, measurement data blocks of a test measurement data stream, etc.), test sensor data that matches the sensors included in the ability configuration under test, test RX or TX state information, ability parameters of accessories or other components of the test source device / test sink device (e.g., resolution, sample rate, color gamut, parameter range, data format, etc.).

[0067] When one or more training sets are obtained, at block 708, training module 412 initiates joint training of each DNN of the DNN chain. This joint training typically includes steps of initializing the bias weights and coefficients of the various DNNs with generally pseudo-randomly selected initial values until the output from the last DNN in the chain (i.e., the output from the RX processing module of the test sink device (e.g., sink RX processing module 604)) is obtained, then inputting a set of training data to the TX processing module of the test source device (e.g., source TX processing module 602), wirelessly transmitting the resulting output as a transmission to the RX processing module of the infrastructure component directed towards the source of the test (e.g., source-directed RX processing module 606), and transmitting the resulting output to the next RX DNN in the DNN chain, etc.

[0068] As is frequently employed for the training of DNNs, the feedback obtained as the actual result output at the sink end of the DNN chain is used to change or improve the parameters of one or more DNNs in the chain, such as by backpropagation. Thus, in block 710, the management component 136 and / or the DNN chain itself obtains feedback on the transmitted training set. This feedback can be implemented in any of various forms or combinations of forms. In some embodiments, the feedback includes objective feedback such as the training module 412 or other training modules determining the error between the actual result output and the expected result output and backpropagating this error across the DNNs of the DNN chain. The objective feedback obtained can also include evaluation metrics of some aspects of the signal as the signal passes through one or more links in the DNN chain. For example, in relation to the RF aspect of signal transmission, the objective feedback can include metrics such as the block error rate (BER), signal-to-noise ratio (SNR), signal-to-interference-plus-noise ratio (SINR), etc. The objective feedback can also include objective quality evaluation metrics regarding the quality of the data content itself. For example, in the context of video streaming where the processing by the DNN chain includes the form of lossy compression, the training data set can include one or more video frames, and thus, the objective feedback of the training data set can include the peak SNR (PSNR) value based on the comparison between the video frames recovered at the test sink device and the original corresponding image frames in the training data set.

[0069] In some embodiments, the feedback obtained from repeatedly training a DNN chain using a training data set can include subjective user feedback when the streaming data is presented to the user or "consumed" by the user in some form. For example, in the context of an exemplary test video stream, the user can provide subjective feedback indicating the perceived quality of the video content presented to the test sink device. This feedback can be obtained through explicit queries such as presenting one or more requests to the user to rate some quality aspect of the test video content presented to the test sink device. This feedback can also be obtained indirectly by observing the user's interaction with the playback of the test video content, such as observing changes made by the user, such as contrast settings, playback resolution settings, etc. At block 712, the objective feedback and / or subjective feedback obtained as a result of transmitting the test data set through the DNN chain, and the presentation or other consumption of the output of the result at the test sink device, are then used to update various aspects of one or more of the DNNs of the DNN chain by, for example, backpropagation of the error to change the weights, connections, or corresponding layers of the DNN, or changes managed by the management component 136 in response to such feedback. Then, for the next set of training data selected in the next iteration of block 706, the training process of blocks 706 - 712 is executed and repeated until a certain number of training iterations are performed or until a certain minimum error rate is achieved.

[0070] As a result of the joint training (or individual training) of the neural networks along the neural network chain between the test source device and the test sink device, each neural network obtains a specific neural network architecture configuration, or in the case where the neural network being implemented is a DNN, obtains a DNN architecture configuration, and the DNN architecture configuration characterizes the architecture and parameters of the corresponding DNN, such as the number of hidden layers, the number of nodes in each layer, the connections between each layer, weights, coefficients, and other bias values implemented at each node. Thus, when the joint training or individual training of the DNNs of the DNN chain for the selected chain configuration is completed, in block 714, some or all of the trained DNN configurations are distributed to the nodes of system 100, and each node stores the resulting DNN configuration of the corresponding DNN as the DNN architecture configuration. In at least one embodiment, the DNN architecture configuration can be generated by extracting the architecture and parameters of the corresponding DNN, such as the number of hidden layers, the number of nodes, connections, coefficients, weights, and other bias values, at the end of the joint training.

[0071] If one or more other chain configurations remain to be trained, method 700 returns to block 704 to select the next chain configuration to be trained together, and another iteration of the sub-processing of blocks 704 - 714 is repeated for the next chain configuration selected by training module 412. Otherwise, if the DNNs of the DNN chain have been jointly trained for all the intended chain configurations, method 700 is complete, and system 100 can move on to support RF-based transmission of data streamed between devices 102 using the trained DNNs, as described below with reference to FIGS. 8 - 10.

[0072] As described above, the joint training process can be performed using an offline test node (i.e., while there is no active communication of control information or user plane data), or while the actual nodes of the intended transmission path are online (i.e., while active communication of control information or user plane data is occurring). Further, in some embodiments, rather than training all of the DNNs together, in some cases, a subset of the DNNs can be trained or retrained while other DNNs are kept unchanged. As an example, the neural network manager 410 may detect that a particular processing module is operating inefficiently or inaccurately, for example, due to the presence of an undetected interference source in proximity to the node implementing the processing module, or in response to a previously unreported loss of an accessory capability. Thus, the neural network manager 410 may schedule an individual retraining of the DNN(s) of that processing module while keeping the other DNNs of the other processing modules of the node in their current configuration.

[0073] Furthermore, while there may be various nodes supporting a number of capability configurations, it will be appreciated that a wide variety of nodes may support the same or similar capability configurations. Thus, it is not necessary to repeat the joint training for all nodes incorporated in the transmission path of a data stream. After the joint training of a representative test node, the test node may transmit to the management component 136 a representation of its trained DNN architecture configuration with respect to the capability configuration, and the management component 136 may store this DNN architecture configuration and later transmit it to other nodes supporting the same or similar capability configurations for implementation in the DNNs of the transmission path.

[0074] Furthermore, the DNN architecture configuration may often change over time when the corresponding nodes are operating using that DNN. Thus, as operation proceeds, the neural network management module of a given node may be configured to send to the management component 136 a representation of one or more updated architecture configurations of the DNN employed by that node, such as by supplying updated gradients and associated information in response to a trigger. This trigger may be, for example, the expiration of a periodic timer, a query from the management component 136, a determination that the magnitude of the change exceeds a specified threshold. The management component 136 then incorporates these received DNN updates into the corresponding DNN architecture configuration, thus obtaining an updated DNN architecture configuration that can be used to distribute to the nodes in the transmission path as needed.

[0075] FIG. 8 shows an exemplary method 800 for transmitting a data stream from a data source device 102-1 to a data sink device 102-2 using an end-to-end DNN chain that reaches from the data source device 102-1 to the data sink device along a transmission path by a network infrastructure 104, according to some embodiments. Method 800 begins at block 802, where the data source device 102-1 signals the start of a data stream for transmission to the data sink device 102-2. In some embodiments, in a mobile communication implementation that relies on an IMS service or other application server in the network infrastructure 104 connecting the two devices 102, the start of the data stream may be instructed by the device 102 or application server registering for the IMS service to prepare the IMS service or application to provide appropriate support for the data stream.

[0076] In response to an instruction to send a data stream, the management component 136 identifies the transmission path intended for the data stream and identifies the nodes along the transmission path. Once the nodes are so identified, at block 804, the management component 136 selects a DNN architecture configuration to be implemented by some or all of the DNNs along the DNN chain formed by the processing modules of the nodes in the transmission path, and sends configuration commands to each affected node to instruct the node to implement the selected DNN architecture configuration for its corresponding DNN. In other embodiments, some or all of the nodes are configured to select their own DNN architecture configurations without relying on the management component 136. In either approach, the particular DNN architecture configuration selected for implementation in the DNNs in the DNN chain can be determined using any of a variety of techniques. As represented by block 805, each DNN may have a default DNN architecture configuration, which may be associated with the type of data being streamed or other characteristics of the data stream, based on, for example, the RAT or other interface type supported by the DNN. For example, each node may have a default DNN that is first implemented for some type of voice data stream. Alternatively, as represented at block 807, the DNN architecture configuration implemented for the DNNs in the DNN chain may be based on the reported capabilities of the nodes implementing the DNNs, the reported capabilities of one or more other nodes in the transmission path, or a combination thereof.For example, data source device 102-1 may report an ability to support 5G NR RAT of millimeter wave (mmWave) and an ability to generate a video stream with a pixel depth of 12 bits, and data sink device 102- may report an ability to support Sub-6 5G NR RAT and an ability to display video content with a pixel depth of 12 bits. Thus, management component 136 instructs data source device 102-1 to implement a specific DNN architecture configuration for source TX processing module 602 trained for channel encoding for mmWave RAT transmission and data encoding for a pixel depth of 12 bits, and instructs source-directed RX processing module 606 of source-directed infrastructure component 108-1 to implement a specific DNN architecture configuration trained to perform channel decoding for mmWave RAT transmission, instructs sink-directed TX processing module 608 of sink-directed infrastructure component 108-2 to implement a DNN architecture configuration trained to perform channel encoding for Sub-6 RAT transmission, and instructs sink RX processing module 604 to implement a DNN architecture configuration trained to perform channel decoding for Sub-6 RAT transmission and data decoding for a pixel depth of 12 bits. Additionally, as represented in block 809, first, based on one or more preferences indicated by a user or a user device, such as a preference indicated by the user regarding power savings for performance, a preference indicated by the user regarding a specific capability parameter (such as image resolution or frame rate), a specific DNN architecture configuration implemented by one or more DNNs may be selected.

[0077] Once the DNN of the DNN chain is configured for the first time, data streaming can begin. Thus, in block 806, the data source module 122 generates an outgoing data block for data streaming (e.g., the outgoing data block 624 of FIG. 6). In block 808, the source TX processing module 602 receives the data block as an input, along with sensor data from the sensors of the data source device 102-1, capability data regarding the capabilities of one or more nodes, and one or more other inputs such as the current transmission parameters of the RF antenna interface 204 of the data source device 102-1, and from these inputs, generates an output signal representing a source-encoded (compressed) and channel-encoded representation of the outgoing data block suitable for digital-to-analog conversion and RF transmission by the RF antenna interface 204 to the source infrastructure component 108-1 of the network infrastructure 104.

[0078] In block 810, this output signal is received as an input, along with sensor data, one or more other inputs such as the current RX parameters, by the source-directed RX processing module 606 of the source infrastructure component 108-1, and one or more DNNs of the source-directed RX processing module 606 generate a corresponding output signal based on these inputs. Then, the information of this output signal is processed by some subsequent DNN in the DNN chain at a node of the network infrastructure 104 as described above. On the other side of the network infrastructure 104, the signal representing this information is input to the sink-directed TX processing module 608 along with sensor data, current TX parameters, capability information, etc., and one or more DNNs of the sink-directed TX processing module 608 process these inputs to generate an output signal that is channel-encoded in a manner suitable for digital-to-analog conversion and RF transmission to the data sink device 102-2, which is herein referred to as the "infrastructure output signal".

[0079] In block 812, the RF antenna interface 204 of the data sink device 102-2 receives an RF transmission, performs initial processing to generate a digital output representing an infrastructure output signal, and this digital output is supplied as an input to the sink RX processing module 604, along with other inputs such as sensor data from a sensor of the data sink device 102-2, current RX parameters, and capability information. One or more DNNs of the sink RX processing module 604 efficiently process these inputs to channel decode the signal and destination decode the underlying data to generate an incoming data block (e.g., the incoming data block 646 of FIG. 6) representing a recovered version of the data of the transmitted data block generated in block 806. The incoming data block is then supplied to the data sink module 126 for further processing.

[0080] The processing of blocks 806-812 can be repeated for each transmitted data block generated by data source module 122 to be included in the transmitted data stream. During this repeated streaming process, at block 814, data sink device 102-2 can supply feedback regarding received data or a received signal representing received data. As described above, this feedback can be supplied as objective feedback, in the form of, for example, BER, SNR, PSNR, or other quantifiable error parameters. Additionally, or alternatively, this feedback can include subjective feedback obtained from a user, such as an explicit solicitation of user feedback (such as by a query regarding the user's QoE perception), an observation of the user's control or interaction with the content of the data stream, or a presentation thereof. At block 816, this feedback is used by management component 136 or other infrastructure components to update the DNN architecture configuration of one or more DNNs in the chain, such as by backpropagation to change aspects of the currently used DNN architecture configuration, activating management component 136 to switch one DNN architecture configuration to another, or a combination thereof.

[0081] Furthermore, during the streaming process, the capabilities of one or more nodes in the transmission path may change in a way that affects the processing performed by the end-to-end DNN chain. For example, a video stream may start at a data sink device 102-1 connected to an external display panel, and thus, the generation of video data by the DNN chain and subsequent processing thereof may be configured based on the capabilities of the external display panel. However, during the streaming process, the user may switch to using the built-in display panel of the data sink device 102-2, and changes in one or more of the DNNs may be required to adapt to this change. Thus, when the capabilities of the nodes change (as represented in block 818), the management component 136 or the affected nodes themselves update one or more DNNs affected by the change in capabilities, select a replacement DNN architecture configuration for the affected DNNs, or a combination thereof, in block 820, to adapt to the change in capabilities.

[0082] Next, turning to FIGS. 9 and 10, a (ladder-shaped) transaction diagram is shown representing an example of the configuration and operation of an end-to-end DNN chain for the encoded wireless transmission of a data stream, according to at least one embodiment. For ease of understanding, the processes represented in FIGS. 9 and 10 are described in the exemplary scenario of FIG. 1, where the infrastructure component 108-1 towards the source is a mobile communication BS and the infrastructure component 108-2 towards the sink is a WiFi AP. The exemplary scenario used herein further utilizes a video stream as the data stream. However, it will be understood that the principles described are equally applicable to other types of data streams and to other combinations of infrastructure components directed to devices, such as all BS configurations, all WiFi AP configurations, a WiFi AP as the infrastructure component towards the source and a mobile communication BS as the infrastructure component towards the sink, etc.

[0083] FIG. 9 shows a transaction 900 for initially configuring a DNN chain to implement a particular DNN architecture configuration trained according to a selected chain configuration. In the example shown, the management component 136 is responsible for selecting the DNN architecture configuration for the DNNs of the end-to-end DNN chain. Thus, the process begins with each node in the DNN reporting its capabilities to the management component 136. This includes transmitting capability messages 901, 902, 903, 904, 905, and 906 from CN1 110-1, CN2 110-2, BS108-1, AP108-2, data source device 102-1, and data sink device 102-2, respectively. The capability information provided by BS108-1, AP108-2, and devices 102-1, 102-2 may include, for example, RAT capabilities, power capabilities, processing capabilities, etc. For devices 102-1, 102-2, the capability information may further include accessory capability information such as the presence of a camera used to capture video content and its parameters in the video stream at the data source device 102-1, the presence of an HMD used to display video content at the data sink device 102-2, etc.

[0084] The management component 136 identifies the DNN architecture configuration to be adopted in one or more DNNs of each node based on the received capability information and other considerations such as the indicated preferences and predetermined default settings, and sends a configuration message to each of the nodes CN1 110-1, CN2 110-2, BS108-1, AP108-2, data source device 102-1, and data sink device 102-2, which identifies or indicates the DNN architecture configuration (s) to be adopted by that node. In response to receiving the configuration message, the corresponding node implements the identified DNN architecture configuration in its corresponding DNN. At this point, the end-to-end DNN chain in the transmission path is initialized and ready to start the data streaming process.

[0085] Transaction diagram 1000 of FIG. 10 shows this data streaming process from the configuration of FIG. 900 of FIG. 9. This streaming process begins with data source module 122 generating a first data block 1001 (block 1) of video content. Block 1 is processed by source TX processing module 602 to generate an output signal 1002 (output 1), which is the source-encoded and channel-encoded representation of the first data block 1001. Then, output 1 is wirelessly transmitted to BS108-1. One or more processing modules of BS108-1 use this output signal 1002 as an input to generate a corresponding output signal 1003 (output 1-1) that is transmitted to CN1 110-1. One or more processing modules of CN1 110-1 process output signal 1003 to generate an output signal 1004 (output 1-2) that is transmitted to CN2 110-2 through one or more networks 112. One or more processing modules of CN2 110-2 process output signal 1004 to generate a corresponding output signal 1005 (output 1-3) that is transmitted to AP108-2. And also, one or more processing modules of AP108-2 process output signal 1005 to generate an output signal 1006 (output 1-4) that is wirelessly transmitted to data sink device 102-2. At data sink device 102-2, sink RX processing module 604 receives output signal 1006 as an input and may further receive as inputs sensor data, current RX parameters, capability information, and other data. From these inputs, it generates an output data block 1007 that represents the data-decoded and channel-decoded version of the data represented in output signal 1006, and thus represents the recovered representation of the transmitted data block 1001 generated by data source module 122. This recovered data block 1007 is then supplied to data sink module 126, which processes the data to present the corresponding video content to the user by means of a display panel.In this example, the user has not yet provided subjective feedback, and thus, at this point, the feedback provided to the management component 136 is only the objective feedback 1108 (feedback 1) that represents data such as the SNR value or the average contrast value, or a quantifiable aspect of the signal used to convey this data.

[0086] This process is repeated for the transmission of the next transmitted data block 1102 (block 2) generated by the data source module 122, resulting in the recovered data block 1017 output by the sink RX processing module 604 and used by the data sink module 126 when displaying the corresponding video content to the user. The data sink device 102-2 can supply the management component 136 with objective feedback 1018 regarding the recovered data block 1017, similar to the recovered data block 1007. Further, at this point, the user has viewed a sufficient amount of video content and provides subjective feedback 1019 that indicates the subjective impression of the QoE of the user's streaming process up to this point.

[0087] Furthermore, following the transmission of data block 1011, the capabilities of both the data source device 102-1 and the data sink device 102-2 change. In this example, the data source device 102-1 is switched from wall outlet power to battery-only power, which is reported to the management component 136 as a capabilities update message 1020, and the data sink device 102-2 is disconnected from an external flat panel TV and thus switched to use its built-in display panel, which is reported to the management component 136 as a capabilities message 1021. In response to these reported capability changes and the objective and subjective feedback provided in response to the transmission of data blocks 1001 and 1011, the management component 136 determines that the switching of the DNN architecture configurations of both the data source device 102-1 and the data sink device 102-2 is for providing an improved QoE adapted to the capability changes. Accordingly, the management component 136 sends a DNN configuration message 1022 to the source TX processing module 602 to instruct it to switch to its newly selected DNN architecture configuration, and sends a DNN configuration message 123 to the sink RX processing module 604 to instruct it to switch to its newly selected DNN architecture configuration. When the new DNN architecture configurations are implemented at the source and sink ends of the DNN chain, the changed end-to-end DNN chain is used to transmit the next data block 1031 (block 3) generated by the data source module 122 from the data source device 102-1 through the updated DNN chain to the data sink device 102-2, and can be used to receive corresponding feedback in the same manner as described above for data blocks 1001 and 1011.

[0088] Figures 1 to 10 have been described above in the context of system 100. In system 100, there is one data sink device, and both the source end and the sink end of the transmission path involve wireless transmission. Therefore, on the source side, the DNN provides both data encoding and channel encoding, and on the sink side, the DNN provides both data decoding and channel decoding. However, the techniques described herein are not limited to these specific embodiments. As an example, in the alternative system 1100 represented by FIG. 11, the data source device 1102-1 streams data to a plurality of data sink devices, such as the two illustrated data sink devices 1102-2 and 1102-3, via the network infrastructure 1104. The network infrastructure 1104 includes a WLAN AP 1108-1 wirelessly connected to the data source device 1102-1, a mobile communication BS 1108-2 wirelessly connected to the data sink device 1102-2, and a mobile communication BS 1108-3 wirelessly connected to the data sink device 1102-3, as well as CNs 1110-1, 1110-2, 1110-3 that interconnect the AP 1108-1, BS 1108-2, and BS 1108-3 via one or more non-core networks 1112. The network infrastructure 1104 may further include an application server 1115 to support this plurality of destination streaming. In this example, system 1100 includes a plurality of transmission paths, and thus it will be understood that the data streaming process involves the use of DNNs that form an end-to-end DNN tree. However, the aforementioned techniques for the training, configuration, and usage of an end-to-end DNN chain for streaming data from a data source device to one data sink device are applicable to a variant form of the end-to-end DNN tree in which each root within the end-to-end DNN tree can be separately jointly trained using the same processing as described above, and also to the DNN architecture configuration implemented such that the DNN architecture configuration existing in either root is selected in a manner that is adaptable to either root.

[0089] Furthermore, FIG. 12 shows an exemplary remote cloud game system 100 employing the aforementioned technology according to some embodiments. In the illustrated cloud game system 1200, the cloud game server 1202-1 is connected to the user device 1202-2 through a non-core network 112 (e.g., the Internet), and the network infrastructure 1204 includes one or more core networks and a base station (not shown) wirelessly connected to the user device 1202-2. In this example, the video game application 1222 (an embodiment of the data source module 122) runs an instance of a video game and generates a video stream representing video content and an audio stream representing audio content of the gameplay of the video game application 1220, respectively. Next, the audio packets and video packets of these streams are supplied to one or more source TX DNNs 1203 of the cloud game server 1202-1, and the source TX DNNs 1203 process this input together with other inputs to generate data-encoded representations of the audio data and video data, and these representations are transmitted to the network infrastructure 1204 through the network 1212. Next, one or more infrastructure DNNs 1205 in the network infrastructure 1204 process the continuous signal representing the underlying information to finally generate channel-encoded output signals, and these output signals are wirelessly transmitted to the user device 1202-2, where the output signals are processed together with other inputs by the sink RX DNN 1207 to generate corresponding data-decoded and channel-decoded representations of the corresponding video content and audio content, and then these representations are supplied to the data sink module 1226 (e.g., a web browser) to provide the user with the audio content and video content.

[0090] Various aspects of the present disclosure may be better understood with reference to the following examples, which may be implemented individually or in various combinations. Example 1: A method implemented by a computer in a data source device, comprising: receiving, as an input to a transmission-side neural network of the data source device, a first data block of a data stream; generating, in the transmission-side neural network, a first output representing a data-encoded and channel-encoded version of the first data block based on the first data block; and controlling, based on the first output, a radio frequency (RF) antenna interface of the data source device to transmit a first RF signal representing a data-encoded and channel-encoded version of the first data block. Example 2: The method according to Example 1, further comprising, in the data source device, selecting a first neural network architecture configuration from a plurality of neural network architecture configurations based on at least one of one or more capabilities of at least one of the data source device or a data sink device configured to receive the data source device or the data stream, or at least one of user-specified preferences. The step of generating the first output includes generating, in the transmission-side neural network, the first output based on the first neural network architecture configuration of the transmission-side neural network. Example 3: The method according to Example 2, further comprising implementing, in response to a command from an infrastructure component of the network infrastructure, a first neural network architecture configuration selected from a plurality of neural network architecture configurations of the transmission-side neural network. The step of generating the first output includes generating, in the transmission-side neural network, the first output based on the first neural network architecture configuration of the transmission-side neural network. Example 4: The method according to Example 3, further comprising receiving a command from a network infrastructure component in response to providing one or more representations of one or more capabilities of at least one of a data source device or a data sink device with respect to the infrastructure component. Example 5: The method according to any one of Examples 2 to 4, further comprising changing a transmitting neural network in response to a change in the capabilities of at least one of the data source device or the data sink device, and implementing a second neural network architecture configuration; receiving a second data block of the data stream as an input to the transmitting neural network; and generating a second output using the second neural network architecture configuration based on the second data block in the transmitting neural network. The second output represents a data-encoded and channel-encoded version of the second data block. The method includes controlling the RF antenna interface of the data source device based on the second output to transmit a second RF signal representing the data-encoded and channel-encoded version of the second data block. Example 6: The method according to Example 5, further comprising receiving a command from an infrastructure component of the network infrastructure in response to providing one or more representations of a change in capabilities with respect to the infrastructure component. The command instructs the data source device to implement the second neural network architecture configuration. Example 7: The method according to any one of the preceding examples, wherein the step of generating the first output includes generating the first output in the transmitting neural network further based on at least one of sensor data input from one or more sensors of the data source device to the transmitting neural network, current operating parameters of the RF antenna interface, and capability information representing the current capabilities of at least one of the data source device or the data sink device. Example 8: A method according to any one of the preceding examples, further comprising the step of participating in the joint training of the neural network architecture configuration for the receiving-side neural network of the data sink device, together with at least one of the neural network architecture configuration for the neural network architecture components of the network infrastructure in the transmission path between the data source device and the data sink device. Example 9: A method implemented by a computer in a data sink device, comprising the steps of: receiving, at a radio frequency (RF) antenna interface of the data sink device, a first RF signal representing a data-encoded and channel-encoded version of a first data block of a data stream; supplying, as an input to a receiving-side neural network of the data sink device, a first input representing the first RF signal; generating, in the receiving-side neural network, a first recovered data block representing a recovered, channel-decoded and data-decoded version of the first data block; and supplying the first recovered data block for processing in one or more software applications of the data sink device. Example 10: The method according to Example 9, further comprising the step of selecting a first neural network architecture configuration from a plurality of neural network architecture configurations based on at least one of one or more capabilities of at least one of the data sink device or the data source device, or at least one of user-specified preferences. The step of generating the first recovered data block includes the step of generating, in the receiving-side neural network, the first recovered data block based on the first neural network architecture configuration of the receiving-side neural network. Example 11: The method according to Example 10, further comprising receiving, in response to a supply of one or more representations of one or more capabilities of at least one of a data sink device or a data source device to infrastructure components of a network infrastructure, a command from the infrastructure components of the network infrastructure; and implementing, in response to the command, a first neural network architecture configuration for a receiving neural network. Example 12: The method according to Example 10, further comprising changing a receiving neural network in response to a change in capabilities of at least one of a data sink device or a data source device and implementing a second neural network architecture configuration; receiving, at an RF antenna interface, a second RF signal representing a data-encoded and channel-encoded version of a second data block of a data stream; supplying, as an input to the receiving neural network of the data sink device, a second input representing the second RF signal; generating, in the receiving neural network, a second recovered data block representing a recovered, channel-decoded and data-decoded version of the second data block; and supplying the second recovered data block for processing in one or more software applications. Example 13: The method according to Example 12, further comprising receiving, in response to a supply of one or more representations of a change in capabilities to infrastructure components, a command from the infrastructure components of the network infrastructure. The command instructs the data sink device to implement the second neural network architecture configuration. Example 14: The method according to any one of Examples 9 to 13, wherein the step of generating the first recovered data block further comprises generating, in the receiving-side neural network, the first recovered data block based on at least one of sensor data input to the receiving-side neural network from one or more sensors of the data sink device, current operating parameters of the RF antenna interface, or ability information representing the current ability of at least one of the data sink device or the data source device. Example 15: The method according to any one of Examples 9 to 14, further comprising participating in the joint training of the neural network architecture configuration for the transmitting-side neural network of the data source device and at least one of the neural network architecture configurations for the infrastructure components of the network infrastructure in the transmission path between the data source device and the data sink device. Example 16: The method according to any one of Examples 9 to 15, further comprising supplying feedback to the infrastructure components of the network infrastructure in the transmission path between the data sink device and the data source device in response to the generation of the first recovered data block. The feedback represents a quality metric of the first recovered data block. Example 17: The method according to Example 16, wherein the feedback includes an objective quality metric generated by the data sink device that is independent of user input. Example 18: The method according to Example 16 or Example 17, wherein the feedback includes a subjective quality metric based on user input from the user of the data sink device. Example 19: A method according to any one of Examples 16 to 18, further comprising receiving, from an infrastructure component, an updated neural network architecture configuration for implementation in a receiving-side neural network in response to supplying feedback to the infrastructure component. Example 20: A method implemented by a computer in an infrastructure component of a network infrastructure, the method comprising configuring a data source device to implement a first neural network architecture configuration for a transmitting-side neural network of the data source device. The transmitting-side neural network is configured to generate, for each input data block of a data stream generated at the data source device, an output corresponding to transmission by a radio frequency (RF) antenna interface of the data source device, the corresponding output representing a data-encoded and channel-encoded version of the input data block. The method further comprises configuring a data sink device to implement a second neural network architecture configuration for a receiving-side neural network of the data sink device. The receiving-side neural network is configured to generate, for each input from an RF antenna interface of the data sink device, a corresponding data block for supply to one or more software applications of the data sink device, the corresponding data block representing a recovered, channel-decoded and data-decoded version of the corresponding data block of the data stream. Example 21: A method according to Example 20, further comprising configuring the at least one infrastructure component in a transmission path between the data source device and the data sink device to implement a third neural network architecture configuration for a neural network of the at least one infrastructure component. Example 22: The method according to Example 21, wherein the step of configuring at least one infrastructure component includes implementing a third neural network architecture configuration in response to receiving capability information from at least one of a data source device, a data sink device, or an infrastructure component of a network infrastructure, so as to configure at least one infrastructure component. Example 23: The method according to any one of Examples 20 to 22, wherein the step of configuring the data source device to implement the first neural network architecture configuration includes implementing the step of configuring the data source device to implement the first neural network architecture configuration in response to receiving capability information from at least one of a data source device, a data sink device, or an infrastructure component of a network infrastructure. The step of configuring the data sink device to implement the second neural network architecture configuration includes implementing the step of configuring the data sink device to implement the second neural network architecture configuration in response to receiving capability information from at least one of a data source device, a data sink device, or an infrastructure component of a network infrastructure. Example 24: The method according to Example 23, further including at least one of the step of configuring the data source device to implement a modified first neural network architecture configuration for the transmitting neural network in response to receiving an indication of a change in the capability of at least one of a data source device, a data sink device, or an infrastructure component of a network infrastructure, and the step of configuring the data sink device to implement a modified second neural network architecture configuration for the transmitting neural network in response to receiving an indication of a change in the capability of at least one of a data source device, a data sink device, or an infrastructure component of a network infrastructure. Example 25: A method according to any one of Examples 20 to 24, further comprising receiving feedback from the data sink device in response to the data sink device generating a data block recovered using the receiving neural network. The feedback represents a quality metric of the recovered data block. The method includes determining a modified neural network architecture configuration based on the feedback, and configuring at least one of the data sink device or the data source device to implement the modified neural network architecture configuration. Example 26: A method according to Example 25, wherein the feedback includes an objective quality metric generated by the data sink device that is independent of user input. Example 27: A method according to Example 25 or Example 26, wherein the feedback includes a subjective quality metric based on user input from a user of the data sink device. Example 28: A method according to any one of Examples 20 to 27, further comprising training a first neural network architecture configuration and a second neural network architecture configuration together. Example 29: A method according to any one of Examples 1 to 8, Example 15, Example 16, and Examples 20 to 28, wherein the data source device includes a user device. Example 30: A method according to any one of Examples 1 to 8, Example 15, Example 16, and Examples 20 to 28, wherein the data source device includes a server. Example 31: A method according to any one of Examples 8 to 30, wherein the data sink device includes a user device. Example 32: A method according to any one of the preceding examples, wherein the data stream includes a real-time data stream. Example 33: A method according to Example 32, wherein the real-time data stream includes an audio stream of a voice call. Example 34: The method according to Example 33, wherein the real-time data stream includes at least one of an audio stream or a video stream of a video call. Example 35: The method according to Example 32, wherein the data source device includes a remote video game server. The data sink device includes a user device. The real-time data stream includes a rendered video stream. Example 36: The method according to any one of Examples 1 to 8 and Examples 20 to 30, wherein the transmitting neural network includes a deep neural network. Example 37: The method according to any one of Examples 9 to 28, wherein the receiving neural network includes a deep neural network. Example 38: The method according to any one of Example 3, Example 5, Example 10, or Example 11, wherein the one or more capabilities include at least one of a sensor capability, a processing resource capability, a power capability, an RF antenna interface capability, a data generation capability, a data consumption capability, and a device accessory capability. Example 39: A device including a network interface, at least one processor coupled to the network interface, and a memory storing executable instructions. The executable instructions are configured to operate the at least one processor to execute the method according to any one of Examples 20 to 28. Example 40: A device including a radio frequency (RF) antenna interface, at least one processor coupled to the RF antenna interface, and a memory storing executable instructions. The executable instructions are configured to operate the at least one processor to execute the method according to any one of Examples 1 to 19.

[0091] In some embodiments, certain aspects of the foregoing technology may be implemented by one or more processors of a processing system that executes software. The software includes one or more sets of executable instructions that are stored on or embodied in a non-transitory computer-readable storage medium. The instructions and certain data that the software may include, when executed by one or more processors, operate the one or more processors to cause the one or more aspects of the foregoing technology to be performed. The non-transitory computer-readable storage medium may include, for example, magnetic or optical disk storage devices, flash memory, cache, semiconductor memory devices such as random access memory (RAM), or one or more other non-volatile memory devices, among others. The executable instructions stored on the non-transitory computer-readable storage medium may be in source code, assembly language code, object code, or another instruction format that is interpretable or executable by one or more processors.

[0092] A computer-readable storage medium can include any storage medium or combination of storage media that is accessible by a computer system during use to supply instructions and / or data to the computer system. Such storage media can include, but are not limited to, optical media (e.g., compact disc (CD)), digital versatile disc (DVD), Blu-ray disc, magnetic media (e.g., floppy disk, magnetic tape, or magnetic hard drive), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical systems (MEMS)-based storage media. The computer-readable storage medium can be incorporated into a computing system (e.g., system RAM or ROM), fixed and attached to a computing system (e.g., magnetic hard disk), removably attached to a computing system (e.g., optical disc or universal serial bus (USB)-based flash memory), or coupled to the computer system via a wired or wireless network (e.g., network-accessible storage mechanism (NAS)).

[0093] Note that not all of the activities or elements described above in the summary are required, that some of a particular activity or device may not be required, and that one or more additional activities may be performed or elements may be included in addition to those described. Moreover, the order in which activities are listed is not necessarily the order in which they are performed. Also, the concepts are described with reference to specific embodiments. However, one of ordinary skill in the art will understand that various modifications and changes can be made without departing from the scope of the disclosure as described below in the claims. Accordingly, the specification and figures should be regarded as illustrative rather than restrictive, and such modifications are intended to be included within the scope of the disclosure.

[0094] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, no benefit, advantage, solution to a problem, nor any function(s) that may result in or make more prominent any benefit, advantage, or solution should be construed as a critical, required, or essential function of any or all of the claims. Also, since the disclosed subject matter can be modified and practiced in different but equivalent manners apparent to those skilled in the art who benefit from the teachings herein, the specific embodiments disclosed above are illustrative only. No limitation other than those described in the following claims is intended with respect to the details of the structure or design shown herein. Accordingly, it is evident that the specific embodiments disclosed above can be varied or modified and all such variations are to be regarded as within the scope of the disclosed subject matter. Therefore, the protection sought herein is as set forth in the following claims.

Claims

Claim 1 A method implemented by a computer in a data source device, comprising: receiving, as an input to a transmitting neural network of the data source device, a first data block of a data stream, wherein the transmitting neural network implements a first neural network architecture configuration, and the method further comprises: generating, in the transmitting neural network implementing the first neural network architecture configuration, a first output representing a data-encoded and channel-encoded version of the first data block based on the first data block; controlling a radio frequency (RF) antenna interface of the data source device based on the first output to transmit a first RF signal representing the data-encoded and channel-encoded version of the first data block; changing the transmitting neural network in response to a change in communication conditions in a transmission path between the data source device and a data sink device, and implementing a second neural network architecture configuration The method comprising. Claim 2 The method according to claim 1, wherein the step of changing the transmitting neural network and implementing the second neural network architecture configuration is further performed in response to a change in the capabilities of the data source device. Claim 3 The method according to claim 1, wherein the step of changing the transmitting neural network and implementing the second neural network architecture configuration is further performed in response to a change in the capabilities of the data sink device. Claim 4 The method according to claim 1, further comprising selecting the first neural network architecture configuration from a plurality of neural network architecture configurations based on at least one of one or more capabilities of at least one of the data source device or the data sink device configured to receive the data source device or the data stream, or user-specified preferences. Claim 5 The method according to claim 1, further comprising the step of implementing the first neural network architecture configuration selected from a plurality of neural network architecture configurations of the transmission-side neural network in response to a command from an infrastructure component of the network infrastructure.

6. Receiving a second data block of the data stream as an input to the modified transmission-side neural network that implements the second neural network architecture configuration; In the transmission-side neural network, further comprising the step of generating a second output using the second neural network architecture configuration based on the second data block; The second output represents a data-encoded and channel-encoded version of the second data block; The method is Based on the second output, controlling the RF antenna interface of the data source device to transmit a second RF signal representing the data-encoded and channel-encoded version of the second data block, the method according to claim 4 or claim 5.

7. The step of generating the first output includes, in the transmission-side neural network, generating the first output further based on at least one of sensor data input to the transmission-side neural network from one or more sensors of the data source device, current operating parameters of the RF antenna interface, or capability information representing the current capabilities of at least one of the data source device or the data sink device, the method according to claim 1.

8. A method implemented by a computer in a data sink device, Receiving, at a radio frequency (RF) antenna interface of the data sink device, a first RF signal representing a data-encoded and channel-encoded version of a first data block of a data stream; Supplying, as an input to the receiving-side neural network of the data sink device, a first input representing the first RF signal, wherein the receiving-side neural network implements a first neural network architecture configuration, and the method further comprises In the receiving-side neural network implementing the first neural network architecture configuration, generating a first recovered data block representing a recovered, channel-decoded, and data-decoded version of the first data block Supplying the first recovered data block for processing in one or more software applications of the data sink device A method comprising changing the receiving-side neural network in response to a change in the communication situation in the transmission path between the data sink device and the data source device, and implementing a second neural network architecture configuration

9. The method according to claim 8, wherein the step of changing the receiving-side neural network and implementing the second neural network architecture configuration is further executed in response to a change in the capabilities of the data source device

10. The method according to claim 8, wherein the step of changing the receiving-side neural network and implementing the second neural network architecture configuration is further executed in response to a change in the capabilities of the data sink device

11. The method according to claim 8, further comprising selecting the first neural network architecture configuration from a plurality of neural network architecture configurations based on at least one of one or more capabilities of at least one of the data sink device or the data source device, or at least one of user-specified preferences

12. In the RF antenna interface, receiving a second RF signal representing a data-encoded and channel-encoded version of a second data block of the data stream Supplying, as an input to the changed receiving-side neural network of the data sink device, a second input representing the second RF signal In the receiving-side neural network, generating a second recovered data block representing a recovered, channel-decoded, and data-decoded version of the second data block; supplying the second recovered data block for processing in the one or more software applications; The method according to claim 11, further comprising. **Claim 13** The step of generating the first recovered data block includes generating the first recovered data block in the receiving-side neural network, further based on at least one of sensor data input to the receiving-side neural network from one or more sensors of the data sink device, current operating parameters of the RF antenna interface, or ability information representing at least one current ability of at least one of the data sink device or the data source device. The method according to any one of claims 8 to 12. **Claim 14** supplying feedback of a quality metric of the first recovered data block to a first infrastructure component of a network infrastructure in the transmission path between the data sink device and the data source device; receiving, in response to the feedback, an updated neural network architecture configuration for implementation in the receiving-side neural network from a second infrastructure component; The method according to any one of claims 8 to 12, further comprising. **Claim 15** The method according to claim 14, wherein the feedback includes one or more of an objective quality metric generated by the data sink device independently of user input or a subjective quality metric based on user input from a user of the data sink device. **Claim 16** The method according to any one of claims 8 to 12, wherein the data sink device comprises a device configured to wirelessly connect to a base station, a wireless access point, or another component of an infrastructure network. **Claim 17** The method according to any one of claims 8 to 12, wherein the data sink device is a user equipment or a server. **Claim 18** A method executed by a computer in a first infrastructure component of a network infrastructure, comprising: configuring the data source device to implement a first neural network architecture configuration for a transmission-side neural network of the data source device, the transmission-side neural network implementing the first neural network architecture configuration and being configured to generate, for each input data block of a data stream generated at the data source device, an output corresponding to transmission by a radio frequency (RF) antenna interface of the data source device, the corresponding output representing a data-encoded and channel-encoded version of the input data block; The method further comprises: configuring the data sink device to implement a second neural network architecture configuration for a reception-side neural network of the data sink device, the reception-side neural network implementing the second neural network architecture configuration and being configured to generate, for each input from an RF antenna interface of the data sink device, a corresponding data block for supply to one or more software applications of the data sink device, the corresponding data block representing a recovered, channel-decoded and data-decoded version of the corresponding data block of the data stream; The method further comprises: in response to receiving an indication of a change in the capabilities of an infrastructure component of the network infrastructure in a transmission path between the data source device and the data sink device, configuring the data source device to implement a modified neural network architecture configuration for the transmission-side neural network, or in response to receiving an indication of a change in the capabilities of an infrastructure component of the network infrastructure in a transmission path between the data source device and the data sink device, configuring the data sink device to implement a modified neural network architecture configuration for the transmission-side neural network ​ further comprising at least one of Method.

19. In the transmission path between the data source device and the data sink device, further comprising the step of configuring the second infrastructure component so as to implement a third neural network architecture configuration for the neural network of the second infrastructure component, wherein the second infrastructure component comprises the first infrastructure component or another infrastructure component, the method according to claim 18.

20. The step of configuring the second infrastructure component comprises the step of configuring the second infrastructure component so as to implement the third neural network architecture configuration in response to receiving capability information from at least one of the data source device, the data sink device, or the infrastructure components of the network infrastructure, the method according to claim 19.

21. The first infrastructure component comprises one of a base station, a wireless access point, or a server The second infrastructure component comprises one of a base station or a wireless access point being at least one of, the method according to claim 19.

22. The step of configuring the data source device so as to implement the first neural network architecture configuration comprises the step of configuring the data source device so as to implement the first neural network architecture configuration in response to receiving capability information representing one or more capabilities from at least one of the data source device, the data sink device, or the infrastructure components of the network infrastructure, The step of configuring the data sink device to implement the second neural network architecture configuration includes the step of configuring the data sink device to implement the second neural network architecture configuration in response to receiving capability information representing one or more capabilities from at least one of the data source device, the data sink device, or the infrastructure components of the network infrastructure. The method according to claim 18.

23. The method further includes receiving feedback from the data sink device in response to the data sink device generating a data block recovered using the receiving-side neural network, the feedback representing a quality metric of the recovered data block, The method includes: determining a modified neural network architecture configuration based on the feedback; and configuring at least one of the data sink device or the data source device to implement the modified neural network architecture configuration The method according to any one of claims 18 to 22.

24. The data source device comprises one of a user device or a server, and the data sink device comprises the other of a user device or a server. The method according to any one of claims 1 to 5, claims 7 to 12, and claims 18 to 22.

25. The data stream includes a real-time data stream, The real-time data stream includes one of an audio stream of a voice call, an audio stream or a video stream of a video call, or The data source device comprises a remote video game server, the data sink device comprises a user device, and the real-time data stream includes a rendered video stream, At least one of which. The method according to any one of claims 1 to 5, claims 7 to 12, and claims 18 to 22.

26. The one or more capabilities include at least one of a sensor capability, a processing resource capability, a power supply capability, an RF antenna interface capability, a data generation capability, a data consumption capability, and a device accessory capability, according to any one of claims 1 to 5, claim 7, claim 11, claim 12, or claim 22.

27. A network interface, at least one processor coupled to the network interface, a memory storing executable instructions A device comprising: The executable instructions are configured to operate the at least one processor to execute the method according to any one of claims 18 to 22. Device.

28. A radio frequency (RF) antenna interface, at least one processor coupled to the RF antenna interface, a memory storing executable instructions A device comprising: The executable instructions are configured to operate the at least one processor to execute the method according to any one of claims 1 to 5 and claims 7 to 12. Device.

29. A program for causing at least one processor to execute the method according to any one of claims 1 to 5, claims 7 to 12, and claims 18 to 22.

Images