Beam prediction using digital twins

Abstract

A processor-implemented method for multimodal beam management, implemented by a network device, includes: receiving an input stream from one or more sensors by the network device; generating a digital twin modeling the environment of an area observed by the one or more sensors; the digital twin including one or more objects detected based on the input stream; and managing wireless communication signal beams, at least in part, based on the digital twin for communication with at least one user equipment (UE) in the area observed by the one or more sensors.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 505,966, filed November 9, 2023, entitled “MULTI-MODAL BEAM PREDICTION”, the entire disclosure of which is expressly incorporated herein by reference. Technical Field

[0003] This disclosure relates in general to wireless communications, and more specifically to signal beam management using machine learning models. Background Technology

[0004] Wireless communication systems are widely deployed to provide a variety of telecommunications services, such as telephone, video, data, messaging, and broadcasting. Typical wireless communication systems employ multiple access technologies that enable communication with multiple users by sharing available system resources (e.g., bandwidth, transmit power, etc.). Examples of such multiple access technologies include Code Division Multiple Access (CDMA) systems, Time Division Multiple Access (TDMA) systems, Frequency Division Multiple Access (FDMA) systems, Orthogonal Frequency Division Multiple Access (OFDMA) systems, Single Carrier Frequency Division Multiple Access (SC-FDMA) systems, Time Division Synchronous Code Division Multiple Access (TD-SCDMA) systems, and Long Term Evolution (LTE). LTE / LTE-Advanced is an enhancement set of the Universal Mobile Telecommunications System (UMTS) mobile standard released by the 3rd Generation Partnership Project (3GPP). Narrowband (NB) Internet of Things (IoT) and Enhanced Machine-Type Communications (eMTC) are enhancement sets of LTE for machine-type communications.

[0005] A wireless communication network may include multiple base stations (BSs) capable of supporting communication for multiple user equipment (UEs). UEs can communicate with the base stations (BSs) via downlinks and uplinks. A downlink (or forward link) refers to the communication link from the BS to the UE, while an uplink (or reverse link) refers to the communication link from the UE to the BS. As will be described in more detail, a BS may be referred to as a Node B, Evolved Node B (eNB), gNB, Access Point (AP), Radio Headend, Transmit and Receive Point (TRP), New Radio (NR) BS, 5G Node B, etc.

[0006] The above multiple access technologies have been adopted in various telecommunications standards to provide a common protocol that enables different user equipment to communicate at the city, country, region, and even global levels. New Radio (NR) (also known as 5G) is a set of enhancements to the LTE mobile standard released by the 3rd Generation Partnership Project (3GPP). NR is designed to better support mobile broadband internet access by using Orthogonal Frequency Division Multiplexing (OFDM) with a Cyclic Prefix (CP) on the downlink (DL), and CP-OFDM and / or SC-FDM (e.g., also known as Discrete Fourier Transform Extended OFDM (DFT-s-OFDM)) on the uplink (UL), as well as supporting beamforming, multiple-input multiple-output (MIMO) antenna technologies and carrier aggregation to improve spectral efficiency, reduce costs, improve service, utilize new spectrum, and better integrate with other open standards.

[0007] Artificial neural networks can include interconnected groups of artificial neurons (e.g., neuron models). Artificial neural networks can be computing devices or represented as methods performed by computing devices. Convolutional neural networks (such as deep convolutional neural networks) are a type of feedforward artificial neural network. Convolutional neural networks can include individual neuron layers that can be configured in tiled receptive fields. Applying neural network processing to wireless communication to achieve greater efficiency would be desirable. Summary of the Invention

[0008] This disclosure is set forth in the independent claims. Some aspects of this disclosure are described in the dependent claims.

[0009] In some aspects of this disclosure, a processor-implemented method includes receiving an input stream from one or more sensors by a network device. The processor-implemented method also includes generating a digital twin by the network device that models the environment of an area observed by the one or more sensors. The digital twin includes one or more objects detected based on the input stream. The processor-implemented method further includes managing wireless communication signal beams by the network device, at least partially based on the digital twin, for communicating with at least one user equipment (UE) in the area observed by the one or more sensors.

[0010] Various aspects of this disclosure relate to an apparatus having at least one memory and one or more processors coupled to the at least one memory. The processors are configured to receive input streams from one or more sensors by a network device. The processors are also configured to generate a digital twin by the network device modeling the environment of an area observed by the one or more sensors. The digital twin includes one or more objects detected based on the input streams. The processors are further configured to manage wireless communication signal beams by the network device, at least partially based on the digital twin, for communication with at least one user equipment (UE) in the area observed by the one or more sensors.

[0011] In some aspects of this disclosure, a processor-implemented method executed by at least one processor includes a UE initiating wireless communication with a network device. The method also includes the UE receiving a wireless communication signal beam from the network device. This wireless communication signal beam is managed based on a digital twin and input streams from one or more sensors from one or more objects in the environment being observed by the UE and modeled within the digital twin.

[0012] Various aspects of this disclosure relate to an apparatus having at least one memory and one or more processors coupled to the at least one memory. The processor is configured to initiate wireless communication with a network device by a UE. The processor is configured to receive a wireless communication signal beam from the network device by the UE. The wireless communication signal beam is managed based on a digital twin and input streams from one or more sensors from one or more objects in the environment observed by the UE and modeled in the digital twin.

[0013] The aspects as a whole include, as described substantially with reference to the accompanying drawings and description, and as illustrated in the accompanying drawings and description, methods, apparatus, systems, computer program products, non-transitory computer-readable media, user equipment, base stations, wireless communication devices, and processing systems.

[0014] The features and technical advantages of the examples according to this disclosure have been summarized rather extensively above in order to better understand the detailed description that follows. Additional features and advantages will be described. The disclosed concepts and specific examples can be readily utilized as the basis for modifying or designing other structures for achieving the same purpose of this disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the disclosed concepts, in both their organization and manner of operation, and the associated advantages, will be better understood by considering the following description in conjunction with the accompanying drawings. Each drawing in the accompanying drawings is for illustrative and descriptive purposes and not as a limitation of the definitions in the claims. Attached Figure Description

[0015] To gain a detailed understanding of the features of this disclosure, reference can be made to various aspects, some of which are illustrated in the accompanying drawings. However, it should be noted that the drawings illustrate only certain aspects of this disclosure and should therefore not be considered as limiting its scope, as the description may allow for other equivalent aspects. The same reference numerals in different drawings may identify the same or similar elements.

[0016] Figure 1 It is a block diagram that conceptually illustrates examples of wireless communication networks according to various aspects of this disclosure.

[0017] Figure 2 This is a block diagram that conceptually illustrates examples of communication between a base station and a user equipment (UE) in a wireless communication network according to various aspects of this disclosure.

[0018] Figure 3 This is a block diagram illustrating an example decomposed base station architecture according to various aspects of this disclosure.

[0019] Figure 4 Example implementations of designing neural networks using a system-on-a-chip (SoC) including a general-purpose processor, according to certain aspects of this disclosure, are illustrated.

[0020] Figure 5A , Figure 5B and Figure 5C These are illustrations of neural networks according to various aspects of this disclosure.

[0021] Figure 5D This is a diagram illustrating exemplary deep convolutional networks (DCNs) according to various aspects of this disclosure.

[0022] Figure 6 This is a block diagram illustrating exemplary deep convolutional networks (DCNs) according to various aspects of this disclosure.

[0023] Figure 7 This is a high-level block diagram illustrating an example processing pipeline for wireless communication according to various aspects of this disclosure.

[0024] Figure 8 This is a diagram illustrating example architectures for beam prediction based on various aspects of this disclosure.

[0025] Figure 9 This is a block diagram illustrating an example pipeline for multimodal beam prediction according to various aspects of this disclosure.

[0026] Figure 10 This is an illustration of examples of segmentation of an input image according to various aspects of this disclosure.

[0027] Figure 11This is a block diagram illustrating example processes for object tracking and matching according to various aspects of this disclosure.

[0028] Figure 12 This is a block diagram illustrating example rule-based decision-making based on blocking detection according to various aspects of this disclosure.

[0029] Figure 13 This is an example diagram illustrating various aspects of blocking prediction based on this disclosure.

[0030] Figure 14 This is a flowchart illustrating an example process for beam management by a network device using digital twins and input streams from sensors, according to various aspects of this disclosure.

[0031] Figure 15 This is a flowchart illustrating an example process for wireless communication by a user equipment using a beam estimated based on a digital twin and an input stream from a sensor, according to various aspects of this disclosure. Detailed Implementation

[0032] Various aspects of this disclosure are described more fully below with reference to the accompanying drawings. However, this disclosure may be embodied in many different forms and should not be construed as limited to any particular structure or function presented throughout this disclosure. Rather, these aspects are provided so that this disclosure will be comprehensive and complete, and will fully convey the scope of this disclosure to those skilled in the art. Based on the teachings, those skilled in the art will recognize that the scope of this disclosure is intended to cover any aspect of this disclosure, whether implemented independently of or in combination with any other aspect of this disclosure. For example, an apparatus or method may be implemented using any number of the aspects set forth. Furthermore, the scope of this disclosure is intended to cover such apparatus or methods practiced using other structures, functions, or structures and functions other than or supplementing the various aspects of this disclosure set forth. It should be understood that any aspect of this disclosure may be embodied by one or more elements of the claims.

[0033] Several aspects of a telecommunications system will now be presented with reference to various devices and techniques. These devices and techniques will be described in detail below and illustrated in the accompanying drawings by various blocks, modules, components, circuits, steps, processes, algorithms, etc. (collectively, “elements”). These elements can be implemented using hardware, software, or a combination thereof. Whether such elements are implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole.

[0034] It should be noted that while the aspects may be described using terms commonly associated with 5G, 6G and subsequent wireless technologies, the aspects of this disclosure may be applied in other generation-based communication systems, such as and including 3G and / or 4G technologies.

[0035] Sixth-generation (6G) wireless communication systems may include support for applications such as augmented reality, multi-sensor communication, and high-fidelity holograms (e.g., via cameras, light detection and ranging (LiDAR) sensors, and inertial measurement unit (IMU) sensors). Such applications can also help solve communication problems and make communication more efficient.

[0036] Maintaining high performance can be challenging as demand for services continues to grow and the number of supported devices increases. These challenges may include adapting to the different characteristics encountered by various data modalities. Furthermore, the fusion of different data modalities involves carefully designing feature extraction and fusion processes.

[0037] Therefore, various aspects of this disclosure can employ machine learning (ML), and more generally artificial intelligence (AI), to leverage multimodal data and context awareness. In various aspects, the camera can collect or infer information from the environment for specific downstream tasks (e.g., obstruction estimation for communication) or to assist downstream ML systems.

[0038] Specific aspects of the subject matter described in this disclosure can be implemented to achieve one or more of the following potential advantages. In some examples, the described techniques, such as beam management based on digital twins and input streams from one or more sensors, can enable more reliable communication in congested areas or during severe weather.

[0039] Figure 1This is an illustration of a network 100 in which various aspects of the present disclosure may be practiced. Network 100 may be a 5G or NR network, or some other wireless network (such as an LTE network). Wireless network 100 may include multiple BS 110s (shown as BS 110a, BS 110b, BS 110c, and BS 110d) and other network entities. A BS is an entity that communicates with a user equipment (UE) and may also be referred to as a base station, NR BS, Node B, gNB, 5G Node B, access point, Transmit and Receive Point (TRP), network node, network entity, etc. A BS may be implemented as an aggregated base station, a decomposed base station, an Integrated Access and Backhaul (IAB) node, a relay node, a sidelink node, etc. A BS may be implemented in an aggregated or monolithic base station architecture, or alternatively in a decomposed base station architecture, and may include one or more of a central unit (CU), a distributed unit (DU), a radio unit (RU), a near real-time (near RT) RAN intelligent controller (RIC), or a non-real-time (non-RT) RIC. Each BS can provide communication coverage for a specific geographic area. In 3GPP, depending on the context in which the term is used, the term "cell" can refer to the coverage area of ​​a BS and / or the BS subsystem serving that coverage area.

[0040] A BS can provide communication coverage for macrocells, picocells, femtocells, and / or another type of cell. A macrocell can cover a relatively large geographic area (e.g., a radius of several kilometers) and can allow unrestricted access for UEs with a service subscription. A picocell can cover a relatively small geographic area and can allow unrestricted access for UEs with a service subscription. A femtocell can cover a relatively small geographic area (e.g., a home) and can allow restricted access for UEs associated with the femtocell (e.g., UEs in a Closed Subscriber Group (CSG)). A BS used for macrocells can be referred to as a macro BS. A BS used for picocells can be referred to as a pico BS. A BS used for femtocells can be referred to as a femtocell BS or a home BS. Figure 1 In the example shown, BS 110a can be a macro BS for macro cell 102a, BS 110b can be a pico BS for pico cell 102b, and BS 110c can be a femto BS for femto cell 102c. A BS can support one or more (e.g., three) cells. The terms “eNB,” “base station,” “NR BS,” “gNB,” “AP,” “node B,” “5G NB,” “TRP,” and “cell” are used interchangeably.

[0041] In some respects, the cell does not need to be stationary, and the geographical area of ​​the cell can be moved depending on the location of the mobile BS. In some respects, the BS can use any suitable transport network to interconnect with each other and / or with one or more other BSs or network nodes (not shown) in the wireless network 100 via various types of backhaul interfaces (such as direct physical connections, virtual networks, etc.).

[0042] The wireless network 100 may also include a relay station. A relay station is an entity that can receive data transmissions from an upstream station (e.g., a BS or a UE) and forward those data transmissions to a downstream station (e.g., a UE or a BS). A relay station can also be a UE that can relay transmissions for other UEs. Figure 1 In the example shown, relay station 110d can communicate with macro BS 110a and UE 120d to facilitate communication between BS 110a and UE 120d. A relay station can also be referred to as a relay BS, relay base station, repeater, etc.

[0043] Wireless network 100 can be a heterogeneous network comprising different types of Base Stations (BSs) (e.g., macro BSs, pico BSs, femto BSs, relay BSs, etc.). These different types of BSs can have different transmit power levels, different coverage areas, and different effects on interference in wireless network 100. For example, macro BSs can have high transmit power levels (e.g., 5 watts to 40 watts), while pico BSs, femto BSs, and relay BSs can have lower transmit power levels (e.g., 0.1 watts to 2 watts).

[0044] Network controller 130 can be coupled to a group of base stations (BSs) and can provide coordination and control for these BSs. Network controller 130 can communicate with the BSs via backhaul. BSs can also communicate with each other (e.g., directly or indirectly via wireless or wired backhaul).

[0045] UEs 120 (e.g., 120a, 120b, 120c) may be distributed throughout the wireless network 100, and each UE may be stationary or mobile. UEs may also be referred to as access terminals, terminals, mobile stations, subscriber units, stations, etc. UEs may be cellular phones (e.g., smartphones), personal digital assistants (PDAs), wireless modems, wireless communication devices, handheld devices, laptops, cordless phones, wireless local loop (WLL) stations, tablets, cameras, gaming devices, netbooks, smartbooks, ultrabooks, medical devices or equipment, biometric sensors / devices, wearable devices (smartwatches, smart clothing, smart glasses, smart wristbands, smart jewelry (e.g., smart rings, smart bracelets)), entertainment devices (e.g., music or video devices or satellite radios), components or sensors of vehicles, smart meters / sensors, industrial manufacturing equipment, GPS devices, or any other suitable device configured to communicate via wireless or wired media.

[0046] Some UEs can be considered Machine-Type Communication (MTC) or Evolved or Enhanced Machine-Type Communication (eMTC) UEs. For example, MTC and eMTC UEs include robots, drones, remote devices, sensors, meters, monitors, location tags, etc., that can communicate with a base station, another device (e.g., a remote device), or some other entity. Wireless nodes can provide connectivity to or to a network (such as the Internet or a wide area network with cellular networks) via wired or wireless communication links. Some UEs can be considered Internet of Things (IoT) devices and / or can be implemented as NB-IoT (Narrowband Internet of Things) devices. Some UEs can be considered Customer Premises Equipment (CPE). UE 120 can be included in a housing that houses the components of UE 120, such as processor components, memory components, etc.

[0047] Generally, any number of wireless networks can be deployed in a given geographical area. Each wireless network can support a specific Radio Access Technology (RAT) and can operate on one or more frequencies. A RAT can also be referred to as a radio technology, air interface, etc. A frequency can also be referred to as a carrier, frequency channel, etc. Within a given geographical area, each frequency can support a single RAT to avoid interference between wireless networks using different RATs. In some cases, NR or 5G RAT networks can be deployed.

[0048] In some respects, two or more UEs 120 (e.g., shown as UE 120a and UE 120e) may communicate directly using one or more sidelink channels (e.g., without using base station 110 as an intermediary). For example, UEs 120 may communicate using peer-to-peer (P2P) communication, device-to-device (D2D) communication, vehicle-to-everything (V2X) protocols (e.g., which may include vehicle-to-vehicle (V2V) protocols, vehicle-to-infrastructure (V2I) protocols, mesh networks, etc.). In this case, UEs 120 may perform scheduling operations, resource selection operations, and / or other operations described elsewhere herein, performed by base station 110. For example, base station 110 may configure UEs 120 via downlink control information (DCI), radio resource control (RRC) signaling, media access control-control element (MAC-CE), or via system information (e.g., system information block (SIB)).

[0049] UE 120 may include beam management module 140. For simplicity, only one UE 120d is shown as including beam management module 140. Beam management module 140 can determine beam updates for wireless communication with network devices. For example, using beam management module 140, UE (e.g., 120) can initiate wireless communication with network devices (e.g., 110 / 130). UE (e.g., 120) can receive wireless communication signal beams from network devices. The wireless communication signal beams can be managed based on a digital twin and input streams from one or more sensors from one or more objects in the environment being observed by the UE (e.g., 120) and modeled in the digital twin.

[0050] Network controller 130 or base station 110 or any other network device (e.g., such as...) Figure 3 The network device (e.g., 110 / 130) may include a beam management module 138 for managing wireless communication signal beams used to communicate with one or more UEs 120. For example, using the beam management module 138, a network device (e.g., 110 / 130) may receive an input stream from one or more sensors. The network device (e.g., 110 / 130) may generate a digital twin modeling the environment of an area observed by the one or more sensors. The digital twin includes one or more objects detected based on the input stream. The network device (e.g., 110 / 130) may manage the wireless communication signal beams, at least in part, based on the digital twin, for communicating with at least one UE (e.g., 120) in the area observed by the one or more sensors.

[0051] As indicated above, Figure 1 This is provided merely as an example. Other examples are available in conjunction with [the relevant documentation / information]. Figure 1 The examples described are different.

[0052] Figure 2 A block diagram of a design 200 for a base station 110 and a UE 120 is shown. The base station and the UE can be... Figure 1 One of the base stations in the base station and Figure 1 One of the UEs in the UE. Base station 110 may be equipped with T antennas 234a to 234t, and UE 120 may be equipped with R antennas 252a to 252r, where generally, T≥1 and R≥1.

[0053] At base station 110, transmit processor 220 can receive data for one or more UEs from data source 212, select one or more modulation and decoding schemes (MCS) for each UE based at least in part on the Channel Quality Indicator (CQI) received from each UE, process (e.g., encode and modulate) the data for each UE based at least in part on the MCS selected for each UE, and provide data symbols for all UEs. Reducing the MCS decreases throughput but increases transmission reliability. Transmit processor 220 can also process system information (e.g., for semi-static resource partitioning information (SRPI), etc.) and control information (e.g., CQI requests, grants, upper-layer signaling, etc.), and provide overhead symbols and control symbols. Transmit processor 220 can also generate reference symbols for reference signals (e.g., cell-specific reference signals (CRS)) and synchronization signals (e.g., primary synchronization signal (PSS) and secondary synchronization signal (SSS)). The transmit (TX) multiple-input multiple-output (MIMO) processor 230 can perform spatial processing (e.g., pre-decoding) on ​​data symbols, control symbols, overhead symbols, and / or reference symbols, where applicable, and can provide T output symbol streams to T modulators (MODs) 232a to 232t. Each modulator 232 can process the corresponding output symbol stream (e.g., for orthogonal frequency division multiplexing (OFDM), etc.) to obtain an output sample stream. Each modulator 232 can further process (e.g., convert to analog, amplify, filter, and up-convert) the output sample stream to obtain a downlink signal. The T downlink signals from modulators 232a to 232t can be transmitted via T antennas 234a to 234t, respectively. Position coding can be used to generate synchronization signals to transmit additional information, according to various aspects described in more detail below.

[0054] At UE 120, antennas 252a to 252r can receive downlink signals from base station 110 and / or other base stations, and can provide the received signals to demodulators (DEMODs) 254a to 254r respectively. Each demodulator 254 can adjust (e.g., filter, amplify, down-convert, and digitize) the received signal to obtain an input sample. Each demodulator 254 can further process these input samples (e.g., for OFDM, etc.) to obtain the received symbols. MIMO detector 256 can obtain the received symbols from all R demodulators 254a to 254r, perform MIMO detection on the received symbols where applicable, and provide the detected symbols. Receiver processor 258 can process (e.g., demodulate and decode) the detected symbols, provide decoded data for UE 120 to data sink 260, and provide decoded control information and system information to controller / processor 280. The channel processor can determine the Reference Signal Received Power (RSRP), Received Signal Strength Indicator (RSSI), Reference Signal Received Quality (RSRQ), and / or Channel Quality Indicator (CQI), etc. In some aspects, one or more components of the UE 120 may be included in a housing.

[0055] On the uplink, at UE 120, the transmitting processor 264 can receive data from data source 262 and control information (e.g., for reports including RSRP, RSSI, RSRQ, CQI, etc.) from controller / processor 280, and process the data and control information. The transmitting processor 264 can also generate reference symbols for one or more reference signals. Symbols from the transmitting processor 264 can be pre-decoded by the TX MIMO processor 266, where applicable, further processed by modulators 254a to 254r (e.g., for Discrete Fourier Transform Extended OFDM (DFT-s-OFDM), CP-OFDM, etc.), and transmitted to base station 110. At base station 110, uplink signals from UE 120 and other UEs can be received by antenna 234, processed by demodulator 254, detected by MIMO detector 236 (where applicable), and further processed by receiving processor 238 to obtain decoded data and control information transmitted by UE 120. The receiver processor 238 can provide the decoded data to the data sink 239 and the decoded control information to the controller / processor 240. The base station 110 may include a communication unit 244 and communicates with the network controller 130 via the communication unit 244. The network controller 130 may include a communication unit 294, a controller / processor 290, and a memory 292.

[0056] The controller / processor 240 of base station 110, the controller / processor 280 of UE 120 and / or Figure 2Any other component may perform one or more techniques associated with determining membership in region-based joint learning, as described in more detail elsewhere. For example, the controller / processor 240 of base station 110, the controller / processor 280 of UE 120, and / or Figure 2 Any other component that can execute or direct, for example Figures 9 to 10 The operation of the process and / or other processes as described. Memory 242 and memory 282 may store data and program code for base station 110 and UE 120, respectively. Scheduler 246 may schedule the UE for data transmission on downlink and / or uplink.

[0057] In some aspects, UE 120 and / or base station 110 may include components for receiving, components for generating, components for managing, and components for initiating. Such components may include combinations of... Figure 2 The UE 120 or one or more components of the base station 110 described.

[0058] As indicated above, Figure 2 This is provided merely as an example. Other examples are available in conjunction with [the relevant documentation / information]. Figure 2 The examples described are different.

[0059] In some cases, different types of devices supporting different types of applications and / or services can coexist in a cell. Examples of different types of devices include UE handsets, Customer Premises Equipment (CPE), vehicles, Internet of Things (IoT) devices, etc. Examples of different types of applications include Ultra Reliable Low Latency Communication (URLLC) applications, Massive Machine-Type Communication (mMTC) applications, Enhanced Mobile Broadband (eMBB) applications, Vehicle-to-Everything (V2X) applications, etc. Furthermore, in some cases, a single device can simultaneously support different applications or services.

[0060] The deployment of communication systems (such as 5G New Radio (NR) systems) can involve various components or parts arranged in multiple ways. In a 5G NR system or network, network nodes, network entities, network mobility elements, radio access network (RAN) nodes, core network nodes, network elements or network equipment (such as base stations (BS)), or one or more units (or components) performing base station functionality can be implemented in aggregated or decomposed architectures. For example, BSs (such as Node B (NB), evolved NB (eNB), NR BS, 5G NB, access point (AP), transmit and receive point (TRP), or cell, etc.) can be implemented as aggregated base stations (also known as standalone BS or monolithic BS) or decomposed base stations.

[0061] Aggregated base stations can be configured to utilize a radio protocol stack that is physically or logically integrated within a single RAN node. Decentralized base stations can be configured to utilize a protocol stack that is physically or logically distributed across two or more units, such as one or more central or centralized units (CUs), one or more distributed units (DUs), or one or more radio units (RUs). In some respects, the CU may be implemented within a RAN node, and one or more DUs may co-located with the CU, or alternatively, may be geographically or virtually distributed across one or more other RAN nodes. DUs may be implemented to communicate with one or more RUs. Each of the CUs, DUs, and RUs may also be implemented as a virtual unit, namely a virtual centralized unit (VCU), a virtual distributed unit (VDU), or a virtual radio unit (VRU).

[0062] Base station type operation or network design can take into account the aggregation characteristics of base station functionality. For example, decomposed base stations can be utilized in Integrated Access Backhaul (IAB) networks, Open Radio Access Networks (O-RAN (such as network configurations advocated by the O-RAN Alliance)), or Virtualized Radio Access Networks (vRAN, also known as Cloud Radio Access Networks (C-RAN)). Decomposition can include distributing functionality across two or more units in various physical locations, as well as virtually distributing the functionality of at least one unit, which enables flexibility in network design. The various units in a decomposed base station or decomposed RAN architecture can be configured for wired or wireless communication with at least one other unit.

[0063] Figure 3 A diagram illustrating an example decomposed base station 300 architecture is shown. The decomposed base station 300 architecture may include one or more central units (CUs) 310, which may communicate directly with the core network 320 via a backhaul link, or indirectly with the core network 320 via one or more decomposed base station units, such as a near real-time (near-RT) RAN Intelligent Controller (RIC) 325 via an E2 link, or a non-real-time (non-RT) RIC 315 associated with a Service Management and Orchestration (SMO) framework 305, or both. CUs 310 may communicate with one or more distributed units (DUs) 330 via corresponding midhaul links (such as F1 interfaces). DUs 330 may communicate with one or more radio units (RUs) 340 via corresponding fronthaul links. RUs 340 may communicate with corresponding UEs 120 via one or more radio frequency (RF) access links. In some implementations, a UE 120 may be served simultaneously by multiple RUs 340.

[0064] Each of these units (e.g., CU 310, DU 330, RU 340, and near-RT RIC 325, non-RT RIC 315, and SMO frame 305) may include one or more interfaces, or may be coupled to one or more interfaces configured to receive or transmit signals, data, or information (collectively, signals) via wired or wireless transmission media. Each of the units, or an associated processor or controller providing instructions to the communication interfaces of these units, may be configured to communicate with one or more other units via transmission media. For example, these units may include wired interfaces configured to receive signals or transmit signals to one or more other units via wired transmission media. Additionally, these units may include wireless interfaces that may include receivers, transmitters, or transceivers (such as radio frequency (RF) transceivers) configured to receive signals or transmit signals to one or more other units via wireless transmission media, or both.

[0065] In some aspects, the CU 310 can host one or more higher-level control functions. Such control functions may include Radio Resource Control (RRC), Packet Data Convergence Protocol (PDCP), Serving Data Adaptation Protocol (SDAP), etc. Each control function can be implemented using an interface configured to signal to other control functions hosted by the CU 310. The CU 310 can be configured to handle user plane functionality (i.e., Central Unit-User Plane (CU-UP)), control plane functionality (i.e., Central Unit-Control Plane (CU-CP)), or a combination thereof. In some implementations, the CU 310 can be logically divided into one or more CU-UP units and one or more CU-CP units. When implemented in an O-RAN configuration, the CU-UP units can communicate bidirectionally with the CU-CP units via an interface such as an E1 interface. The CU 310 can be implemented to communicate with the DU 330 for network control and signaling as needed.

[0066] DU 330 may correspond to a logical unit comprising one or more base station functions for controlling the operation of one or more RU 340s. In some aspects, DU 330 may, at least in part, host one or more of the Radio Link Control (RLC) layer, the Media Access Control (MAC) layer, and one or more high physical (PHY) layers (such as modules for forward error correction (FEC) encoding and decoding, scrambling, modulation and demodulation, etc.) depending on functional splits (such as those defined by the 3rd Generation Partnership Project (3GPP). In some aspects, DU 330 may also host one or more low PHY layers. Each layer (or module) may be implemented using an interface configured to communicate signals with other layers (and modules) hosted by DU 330 or with control functions hosted by CU310.

[0067] Lower-layer functionality can be implemented by one or more RU 340s. In some deployments, an RU340 controlled by a DU 330 may correspond to a logical node that at least partially hosts RF processing functions or low-PHY layer functions (such as performing Fast Fourier Transform (FFT), Inverse FFT (iFFT), digital beamforming, Physical Random Access Channel (PRACH) extraction and filtering, etc.) based on functional decomposition such as lower-layer functional decomposition, or both. In such architectures, the RU 340 may be implemented to handle over-the-air (OTA) communications with one or more UE 120s. In some specific implementations, the real-time and non-real-time aspects of communication with the control plane and user plane of the RU 340 may be controlled by the corresponding DU 330. In some scenarios, this configuration allows the DU330 and CU 310 to be implemented in cloud-based RAN architectures such as vRAN architectures.

[0068] The SMO framework 305 can be configured to support RAN deployment and provisioning of both non-virtualized and virtualized network elements. For non-virtualized network elements, the SMO framework 305 can be configured to support the deployment of dedicated physical resources for RAN coverage requirements, which can be managed via operation and maintenance interfaces such as the O1 interface. For virtualized network elements, the SMO framework 305 can be configured to interact with a cloud computing platform such as the Open Cloud (O-Cloud) 390 to perform network element lifecycle management (such as instantiating virtualized network elements) via a cloud computing platform interface such as the O2 interface. Such virtualized network elements may include, but are not limited to, CU 310, DU 330, RU 340, and near-RT RIC 325. In some implementations, the SMO framework 305 can communicate with the hardware aspects of the 4G RAN (such as the Open eNB (O-eNB) X11) via the O1 interface. Additionally, in some implementations, the SMO framework 305 can communicate directly with one or more RU 340s via the O1 interface. The SMO framework 305 may also include a non-RT RIC 315 configured to support the functionality of the SMO framework 305.

[0069] The non-RT RIC 315 can be configured to include logical functions that enable non-real-time control and optimization of RAN elements and resources, including artificial intelligence / machine learning (AI / ML) workflows for model training and updates, or policy-based guidance for applications / features in the near-RT RIC 325. The non-RT RIC 315 can be coupled to or communicate with the near-RT RIC 325, such as via an A1 interface. The near-RT RIC 325 can be configured to include logical functions that enable near real-time control and optimization of RAN elements and resources via data collection and actions on interfaces connecting one or more CUs 310s, one or more DUs 330s, or both, and O-eNBs to the near-RT RIC 325, such as via an E2 interface.

[0070] In some implementations, to generate AI / ML models to be deployed in the near-RT RIC 325, the non-RT RIC 315 may receive parameters or external enrichment information from an external server. This information can be utilized by the near-RT RIC 325 and can be received from non-network data sources or network functions at the SMO framework 305 or the non-RT RIC 315. In some examples, the non-RT RIC 315 or near-RT RIC 325 may be configured to tune RAN behavior or performance. For example, the non-RT RIC 315 may monitor long-term trends and patterns in performance and employ AI / ML models to perform corrective actions via the SMO framework 305 (e.g., via O1 reconfiguration) or via the creation of RAN management policies (e.g., A1 policies).

[0071] Figure 4 An example implementation of a System-on-Chip (SOC) 400 according to certain aspects of this disclosure is illustrated, which may include a Central Processing Unit (CPU) 402 or a multi-core CPU configured to generate gradients for training a neural network. The SOC 400 may be included in a base station 110 or a UE 120. Variables (e.g., neural signals and synaptic weights), system parameters associated with the computing device (e.g., a weighted neural network), latency, frequency window (bin) information, and task information may be stored in a memory block associated with a Neural Processing Unit (NPU) 408, a memory block associated with the CPU 402, a memory block associated with a Graphics Processing Unit (GPU) 404, a memory block associated with a Digital Signal Processor (DSP) 406, a memory block 418, or may be distributed across multiple blocks. Instructions executed at the CPU 402 may be loaded from the program memory associated with the CPU 402 or may be loaded from memory block 418.

[0072] The SOC 400 may also include additional processing blocks tailored for specific functions, such as a GPU 404, a DSP 406, a connectivity block 410 (which may include fifth-generation (5G) connectivity, fourth-generation LTE (4G) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 412 capable of, for example, detecting and recognizing gestures. In one implementation, the NPU is implemented within a CPU, DSP, and / or GPU. The SOC 400 may also include a sensor processor 414, an image signal processor (ISP) 416, and / or a navigation module 420, which may include a global positioning system.

[0073] The SOC 400 may be based on the ARM instruction set. In one aspect of this disclosure, the instructions loaded into the general-purpose processor 402 may include code for receiving input streams from one or more sensors by a network device. The instructions loaded into the general-purpose processor 402 may also include code for generating a digital twin by the network device that models the environment of an area observed by the one or more sensors. The digital twin includes one or more objects detected based on the input streams. The instructions loaded into the general-purpose processor 402 may also include code for managing wireless communication signal beams by the network device, at least partially based on the digital twin, for communicating with at least one user equipment (UE) in the area observed by the one or more sensors.

[0074] In various aspects of this disclosure, the instructions loaded into the general-purpose processor 402 may include code for initiating wireless communication with a network device by the UE. The instructions loaded into the general-purpose processor 402 may also include code for receiving a wireless communication signal beam from the network device by the UE. This wireless communication signal beam is managed based on a digital twin and input streams from one or more sensors from one or more objects in the environment observed by the UE and modeled in the digital twin.

[0075] Deep learning architectures perform object recognition tasks by learning to represent inputs at progressively higher levels of abstraction in each layer, thereby constructing useful feature representations of the input data. In this way, deep learning addresses a major bottleneck of traditional machine learning. Before deep learning, machine learning methods for object recognition problems often relied heavily on human-designed features, possibly in conjunction with shallow classifiers. Shallow classifiers could be two-class linear classifiers, where a weighted sum of feature vector components is compared to a threshold to predict which class the input belongs to. Human-designed features could be templates or kernels customized for a specific problem domain by engineers with domain expertise. In contrast, while deep learning architectures can learn to represent features similar to those that human engineers might design, this requires training. Furthermore, deep networks can learn to represent and recognize novel types of features that humans might not have considered.

[0076] Deep learning architectures can learn hierarchical structures of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power at specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes in visual data or combinations of sounds in auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0077] Deep learning architectures perform particularly well when applied to problems with a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to identify features such as wheels, windshields, and others. These features can then be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0078] Neural networks can be designed with a variety of connectivity patterns. In feedforward networks, information is passed from lower layers to higher layers, where each neuron in a given layer communicates with neurons in higher layers. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In recurrent connections, the output from a neuron in a given layer can be passed to another neuron in the same layer. Recurrent architectures can help identify patterns across more than one block of input data that is sequentially fed to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. Networks with many feedback connections can be helpful when the recognition of higher-level concepts can aid in discerning specific lower-level features of the input.

[0079] The connections between layers in a neural network can be fully connected or locally connected. Figure 5A An example of a fully connected neural network 502 is illustrated. In the fully connected neural network 502, neurons in the first layer can transmit their outputs to each neuron in the second layer, so that each neuron in the second layer will receive inputs from each neuron in the first layer. Figure 5B An example of a locally connected neural network 504 is illustrated. In the locally connected neural network 504, neurons in a first layer can connect to a finite number of neurons in a second layer. More generally, the locally connected layers of the locally connected neural network 504 can be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connection strength can have different values ​​(e.g., 510, 512, 514, and 516). The connectivity pattern of locally connected layers can produce spatially different receptive fields in higher layers because neurons in higher layers in a given region can receive inputs that are tuned to the characteristics of a restricted portion of the total input to the network through training.

[0080] An example of a locally connected neural network is a convolutional neural network. Figure 5C An example of a convolutional neural network 506 is illustrated. Convolutional neural network 506 can be configured such that the connection strength associated with the input for each neuron in the second layer is shared (e.g., 508). Convolutional neural networks may be well-suited for problems where the spatial location of the input is meaningful.

[0081] One type of convolutional neural network is the deep convolutional network (DCN). Figure 5D A detailed example of a DCN 500 designed to recognize visual features from an image 526 input by an image capture device 530 (such as an in-vehicle camera) is illustrated. The DCN 500 in this example can be trained to identify traffic signs and the numbers provided on them. Of course, the DCN 500 can be trained for other tasks, such as identifying lane markings or traffic lights.

[0082] Supervised learning can be used to train the DCN 500. During training, an image (such as image 526 of a speed limit sign) can be presented to the DCN 500, and forward passes can then be computed to produce output 522. The DCN 500 may include a feature extraction part and a classification part. Upon receiving image 526, convolutional layer 532 may apply a convolutional kernel (not shown) to image 526 to generate a first set of feature maps 518. As an example, the convolutional kernel used for convolutional layer 532 may be a 5×5 kernel that generates 28×28 feature maps. In this example, since four different feature maps are generated in the first set of feature maps 518, four different convolutional kernels are applied to image 526 at convolutional layer 532. Convolutional kernels may also be referred to as filters or convolutional filters.

[0083] The first set of feature maps 518 can be subsampled by a max-pooling layer (not shown) to generate a second set of feature maps 520. The max-pooling layer reduces the size of the first set of feature maps 518. That is, the size of the second set of feature maps 520 (e.g., 14×14) is smaller than the size of the first set of feature maps 518 (e.g., 28×28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 520 can be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0084] exist Figure 5D In the example, the second set of feature maps 520 is convolved to generate a first feature vector 524. Furthermore, the first feature vector 524 is further convolved to generate a second feature vector 528. Each feature of the second feature vector 528 may include a number corresponding to a possible feature of image 526, such as "signature", "60", and "100". A softmax function (not shown) converts the numbers in the second feature vector 528 into probabilities. Thus, the output 522 of the DCN 500 can be the probability that image 526 includes one or more features.

[0085] In this example, the probabilities for "sign" and "60" in output 522 are higher than the probabilities for other numbers in output 522 (such as "30", "40", "50", "70", "80", "90", and "100"). Before training, output 522 generated by DCN 500 may be incorrect. Therefore, the error between output 522 and the target output can be calculated. The target output is the baseline ground truth (e.g., "sign" and "60") of image 526. The weights of DCN 500 can then be adjusted so that output 522 of DCN 500 is more closely aligned with the target output.

[0086] To adjust the weights, the learning algorithm computes the gradient vector of the weights. The gradient indicates how much the error will increase or decrease as the weights are adjusted. At the top layers, the gradient directly corresponds to the values ​​of the weights connecting the activated neurons in the penultimate layer to the neurons in the output layer. In lower layers, the gradient depends on the values ​​of the weights and the error gradient computed in the higher layers. The weights can then be adjusted to reduce the error. This method of adjusting weights is called "backpropagation" because it involves "passing backward" through the neural network.

[0087] In practice, the error gradient of the weights can be calculated using a small number of examples to make the calculated gradient approximate the true error gradient. This approximation method is called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, a new image (e.g., a speed limit sign in image 526) can be presented to the DCN 500, and output 522 can be generated through the forward pass of the DCN 500. This output can be considered as an inference or prediction of the DCN 500.

[0088] Deep Belief Networks (DBNs) are probabilistic models that include multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. DBNs are obtained by stacking layers of Restricted Boltzmann Machines (RBMs). An RBM is a type of artificial neural network that learns a probability distribution from a set of inputs. Because RBMs can learn a probability distribution without information about the class each input should be classified into, they are often used for unsupervised learning. Using a hybrid paradigm of supervised and unsupervised learning, the bottom RBM of a DBN can be trained unsupervised and used as a feature extractor, while the top RBM can be trained supervisedly (on the joint distribution of inputs from the previous layer and the target class) and used as a classifier.

[0089] DCN is a network of convolutional networks configured with additional pooling and normalization layers. DCN has achieved state-of-the-art performance on many tasks. DCN can be trained using supervised learning, where both the input and output targets are known for many paradigms and are used to modify the network's weights using gradient descent.

[0090] DCNs can be feedforward networks. Furthermore, as described above, connections from neurons in the first layer of a DCN to a set of neurons in the next higher layer are shared across neurons in the first layer. The feedforward and shared connections of a DCN can be used for fast processing. For example, the computational cost of a DCN may be much smaller than that of a similarly sized neural network that includes recurrent or feedback connections.

[0091] The processing at each layer of a convolutional network can be thought of as a spatially invariant template or base projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then a convolutional network trained on that input can be thought of as three-dimensional, with two spatial dimensions along the image's axes and a third dimension capturing color information. The output of the convolutional connections can be thought of as forming a feature map in the next layer, where each element in the feature map (e.g., 220) receives input from a range of neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values ​​in the feature map can be further processed using non-linear methods (e.g., rectified, max(0,x)). Values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and provides additional local invariance and dimensionality reduction. Normalization corresponding to whitening can also be applied through lateral inhibition between neurons in the feature map.

[0092] Figure 6 This is a block diagram illustrating the DCN 650. The DCN 650 can include multiple different types of layers based on connectivity and weight sharing. Figure 6 As shown, the DCN 650 includes convolutional blocks 654A and 654B. Each convolutional block in 654A and 654B can be configured using a convolutional layer (CONV) 656, a normalization layer (Lnorm) 658, and a max pooling layer (MAX POOL) 660. Although only two convolutional blocks 654A and 654B are shown, this disclosure is not limited thereto, and any number of convolutional blocks 654A and 654B can be included in the DCN 650 according to design preferences.

[0093] Convolutional layer 656 may include one or more convolutional filters that can be applied to the input data to generate feature maps. Normalization layer 658 may normalize the output of the convolutional filters. For example, normalization layer 658 may provide whitening or lateral suppression. Max pooling layer 660 may provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0094] For example, a parallel filter bank of a DCN can be loaded into a SOC 400 (e.g., Figure 4 The CPU 402 or GPU 404 of the SOC 400 can be used to achieve high performance and low power consumption. In an alternative implementation, a parallel filter bank can be loaded onto the DSP 406 or ISP 416 of the SOC 400. Furthermore, the DCN 650 can access other processing blocks that may exist on the SOC 400, such as the sensor processor 414 and navigation module 420, which are dedicated to sensors and navigation, respectively.

[0095] The DCN 650 may also include one or more fully connected layers 662 (FC1 and FC2). The DCN 650 may also include logistic regression (LR) layers 664. Weights (not shown) to be updated are present between each of the layers 656, 658, 660, 662, and 664 of the DCN 650. The output of each layer (e.g., 656, 658, 660, 662, and 664) can be used as input to the next layer in the DCN 650 (e.g., 656, 658, 660, 662, and 664) to learn hierarchical feature representations from the input data 652 (e.g., images, audio, video, sensor data, and / or other input data) supplied at the first convolutional block in convolutional block 654A. The output of the DCN 650 is a classification score 666 of the input data 652. The classification score 666 may be a set of probabilities, where each probability is a probability of the input data, including features from a feature set.

[0096] Figure 7 This is a high-level block diagram illustrating an example processing pipeline 700 for wireless communication according to various aspects of this disclosure. Reference Figure 7 Example processing pipeline 700 may receive a set 702 of visual input from one or more cameras. One or more processors 704 may be used to process the input to generate a digital twin 706 of an environment in which a wireless communication system is employed.

[0097] A digital twin can be thought of as a logical copy of a communication network (e.g., 5G new radio, 6G, or edge cloud), which can be used by applications to estimate key performance metrics of the communication system to make informed optimization decisions. Optimization use cases could include path selection and application rate coding, as well as the design of self-managed autonomous edge computing systems that can operate without human intervention.

[0098] The digital twin 706 can then help determine communication improvements 708. Examples of such communication improvements 708 may include, but are not limited to, beam prediction, handover prediction, obstruction detection, and reflected beam selection.

[0099] One problem in wireless communication systems may be beam prediction in high-frequency communication. High-frequency communication can be challenging because it involves pointing a beam. To increase power to the UE (e.g., 120) while reducing losses between the network device (e.g., 110) and the UE at high frequencies, it may be desirable to accurately locate the UE. However, accurately locating the UE (e.g., 120) can be difficult because the UE may be moving, and possibly at high speeds.

[0100] Figure 8 This is a diagram illustrating an example architecture 800 for beam prediction according to various aspects of this disclosure. Reference Figure 8 Network devices such as base station 110 may be configured for multi-sensor communication and high-fidelity hologram 804 (which may be referred to as a multi-sensor communication unit). Multi-sensor communication unit 804 may include one or more cameras, LiDAR sensors, IMU sensors, and Global Positioning Satellite (GPS) sensors. Data received via multi-sensor communication unit 804 may be aggregated and supplied to ML model 806. Each data mode in the data modes may have different characteristics. Corresponding time data sequences for different data modes may be provided to ML model 806. ML model 806 may process the aggregated data and use the aggregated data to determine beam prediction. Beam prediction can be used for beam steering towards a potentially moving UE 120. Beam steering involves changing the phase of the input signal on the radiating antenna element of base station 110 to steer the signal beam (e.g., a millimeter-wave frequency signal) toward the direction of UE 120.

[0101] In some respects, base station 110 can be initialized. The location of UE 120 can be determined, and a beam can be generated to initiate communication with UE 120. The movement and location of UE 120 can be tracked and used to update the beam for communication. In some respects, beam updating can be performed concurrently with location tracking.

[0102] Conventional methods for beamforming can be time-consuming and can result in significant energy and spectrum resource consumption. A conventional method for beamforming between base station 110 and UE 120 might employ an ellipse at base station 110, sweeping through all possible ellipses defined for UE 120, and then searching for beam pairs that reduce signal loss (e.g., those with beam peak directions). In contrast, architecture 800 can advantageously determine beams aided by multi-sensor data (e.g., via camera, LiDAR sensor, IMU sensor, or GPS sensor) to predict signal beams that reduce energy and resource consumption as well as latency.

[0103] Figure 9 This is a block diagram illustrating an example pipeline 900 for multimodal beam prediction according to various aspects of this disclosure. Reference Figure 9For example, the example pipeline 900 may receive input 902 from one or more cameras. Input 902 may include a two-dimensional (2D) image. However, this disclosure is not limited thereto, and input 902 may be provided from sensors such as LiDAR, radar, IMU, and / or other sensors. Input 902 may be folded and provided to three paths of the example pipeline 900. In a first path, objects included in the scene of input 902 may be classified by classification module 906. In some aspects, classification module 906 may perform object detection tasks, such as (but not limited to) YOLO object detection tasks, to detect objects in the scene of input 902.

[0104] In the second path of example pipeline 900, a mask can be generated. Input 902 can be provided to segmentation module 904 (e.g., a segment anything model (SAM)). Segmentation module 904 can perform a segment anything process on input 902, for example, to generate a static mask. Segmentation module 904 can provide image segmentation to generate a static mask for each object in the scene of input 902. The static mask can provide information about input 902, such as width, height, and location. The static mask can be used to distinguish objects in the scene of input 902.

[0105] The tracking module 910 may employ a static mask and perform an intersection-over-union (IoU) operation. IoU is an evaluation metric used for tasks such as segmentation, object detection, and tracking. IoU evaluates the proximity between the predicted bounding boxes of objects in a scene and the ground truth bounding boxes. A static mask may refer to a mask whose shape or location does not change over a relatively long time period (e.g., more than one hour). A static mask may be determined based on a predefined threshold (e.g., 0.85) such that if the mask has an IoU greater than or equal to the predefined threshold over a predefined time period (e.g., fifteen minutes, thirty minutes, one hour, or any other time period), the mask can be considered static. On the other hand, if the shape and / or location of the mask changes, the mask can be considered dynamic mask 912. In some aspects, dynamic mask 912 may have a similar width and height over time. Therefore, dynamic mask 912 can be used to track objects moving in an environment. In some examples, for instance, the location information of a dynamic mask 912 on an image sequence (e.g., input 902) can be used to track objects in an environment.

[0106] In the third path, input 902 can be provided to depth estimation module 908. Depth estimation module 908 can perform a single depth estimation process on the image of input 902. Single depth estimation can acquire a single image of input 902 and estimate the depth of each pixel in the image.

[0107] A single depth estimate (e.g., generated by depth estimation module 908) can be combined with a static mask (e.g., generated by segmentation module 904) and a dynamic mask 912 in object estimation module 914. Object estimation module 914 can filter the distance of the detected object from the camera for each object detected in the scene at input 902 based on the single depth estimate. Object estimation module 914 can use camera properties to determine angular information (e.g., angle of arrival) of the detected objects. For example, in various aspects, object estimation module 914 can determine angular information from vision (e.g., using a camera) by calculating the pinhole transform. The pinhole transform is a model describing the mathematical relationship between the coordinates of a point in three-dimensional (3D) space and its projection onto the image plane of a pinhole camera (e.g., input 902), where the camera aperture can be described as a point and there is no lens for focusing light. The pinhole transform can be given by: (1) (2) in Represents the angle of the camera object. Indicates inclusion A mask of pixel coordinates, where Is Field of view in direction. Pinhole transformation. It enables the reliable conversion of the angle of a camera object (e.g., a car or a person) to a true Global Positioning Satellite (GPS) angle (e.g., the center of the object). In some respects, due to the field of view... It may be unknown or it may be broken down into... Transformation, therefore, can be learned using neural networks. If an object is detected in the camera, angular information (e.g., (And / or GPS angles) can be used for beam prediction. The object estimation module 914 can also determine the type, size, location, and / or orientation of the detected object.

[0108] In various aspects, the object estimation module 914 may include an artificial neural network (e.g., 350). The object estimation module 914 may extract features of detected objects in the scene of input 902. Based on the determined object information (e.g., type, size, location, and / or orientation) and a 2D image of input 902, the object estimation module 914 may generate a 3D model of the detected objects to model the environment observed by the camera in input 902. The 3D model of the environment may be considered a digital twin 916.

[0109] In various respects, the digital twin 916 may include a subset of objects observed in input 902. Since the example pipeline 900 can be directed to wireless communication as a downstream task, the objects modeled by the object estimation module 914 can be pre-selected instead of generating a complete digital twin 916 (where all objects can be modeled). Therefore, the object estimation module 914 can limit the modeled objects to those that might affect communication parameters (e.g., objects that might potentially block wireless communication signals). In some respects, Kalman filtering, tracking, and additional processing may also be performed.

[0110] The digital twin 916 can then be used for beam management in wireless communications. The digital twin 916 can process aggregated data from one or more sensors (e.g., 804 or 902) and can utilize the aggregated sensor data (e.g., 804 or 902) to determine beam prediction. Beam prediction can be used for beam steering towards a UE 120, which can be stationary or mobile.

[0111] In some aspects, the digital twin 916 can be communicated to one or more UEs (e.g., 120). In this case, the UE (e.g., 120) can operate the digital twin for beam prediction to perform beam steering towards the network device (e.g., 110). Furthermore, the UE (e.g., 120) can feed back beam prediction information (e.g., angle information, positioning information, or a set of beams predicted to perform beam steering towards the network device 110) to the network device (e.g., 110). The feedback can be used by the digital twin 916 operating at the network device (e.g., 110) for example (but not limited to) updating object detection information, positioning information, angle information, sensor data developed at the UE (e.g., GPS data, image data, etc.), and beam prediction for beam steering towards each of the UEs. Therefore, in various aspects, the digital twin 916 of the UE (e.g., 120) and the digital twin 916 of the network device (e.g., 110) can collaboratively manage the beam for wireless communication between the UE (e.g., 120) and the network device (e.g., 110). Thus, the digital twin 916 can be used to predict the movement of an object (e.g., tracking) and proactively predict whether the beam (e.g., angular distribution or power) should be changed based on the predicted movement. Furthermore, the digital twin 916 of the receiving device (e.g., 110 or 120) can determine the beam to be measured. Subsequently, the digital twin 916 of the network device (e.g., 110) and / or the digital twin of the UE (e.g., 120) can (independently or collaboratively) decide whether to change the beam. Moreover, the digital twin 916 of the network device (e.g., 110) and / or the digital twin of the UE (e.g., 120) can directly determine (independently or collaboratively) when to change the beam.

[0112] Figure 10This is an example illustration of segmenting an input image into 1000 according to various aspects of this disclosure. For example... Figure 10 As shown, example segmentation 1000 may include objects detected in the scene of image 1002. For example, as referenced... Figure 9 As described, segmentation module 904 (e.g., Segment All Model (SAM)) can generate a static mask for each object detected in the image input to 902. Doing so provides a bounding box for each detected object. For example, as... Figure 10 As shown, detected objects may include static objects fixed in the environment, such as trees 1004a to 1004d, poles 1006a to 1006c, or a parked car 1008a. Furthermore, detected objects in example segment 1000 may include dynamic objects, such as objects that may be moving in the environment. Dynamic objects may, for example, include a car 1008b or a person 1010 on street 1012. However, object detection may be limited to objects that might affect communication parameters. Figure 10 As shown in the example, the detected objects may include objects that may block the signal transmission, but may not include objects such as roads (e.g., the bounding box of street 1012 is not shown).

[0113] Angle information and distance from the camera generating image 1002 can be determined for each detected object (e.g., trees 1004a to 1004d). In various aspects, for example, [the following methods can be used]. Figure 9 The classification module 903 is used to classify the detected objects. Objects can be segmented according to the classification, and tracing can be performed based on the classification. For example, if an object is classified as a tree or other object that may be stationary in the environment, tracing may not be performed. On the other hand, tracing can be performed for objects that may be dynamic (such as a car 1008b).

[0114] Figure 11 This is a block diagram illustrating an example process 1100 for object tracking and matching according to various aspects of this disclosure. Reference Figure 11 Example procedure 1100 may include determining the object angle and distance from the camera at box 1102. For example, as shown in the reference... Figure 9 As described, the object estimation module 914 can estimate the object distance and angle information of each detected object in the scene of the input image. For example, the angle can be estimated using pinhole transformation, and the distance can be measured using single depth estimation (e.g., generated by 908).

[0115] At box 1104, a circle method can be applied to achieve matching. Each detected object can be matched against a path on consecutive time steps in 3D. In some examples, a list of detected objects can be generated at time instance t=0. The list can include candidate objects to which tracing can be performed. Then, at time instance t=1, a scan is performed to detect new objects and identify which of the new objects detected at t=1 is closest to the object detected at t=0. If the new object is within a predefined radius (e.g., a radius of 5m for cars and 0.5m for people), such objects are considered matches and can be assigned to each other. If the new object is outside the predefined radius, such objects are considered distinct objects, and such new objects can be added to the list of objects.

[0116] At box 1106, if an object is inactive (e.g., does not move / is stationary) for a predefined time duration (e.g., N frames), the object will be deactivated to prevent it from being tracked (removed from the list).

[0117] At box 1108, the trajectory of the active object on the previous N samples can be smoothed, for example, using exponential smoothing or other smoothing techniques. At box 1110, the orientation of the object can be estimated based on the previous positioning estimate. At box 1112, the direction in which the object (e.g., a car) is moving (e.g., one side of the road) is determined based on the estimated orientation. Then, at box 1114, the positioning of each object in the list of objects is determined. For example, the object's positioning can be updated to include the known boundaries of the road.

[0118] In some respects, side information can be developed from other sensors (e.g., GPS data) and can be utilized to aid in object tracking. That is, side information can be used to filter and / or track objects more precisely. For example, if the detected object is a car (e.g., car 1008b), GPS data can be used to enforce tracking restrictions based on street characteristics (e.g., 1012) such as orientation, number of lanes, and speed information. Therefore, at box 1114, the detected object (e.g., car 1008b) can be forced to match to the nearest point on the street (e.g., 1012). For example, tracking of the car (e.g., 1008b) can be limited to traveling along the street (e.g., 1012) rather than on a sidewalk or waterway.

[0119] In some respects, one or more thresholds can be applied to track detected objects. For example, distance limits. It can be applied to tracking. Distance limits can be used to determine when communication parameters can be changed.

[0120] Figure 12This is a block diagram 1200 illustrating example rule-based decision-making based on wireless signal link degradation detection according to various aspects of this disclosure. In various aspects of this disclosure, digital twins (e.g., 916) can be used to predict link degradation or signal obstruction. Figure 12 In the example, antenna 1204 on the rear end of vehicle 1202 can be detected. When another object (e.g., a person or tree) is detected between antenna 1204 and the camera on the rear end of vehicle 1202 based on an image, the wireless signal between antenna 1204 and the network may experience link degradation. Furthermore, predictions of such link degradation and / or obstruction can be determined based on distance, angle, direction of travel, speed, and other information. The predictions can also indicate the location of the predicted link degradation and / or obstruction. Rule-based decision-making processes can be implemented based on, for example, the type or location of link degradation or obstruction.

[0121] like Figure 12 As shown in the example, the position of the center mask can be updated depending on the location of the source of link degradation, partial obstruction, or complete obstruction. The center mask can be used to update beam prediction for communication with vehicle 1202. Objects detected or predicted to pass between the camera and vehicle 1202 can be considered obstructions. In cases 11210 and 31230, the down-signal degrader 1206a and the center signal degrader 1206c can be detected or predicted, respectively. In cases 11210 and 31230, antenna 1204 may not be obstructed by degraders 1206a and 1206c. Thus, degraders 1206a and 1206c can be ignored, and the position of the center mask of vehicle 1202 can be preserved. On the other hand, in the case of signal degraders, such as in cases 2 1220 and 4 1240, since degraders 1206b and 1206d may degrade the signal link or may be predicted to degrade the signal link with antenna 1204, cases 2 1220 and 4 1240 can be considered signal link degraded cases. Therefore, the center mask can be updated. For example, the center mask... The calculation can be performed as follows: (3) in and They represent and The maximum value, and and They represent and The minimum value. Center mask. Points available A list. The center point of the mask can refer to the geometric center of the mask. For example, from the center mask, one can estimate... and minimum value and and The maximum value. Subsequently, such values ​​can be used to estimate the geometric center (e.g., by using a mask with a rectangular shape).

[0122] Figure 13 This is an illustration of example blocking predictions based on various aspects of this disclosure. Reference Figure 13 A network device (e.g., base station 110) can wirelessly communicate with user equipment (e.g., 120) of user 1312 in environment 1300. Wireless communication can be managed by the network device (e.g., 110) based on a digital twin (e.g., 916) and input 1304 from a camera 1302 located at the network device (e.g., 110). Bus 1308 can be observed in environment 1300 as indicated by input 1304 (e.g., 1304a, 1304b) and can be represented in the digital twin (e.g., 916). The location of bus 1308 can be tracked based on input 1304a at time t-1 and input 1304b at time t, combined with side information (e.g., GPS data). Additionally, based on the relative positioning of the UE (e.g., 120) and the tracking information for bus 1308, machine learning model 1310 can predict future obstruction of the wireless communication signal between the network device (e.g., 110) and the UE (e.g., 120). In some respects, the machine learning model 1310 may be located on a network device (e.g., 110) or a cloud server.

[0123] In response to a predicted obstruction (e.g., bus 1308), network devices (e.g., 110) and / or UEs (e.g., 120) may implement beam management actions to support wireless communication between the network device (e.g., 110) and the UE (e.g., 120) using a corresponding digital twin 916. For example, beam management actions may include (but are not limited to) active handover to another network device, active resource management, or beam switching (e.g., adaptation).

[0124] As indicated above, Figures 4 to 13 This is provided as an example. Other examples are available with reference to [the relevant information]. Figures 4 to 13 The examples described are different.

[0125] Figure 14 This is a flowchart illustrating an example process 1400 for beam management by a network device using digital twins and input streams from sensors, according to various aspects of this disclosure. (See also:) Figures 1 to 13 Example process 1400 may be executed by one or more processors such as CPU (e.g., 402), GPU (e.g., 404), DSP (e.g., 406) and / or NPU (e.g., 408)).

[0126] At box 1402, one or more processors receive input streams from one or more sensors via a network device. For example, as referenced... Figure 8 As described, network devices such as base station 110 can be configured for multi-sensor communication and high-fidelity hologram 804 (which may be referred to as a multi-sensor communication unit). Multi-sensor communication unit 804 may include one or more cameras, LiDAR sensors, IMU sensors, and Global Positioning Satellite (GPS) sensors. Data received via multi-sensor communication unit 804 can be aggregated and fed to ML model 806. Each data modality in the data modalities may have different characteristics. In various aspects, element-wise operations such as (but not limited to) summation, multiplication, feature stacking, attention mechanisms, or other fusion techniques can also be used to generate a fusion of inputs from different modalities.

[0127] At box 1404, one or more processors, via a network device, generate a digital twin that models the environment of an area observed by one or more sensors. The digital twin includes one or more objects detected based on an input stream. See, for example, the reference... Figure 9 As described, the object estimation module 914 can extract features of detected objects in the scene of input 902. Based on the determined object information (e.g., type, size, location, and / or orientation) and the 2D image of input 902, the object estimation module 914 can generate a 3D model of the detected objects to model the environment observed by the camera in input 902. The 3D model of the environment can be considered a digital twin 916.

[0128] At box 1406, one or more processors, at least partially based on digital twins, manage wireless communication signal beams by a network device for communication with at least one user equipment (UE) in an area observed by one or more sensors. For example, as Figure 9 As described, the digital twin 916 can then be used for beam management in wireless communications. The digital twin 916 can process aggregated data from one or more sensors (e.g., 804 or 902) and can utilize the aggregated sensor data (e.g., 804 or 902) to determine beam prediction. Beam prediction can be used for beam steering towards a UE 120, which may be stationary or mobile.

[0129] Figure 15 This is a flowchart illustrating an example process 1500 for wireless communication by user equipment using a beam estimated based on digital twins and input streams from sensors, according to various aspects of this disclosure. (See also:) Figures 1 to 13 Example process 1500 can be executed by one or more processors such as CPU (e.g., 402), GPU (e.g., 404), DSP (e.g., 406) and / or NPU (e.g., 408)).

[0130] At block 1502, one or more processors are initiated by the UE to communicate wirelessly with a network device. UE 120 may include beam management module 140. Beam management module 140 can determine beam updates for wireless communication with the network device. For example, using beam management module 140, the UE (e.g., 120) can initiate wireless communication with a network device (e.g., 110 / 130).

[0131] At box 1504, one or more processors receive wireless communication signal beams from the network device by the UE. These wireless communication signal beams are managed based on a digital twin and input streams from one or more sensors from one or more objects in the environment observed by the UE and modeled within the digital twin. For example, as... Figure 9 As described, the digital twin 916 can then be used for beam management in wireless communications. The digital twin 916 can process aggregated data from one or more sensors (e.g., 804 or 902) and can utilize the aggregated sensor data (e.g., 804 or 902) to determine beam prediction.

[0132] Specific implementation examples are provided in the following numbered clauses.

[0133] 1. An apparatus, the apparatus comprising: At least one memory; and At least one processor, coupled to the at least one memory, is configured to: The network device receives the input stream from one or more sensors. The network device generates a digital twin modeling the environment of the area observed by the one or more sensors, the digital twin including one or more objects detected based on the input stream; and The network device manages wireless communication signal beams, at least in part, based on the digital twin, for communication with at least one user equipment (UE) in the area observed by the one or more sensors.

[0134] 2. The apparatus according to Clause 1, wherein the at least one processor is further configured to generate a predictive beam for wireless communication with the at least one UE by the network device using the digital twin based on the input stream from the one or more sensors.

[0135] 3. The apparatus according to clause 1 or 2, wherein the at least one processor is further configured to: The network device generates a second predicted beam for uplink communication between the at least one UE and the network device; and The network device sends an indication of the second predicted beam to the at least one UE.

[0136] 4. The apparatus according to any one of Clauses 1 to 3, wherein the one or more sensors include one or more of the following: a camera, an inertial measurement unit (IMU) sensor, a light detection and ranging (LiDAR) sensor, and a global positioning satellite (GPS) sensor.

[0137] 5. The apparatus according to any one of clauses 1 to 4, wherein the network device manages the wireless communication signal beam based on the fusion of information from the input stream.

[0138] 6. The apparatus according to any one of clauses 1 to 5, wherein the at least one processor is further configured to: The network device tracks the location of objects among the one or more objects detected based on the input stream, based on the object type. The network device updates the digital twin based on the tracking; and The network device adapts the wireless communication signal beam based on the updated digital twin.

[0139] 7. The apparatus according to any one of clauses 1 to 6, wherein the at least one processor is further configured to predict, by the network device, an obstruction to the wireless communication signal of the first UE based on a first position of an object relative to a second position of the antenna of the first UE among the at least one UE and the digital twin.

[0140] 8. The apparatus according to any one of clauses 1 to 7, wherein one or more of the first UE or the object is moving.

[0141] 9. The apparatus according to any one of clauses 1 to 8, wherein the at least one processor is further configured to: The network device determines the link degradation caused by the object based on the degree of overlap between the first and second locations; and The network device adapts its wireless communication with the at least one UE based on a link degradation type, and the wireless communication is adapted by one of the following: adapting the wireless communication signal beam, changing the frequency used for the wireless communication, or initiating a handover.

[0142] 10. The apparatus according to any one of clauses 1 to 9, wherein the at least one processor is further configured to: Determine the channel state associated with the wireless communication signal beam; and The channel state is supplied as feedback to the digital twin.

[0143] 11. An apparatus comprising: At least one memory; and At least one processor, coupled to the at least one memory, is configured to: Wireless communication initiated by the user equipment (UE) with network equipment; and The UE receives a wireless communication signal beam from the network device, the wireless communication signal beam being managed based on a digital twin and input streams from one or more sensors from one or more objects observing the UE and the environment modeled in the digital twin.

[0144] 12. The apparatus according to Clause 11, wherein the wireless communication signal beam is predicted based on the input stream from the one or more sensors.

[0145] 13. The apparatus according to clause 11 or 12, wherein the at least one processor is further configured to: Receive from the network device an indication of a predicted beam for uplink communication with the network device, the predicted beam being determined using the digital twin based on the input stream from the one or more sensors; and The UE generates wireless communication signals based on the predicted beam to communicate with the network device.

[0146] 14. The apparatus according to any one of Clauses 11 to 13, wherein the one or more sensors include one or more of the following: a camera, an inertial measurement unit (IMU) sensor, a light detection and ranging (LiDAR) sensor, and a global positioning satellite (GPS) sensor.

[0147] 15. The apparatus according to any one of clauses 11 to 14, wherein the wireless communication signal beam is managed based on the fusion of information from the input stream.

[0148] 16. The apparatus according to any one of clauses 11 to 15, wherein the wireless communication signal beam is selected from a set of multiple candidate beams determined by the digital twin.

[0149] 17. A processor-implemented method implemented by a network device, the processor-implemented method comprising: The network device receives input streams from one or more sensors; The network device generates a digital twin modeling the environment of the area observed by the one or more sensors, the digital twin including one or more objects detected based on the input stream; and The network device manages wireless communication signal beams, at least in part, based on the digital twin, for communication with at least one user equipment (UE) in the area observed by the one or more sensors.

[0150] 18. The processor-implemented method according to Clause 17, further comprising: The network device uses the digital twin to generate a predictive beam for wireless communication with the at least one UE based on the input stream from the one or more sensors.

[0151] 19. The processor-implemented method according to clause 17 or 18, further comprising: The network device generates a second predicted beam for uplink communication between the at least one UE and the network device; and The network device sends an indication of the second predicted beam to the at least one UE.

[0152] 20. A method implemented by a processor according to any one of Clauses 17 to 19, wherein the network device manages the wireless communication signal beam based on the fusion of information from the input stream.

[0153] 21. The processor-implemented method according to any one of clauses 17 to 20, wherein the processor-implemented method further comprises: The network device tracks the location of objects among the one or more objects detected based on the input stream, based on the object type. The network device updates the digital twin based on the tracking; and The network device adapts the wireless communication signal beam based on the updated digital twin.

[0154] 22. The method implemented by the processor according to any one of Clauses 17 to 21, the method further comprising the network device predicting an obstruction to the wireless communication signal of the first UE based on a first position of an object relative to a second position of an antenna of the first UE among the at least one UE and the digital twin.

[0155] 23. The method implemented by the processor according to any one of Clauses 17 to 22, wherein one or more of the first UE or the objects are moving.

[0156] 24. The processor-implemented method according to any one of clauses 17 to 23, wherein the processor-implemented method further comprises: The network device determines the link degradation caused by the object based on the degree of overlap between the first and second locations; and The network device adapts its wireless communication with the at least one UE based on a link degradation type, and the wireless communication is adapted by one of the following: adapting the wireless communication signal beam, changing the frequency used for the wireless communication, or initiating a handover.

[0157] 25. The processor-implemented method according to any one of clauses 17 to 24, wherein the processor-implemented method further comprises: Determine the channel state associated with the wireless communication signal beam; and The channel state is supplied as feedback to the digital twin.

[0158] 26. A processor-implemented method implemented by a user equipment (UE), the processor-implemented method comprising: Wireless communication initiated by the user equipment (UE) with network equipment; and The UE receives a wireless communication signal beam from the network device, the wireless communication signal beam being managed based on a digital twin and input streams from one or more sensors from one or more objects observing the UE and the environment modeled in the digital twin.

[0159] 27. The processor-implemented method according to Clause 26, wherein the wireless communication signal beam is predicted based on the input stream from the one or more sensors.

[0160] 28. The processor-implemented method according to clause 26 or 27, further comprising: Receive from the network device an indication of a predicted beam for uplink communication with the network device, the predicted beam being determined using the digital twin based on the input stream from the one or more sensors; and The UE generates wireless communication signals based on the predicted beam to communicate with the network device.

[0161] 29. The method implemented by the processor according to any one of Clauses 26 to 28, wherein the wireless communication signal beam is managed based on the fusion of information from the input stream.

[0162] 30. The method implemented by the processor according to any one of Clauses 26 to 29, wherein the wireless communication signal beam is selected from a set of multiple candidate beams determined by the digital twin.

[0163] The foregoing disclosure provides examples and descriptions, but is not intended to be exhaustive or to limit the aspects to the precise forms disclosed. Modifications and variations may be made based on the foregoing disclosure, or from practice in various aspects.

[0164] As used, the term "component" is intended to be interpreted broadly as hardware, firmware, and / or a combination of hardware and software. As used, a processor is implemented using hardware, firmware, and / or a combination of hardware and software.

[0165] The threshold is used to describe certain aspects. As used, depending on the context, meeting the threshold can mean that the value is greater than the threshold, greater than or equal to the threshold, less than the threshold, less than or equal to the threshold, equal to the threshold, not equal to the threshold, etc.

[0166] It will be apparent that the described systems and / or methods can be implemented in various forms, including hardware, firmware, and / or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limiting in any way. Therefore, since the operation and performance of these systems and / or methods are described without reference to specific software code, it should be understood that the software and hardware used to implement these systems and / or methods can be designed, at least in part, based on this description.

[0167] Although specific combinations of features are set forth in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of various aspects. In fact, many of these features can be combined in ways not specifically set forth in the claims and / or not disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of various aspects includes each dependent claim combined with every other claim in the claim set. The phrase referring to “at least one of” the list of items means any combination of those items, including single members. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination having multiple of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0168] The elements, actions, or instructions used should not be interpreted as critical or necessary unless explicitly stated otherwise. Furthermore, as used, the articles “a” and “one” are intended to include one or more items and are used interchangeably with “one or more.” Additionally, as used, the terms “set” and “group” are intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and are used interchangeably with “one or more.” If only one item is desired, the phrase “only one” or similar terminology will be used. Furthermore, as used, the terms “have,” “possess,” “have,” etc., are intended to be open-ended terms. Additionally, the phrase “based on” is intended to mean “at least partially based on” unless otherwise explicitly stated.

Claims

1. An apparatus, the apparatus comprising: At least one memory; and At least one processor, coupled to the at least one memory, is configured to: The network device receives the input stream from one or more sensors. The network device generates a digital twin that models the environment of the area observed by the one or more sensors, the digital twin including one or more objects detected based on the input stream; as well as The network device manages wireless communication signal beams, at least in part, based on the digital twin, for communication with at least one user equipment (UE) in the area observed by the one or more sensors.

2. The apparatus of claim 1, wherein the at least one processor is further configured to generate a predictive beam for wireless communication with the at least one UE using the digital twin based on the input stream from the one or more sensors.

3. The apparatus of claim 1, wherein the at least one processor is further configured to: The network device generates a second predicted beam for uplink communication between the at least one UE and the network device; and The network device sends an indication of the second predicted beam to the at least one UE.

4. The apparatus of claim 1, wherein the one or more sensors comprise one or more of the following: a camera, an inertial measurement unit (IMU) sensor, a light detection and ranging (LiDAR) sensor, and a global positioning satellite (GPS) sensor.

5. The apparatus of claim 1, wherein the network device manages the wireless communication signal beam based on the fusion of information from the input stream.

6. The apparatus of claim 1, wherein the at least one processor is further configured to: The network device tracks the location of objects among the one or more objects detected based on the input stream, based on the object type. The network device updates the digital twin based on the tracking; as well as The network device adapts the wireless communication signal beam based on the updated digital twin.

7. The apparatus of claim 1, wherein the at least one processor is further configured to predict, by the network device, an obstruction to the wireless communication signal of the first UE based on a first position of an object relative to a second position of the antenna of the first UE among the at least one UE and the digital twin.

8. The apparatus of claim 7, wherein one or more of the first UE or the object are moving.

9. The apparatus of claim 7, wherein the at least one processor is further configured to: The network device determines the link degradation caused by the object based on the degree of overlap between the first and second locations; and The network device adapts its wireless communication with the at least one UE based on a link degradation type, and the wireless communication is adapted by one of the following: adapting the wireless communication signal beam, changing the frequency used for the wireless communication, or initiating a handover.

10. The apparatus of claim 1, wherein the at least one processor is further configured to: Determine the channel state associated with the wireless communication signal beam; and The channel state is supplied as feedback to the digital twin.

11. An apparatus comprising: At least one memory; and At least one processor, coupled to the at least one memory, is configured to: Wireless communication between the user equipment (UE) and network devices; as well as The UE receives a wireless communication signal beam from the network device, the wireless communication signal beam being managed based on a digital twin and input streams from one or more sensors from one or more objects observing the UE and the environment modeled in the digital twin.

12. The apparatus of claim 11, wherein the wireless communication signal beam is predicted based on the input stream from the one or more sensors.

13. The apparatus of claim 11, wherein the at least one processor is further configured to: Receive from the network device an indication of a predicted beam for uplink communication with the network device, the predicted beam being determined using the digital twin based on the input stream from the one or more sensors; and The UE generates wireless communication signals based on the predicted beam to communicate with the network device.

14. The apparatus of claim 11, wherein the one or more sensors comprise one or more of the following: a camera, an inertial measurement unit (IMU) sensor, a light detection and ranging (LiDAR) sensor, and a global positioning satellite (GPS) sensor.

15. The apparatus of claim 11, wherein the wireless communication signal beam is managed based on the fusion of information from the input stream.

16. The apparatus of claim 11, wherein the wireless communication signal beam is selected from a set of multiple candidate beams determined by the digital twin.

17. A processor-implemented method implemented by a network device, the processor-implemented method comprising: The network device receives input streams from one or more sensors; The network device generates a digital twin that models the environment of the area observed by the one or more sensors, the digital twin including one or more objects detected based on the input stream; as well as The network device manages wireless communication signal beams, at least in part, based on the digital twin, for communication with at least one user equipment (UE) in the area observed by the one or more sensors.

18. The processor-implemented method according to claim 17, further comprising: The network device uses the digital twin to generate a predictive beam for wireless communication with the at least one UE based on the input stream from the one or more sensors.

19. The processor-implemented method according to claim 17, further comprising: The network device generates a second predicted beam for uplink communication between the at least one UE and the network device. as well as The network device sends an indication of the second predicted beam to the at least one UE.

20. The processor-implemented method of claim 17, wherein the network device manages the wireless communication signal beam based on the fusion of information from the input stream.

21. The processor-implemented method according to claim 17, further comprising: The network device tracks the location of objects among the one or more objects detected based on the input stream, based on the object type. The network device updates the digital twin based on the tracking; as well as The network device adapts the wireless communication signal beam based on the updated digital twin.

22. The processor-implemented method of claim 17, further comprising the network device predicting an obstruction to the wireless communication signal of the first UE based on a first position of an object relative to a second position of an antenna of the first UE among the at least one UE and the digital twin.

23. The processor-implemented method of claim 22, wherein one or more of the first UE or the objects are moving.

24. The processor-implemented method according to claim 22, further comprising: The network device determines the link degradation caused by the object based on the degree of overlap between the first location and the second location; as well as The network device adapts its wireless communication with the at least one UE based on a link degradation type, and the wireless communication is adapted by one of the following: adapting the wireless communication signal beam, changing the frequency used for the wireless communication, or initiating a handover.

25. The processor-implemented method according to claim 17, further comprising: Determine the channel state associated with the wireless communication signal beam; as well as The channel state is supplied as feedback to the digital twin.

26. A processor-implemented method implemented by a user equipment (UE), the processor-implemented method comprising: Wireless communication between the user equipment (UE) and network devices; as well as The UE receives a wireless communication signal beam from the network device, the wireless communication signal beam being managed based on a digital twin and input streams from one or more sensors from one or more objects observing the UE and the environment modeled in the digital twin.

27. The processor-implemented method of claim 26, wherein the wireless communication signal beam is predicted based on the input stream from the one or more sensors.

28. The processor-implemented method according to claim 26, further comprising: Receive from the network device an indication of a predicted beam for uplink communication with the network device, the predicted beam being determined using the digital twin based on the input stream from the one or more sensors; as well as The UE generates wireless communication signals based on the predicted beam to communicate with the network device.

29. The processor-implemented method of claim 26, wherein the wireless communication signal beam is managed based on the fusion of information from the input stream.

30. The processor-implemented method of claim 26, wherein the wireless communication signal beam is selected from a set of multiple candidate beams determined by the digital twin.

Images