Method for implementing virtual personal assistant mixed reality service through integrated communication and computing

Abstract

In one aspect of the disclosure, a method, a computer-readable medium, and an apparatus are provided. The method is performed by a user device for providing an artificial intelligence (AI) service. The user device receives a user query for the AI service. The user device evaluates one or more conditions. The user device determines, based on the evaluation, an operation mode from a plurality of operation modes for performing the AI service. The plurality of operation modes can include a local-only mode and at least one split-computing mode. The user device performs the AI service according to the determined operation mode to provide a response to the user query. In the local-only mode, the AI service can be performed completely on the user device. In the at least one split-computing mode, at least a portion of the AI service can be offloaded to a server.

Description

[0001] Cross-references

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 709,637, entitled “Method for realizing a virtual personal assistant mixed reality service using integrated communications and computing,” filed October 21, 2024, the entire contents of which are hereby expressly incorporated by reference. Technical Field

[0003] This disclosure generally relates to wireless communications, and more specifically, to techniques for implementing virtual personal assistant mixed reality services using integrated communications and computing. Background Technology

[0004] The content in this section is merely background information related to the disclosure of this invention and does not constitute prior art.

[0005] Wireless communication systems are widely deployed to provide a variety of telecommunications services, such as telephone, video, data, messaging, and broadcasting. Typical wireless communication systems employ multiple access technologies, which support communication for multiple users by sharing available system resources. Examples of such multiple access technologies include code division multiple access (CDMA) systems, time division multiple access (TDMA) systems, frequency division multiple access (FDMA) systems, orthogonal frequency division multiple access (OFDMA) systems, single-carrier frequency division multiple access (SC-FDMA) systems, and time division synchronous code division multiple access (TD-SCDMA) systems.

[0006] These multiple access technologies have been adopted in various telecommunications standards to provide a common protocol enabling different wireless devices to communicate across cities, countries, regions, and even globally. An example telecommunications standard is 5G New Radio (NR). 5G NR is part of the continuous evolution of mobile broadband promoted by the 3rd Generation Partnership Project (3GPP) to meet new requirements related to latency, reliability, security, scalability (e.g., the Internet of Things (IoT)), and others. Some aspects of 5G NR may be based on the 4G Long Term Evolution (LTE) standard. There is a need for further improvements to 5G NR technology. These improvements can also be applied to other multiple access technologies and telecommunications standards that adopt them. Summary of the Invention

[0007] The following content presents one or more aspects in a simplified form to provide a basic understanding of these aspects. This summary is not a comprehensive overview of all hypothetical aspects, nor is it intended to identify key or essential elements of all aspects, nor to define the scope of any or all aspects. Its sole purpose is to present certain concepts of one or more aspects in a simplified form as a prelude to a more detailed description thereafter.

[0008] In one aspect of this disclosure, a method, computer-readable medium, and apparatus are provided. The method, performed by a user equipment, is used to provide an artificial intelligence (AI) service. The user equipment receives a user query for the AI ​​service. The user equipment evaluates one or more conditions. These conditions may include at least one of the following: local resources of the user equipment, network connectivity resources between the user equipment and a server, or availability of information on the user equipment for responding to the user query. Based on the evaluation, the user equipment determines an operating mode from a plurality of operating modes for performing the AI ​​service. The plurality of operating modes may include a local-only mode and at least one split-computation mode. The user equipment performs the AI ​​service according to the determined operating mode to provide a response to the user query. In the local-only mode, the AI ​​service can be performed entirely on the user equipment. In at least one split-computation mode, at least a portion of the AI ​​service can be offloaded to the server.

[0009] To achieve the foregoing and related objectives, one or more aspects include the following features, which will be described in detail below and specifically pointed out in the claims. The following description and accompanying drawings illustrate certain exemplary features of one or more aspects. However, these features only illustrate a portion of how the principles of the various aspects can be applied, and this description is intended to include all such aspects and their equivalents. Attached Figure Description

[0010] Figure 1 This is a schematic diagram illustrating an example wireless communication system and access network.

[0011] Figure 2 This is a schematic diagram illustrating communication between a base station and user equipment in the access network.

[0012] Figure 3 Example of a distributed access network's logical architecture.

[0013] Figure 4 Example of a distributed access network physical architecture.

[0014] Figure 5 A schematic diagram outlining the design of a distributed artificial intelligence service platform.

[0015] Figure 6 A schematic diagram of the deployment architecture for virtual personal assistant mixed reality services in the retail industry.

[0016] Figure 7 This is a schematic diagram of a centralized service agent message flow with customer site network connectivity.

[0017] Figure 8 A schematic diagram of the centralized service proxy message flow for networks without customer premises connections.

[0018] Figure 9 This is a message flow diagram for a service that does not use a centralized service agent and has no customer site network connection.

[0019] Figure 10 A schematic diagram of a mixed reality service designed for virtual personal assistants.

[0020] Figure 11 This is a diagram illustrating a scenario where the network is available, user equipment resources are insufficient, and / or information is unavailable locally.

[0021] Figure 12 This is a diagram illustrating a scenario where the network is poor, the user's device resources are sufficient, and the information is unavailable locally.

[0022] Figure 13 This is a diagram illustrating a scenario where user devices have sufficient resources and information is available locally.

[0023] Figure 14 The example flowchart demonstrates how integrated communication and computing can be used to implement a virtual personal assistant mixed reality service.

[0024] Figure 15 Another flowchart illustrating how integrated communication and computing are used to implement a virtual personal assistant mixed reality service.

[0025] Figure 16 The example is a flowchart for preparing a service model for a virtual assistant service. Detailed Implementation

[0026] The detailed descriptions listed below, in conjunction with the accompanying drawings, are intended to describe various configurations and are not intended to represent the only configurations in which the concepts described herein can be practiced. The detailed descriptions contain specific details to provide a thorough understanding of the various concepts. However, those skilled in the art will understand that these concepts can be practiced without these specific details. In some cases, to avoid obscuring these concepts, known structures and components are shown in block diagram form.

[0027] This document introduces several aspects of telecommunications systems using various apparatuses and methods. These apparatuses and methods will be described in detail below and illustrated with accompanying drawings as various modules, components, circuits, processes, algorithms, etc. (collectively referred to as "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether an element is implemented in hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0028] For example, a single element, any part of an element, or any combination of elements can be implemented as a “processing system” containing one or more processors. Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, system-on-a-chip (SoCs), baseband processors, field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described in this specification. One or more processors in a processing system can execute software. Software should be understood broadly as instructions, instruction sets, code, code segments, program code, programs, subroutines, software components, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description languages, or other names.

[0029] Therefore, in one or more example aspects, the functionality can be implemented by hardware, software, or any combination thereof. If implemented in software, these functions can be stored or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media. Storage media can be any available medium accessible to a computer. For example, but not limited to, such computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), optical disc storage, magnetic disk storage, other magnetic storage devices, combinations of computer-readable media of the above types, or any medium that can be used to store computer-executable code in the form of instructions or data structures and is accessible to a computer.

[0030] Figure 1 This is a schematic diagram illustrating a wireless communication system and access network 100. The wireless communication system (also known as a Wireless Wide Area Network (WWAN)) includes a base station 102, user equipment (UE) 104, an evolved packet core (EPC) 160, and another core network 190 (e.g., a 5G core (5GC)). The base station 102 may include macro cells (high-power cellular base stations) and / or small cells (low-power cellular base stations). Macro cells include base stations. Small cells include femtocells, picocells, and microcells.

[0031] Base station 102 configured as 4G LTE (collectively referred to as Evolved Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access Network (E-UTRAN)) can interface with EPC 160 via backhaul link 132 (e.g., SI interface). Base station 102 configured as 5G NR (collectively referred to as Next Generation Radio Access Network (NG-RAN)) can interface with core network 190 via backhaul link 184. Among other functions, base station 102 can perform one or more of the following functions: user data transmission, radio channel encryption and decryption, integrity protection, header compression, mobility control functions (e.g., handover, dual connectivity), inter-cell interference coordination, connection establishment and release, load balancing, non-access stratum (NAS) message distribution, NAS node selection, synchronization, radio access network (RAN) sharing, multimedia broadcast multicast service (MBMS), user and device tracking, RAN information management (RIM), paging, location, and warning message delivery. Base stations 102 can communicate with each other directly or indirectly (e.g., via EPC 160 or core network 190) via backhaul link 134 (e.g., X2 interface). Backhaul link 134 can be wired or wireless.

[0032] Base station 102 can wirelessly communicate with UE 104. Each base station 102 can provide communication coverage for a corresponding geographic coverage area 110. Overlapping geographic coverage areas 110 may exist. For example, small cell 102' may have a coverage area 110' that overlaps with the coverage areas 110 of one or more macro base stations 102. A network containing small cells and macro cells can be referred to as a heterogeneous network. Heterogeneous networks may also include evolved home nodes (eNBs) (HeNBs), which can provide services to a restricted group called a closed subscriber group (CSG). The communication link 120 between base station 102 and UE 104 may include uplink (UL) (also known as reverse link) transmission from UE 104 to base station 102 and / or downlink (DL) (also known as forward link) transmission from base station 102 to UE 104. Communication link 120 may employ multiple-input multiple-output (MIMO) antenna technology, including spatial multiplexing, beamforming, and / or transmit diversity. Communication links can be established via one or more carriers. Base station 102 / UE 104 can use a spectrum with a maximum bandwidth of 7 MHz per carrier (e.g., 5, 10, 15, 20, 100, 400 MHz, etc.), allocating a total of Yx MHz (x component carriers) for transmission in carrier aggregation in each direction. Carriers can be adjacent or non-adjacent. Carrier allocation in the DL and UL directions can be asymmetrical (e.g., more or fewer carriers allocated to DL than to UL). Component carriers can include primary component carriers and one or more secondary component carriers. The primary component carrier can be referred to as the primary cell (PCell), and the secondary component carriers can be referred to as secondary cells (SCells).

[0033] Some UEs 104 can communicate with each other via device-to-device (D2D) communication link 158. D2D communication link 158 can use DL / UL WWAN spectrum. D2D communication link 158 can use one or more sidelink channels, such as the physical sidelink broadcast channel (PSBCH), physical sidelink discovery channel (PSDCH), physical sidelink shared channel (PSSCH), and physical sidelink control channel (PSCCH). D2D communication can be performed through various wireless D2D communication systems, such as FlashLinQ, WiMedia, Bluetooth, ZigBee, Wi-Fi based on the IEEE 802.11 standard, LTE, or New Radio (NR).

[0034] The wireless communication system may also include a Wi-Fi access point (AP) 150 that communicates with the Wi-Fi station (STA) 152 via a communication link 154 in the 5 GHz unlicensed spectrum. When communicating in the unlicensed spectrum, the STA 152 / AP 150 may perform a clear channel assessment (CCA) before communication to determine whether the channel is available.

[0035] Small cell 102' can operate in licensed and / or unlicensed spectrum. When operating in unlicensed spectrum, small cell 102' can employ New Radio (NR) and use the same 5 GHz unlicensed spectrum as Wi-Fi access point 150. Employing NR in unlicensed spectrum can improve the coverage and / or increase the capacity of the access network.

[0036] Base station 102, whether a small base station 102' or a large base station (e.g., a macro base station), may include evolved Node B (eNB), gNodeB (gNB), or other types of base stations. Some base stations, such as gNB 180, can communicate with user equipment (UE) 104 in conventional sub-6 GHz spectrum, millimeter wave (mmW) frequencies, and / or near-millimeter wave frequencies. When gNB 180 operates at millimeter wave or near-millimeter wave frequencies, gNB 180 may be referred to as a millimeter wave base station. Extremely high frequency (EHF) is a portion of the radio frequency (RF) spectrum in the electromagnetic spectrum. EHF ranges from 30 GHz to 300 GHz, with wavelengths between 1 mm and 10 mm. Radio waves in this band may be referred to as millimeter waves. Near-millimeter waves can extend down to 3 GHz with wavelengths of 100 mm. The super high frequency (SHF) band ranges from 3 GHz to 30 GHz, also known as centimeter waves. Communication using millimeter-wave / near-millimeter-wave radio frequency bands (e.g., 3 GHz-300 GHz) suffers from extremely high path loss and short range. Millimeter-wave base station 180 can utilize beamforming 182 to communicate with UE 104 to compensate for the extremely high path loss and short range.

[0037] Base station 180 can transmit beamforming signals to UE 104 in one or more transmit directions 108a. UE 104 can receive beamforming signals from base station 180 in one or more receive directions 108b. UE 104 can also transmit beamforming signals to base station 180 in one or more transmit directions. Base station 180 can receive beamforming signals from UE 104 in one or more receive directions. Base station 180 / UE 104 can perform beamforming training to determine the optimal receive and transmit directions for each base station 180 / UE 104. The transmit and receive directions of base station 180 can be the same or different. The transmit and receive directions of UE 104 can be the same or different.

[0038] The evolved packet core (EPC) 160 may include a Mobility Management Entity (MME) 162, other MMEs 164, a Serving Gateway 166, a Multimedia Broadcast Multicast Service (MBMS) Gateway 168, a Broadcast Multicast Service Center (BM-SC) 170, and a Packet Data Network (PDN) Gateway 172. MME 162 can communicate with the Home Subscriber Server (HSS) 174. MME 162 is the control node that handles signaling between UE 104 and EPC 160. Typically, MME 162 provides bearer and connection management. All user Internet Protocol (IP) packets are transmitted through the Serving Gateway 166, which is itself connected to the PDN Gateway 172. The PDN Gateway 172 provides UE IP address allocation and other functions. The PDN Gateway 172 and BM-SC 170 are connected to the IP service 176. IP service 176 may include the Internet, intranet, IP Multimedia Subsystem (IMS), Packet Switched Streaming Service (PS Streaming Service), and / or other IP services. BM-SC 170 can provide and deliver MBMS user services. BM-SC 170 can serve as an entry point for MBMS transmission by content providers, can be used to authorize and initiate MBMS bearer services within a Public Land Mobile Network (PLMN), and can be used to schedule MBMS transmissions. MBMS gateway 168 can be used to distribute MBMS traffic to base station 102 that belongs to a Multicast Broadcast Single Frequency Network (MBSFN) area and broadcast specific services, and can be responsible for session management (start / stop) and collecting billing information related to enhanced MBMS (eMBMS).

[0039] The core network 190 may include an Access and Mobility Management Function (AMF) 192, other AMFs 193, a location management function (LMF) 198, a Session Management Function (SMF) 194, and a User Plane Function (UPF) 195. The AMF 192 can communicate with the Unified Data Management (UDM) 196. The AMF 192 is the control node that handles signaling between the UE 104 and the core network 190. Typically, the SMF 194 provides QoS streaming and session management. All user Internet Protocol (IP) packets are transmitted through the UPF 195. The UPF 195 provides UE IP address allocation and other functions. The UPF 195 connects to an IP service 197. The IP service 197 may include the Internet, an intranet, an IP Multimedia Subsystem (IMS), a Packet Switched Streaming Service (PSStreaming Service), and / or other IP services.

[0040] A base station may also be referred to as a gNB, Node B, evolved Node B (eNB), access point, base transceiver, wireless base station, wireless transceiver, transceiver function, basic service set (BSS), extended service set (ESS), transmit reception point (TRP), or other suitable terms. Base station 102 provides UE 104 with access to EPC 160 or core network 190. Examples of UE 104 include cellular phones, smartphones, session initiation protocol (SIP) phones, laptops, personal digital assistants (PDAs), satellite broadcasting, global positioning systems, multimedia devices, video devices, digital audio players (e.g., MP3 players), cameras, game consoles, tablets, smart devices, wearable devices, vehicles, electricity meters, gas pumps, large or small kitchen appliances, healthcare devices, implants, sensors / actuators, displays, or any other similarly functional device. Some UE 104 devices may be referred to as Internet of Things (IoT) devices (e.g., parking meters, gas pumps, toasters, vehicles, heart monitors, etc.). UE 104 may also be referred to as site, mobile station, user station, mobile unit, user unit, radio unit, remote unit, mobile device, radio device, wireless communication device, remote device, mobile user station, access terminal, mobile terminal, radio terminal, remote terminal, handheld device, user agent, mobile client, client, or other suitable terms.

[0041] Although this disclosure may relate to 5G New Radio (NR), it may also be applied to other similar fields, such as LTE, LTE-Advanced (LTE-A), Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM) or other radio / RF access technologies.

[0042] Figure 2This is a block diagram illustrating communication between a base station 210 and a user equipment (UE) 250 in an access network. In the downlink, IP packets from the evolved packet core (EPC) 160 can be provided to the controller / processor 275. The controller / processor 275 implements Layer 3 and Layer 2 functions. Layer 3 includes the Radio Resource Control (RRC) layer, and Layer 2 includes the Packet Data Convergence Protocol (PDCP) layer, the Radio Link Control (RLC) layer, and the Media Access Control (MAC) layer. The controller / processor 275 provides RRC layer functions related to system information (e.g., Master Information Block (MIB), System Information Blocks (SIBs)) broadcasting, RRC connection control (e.g., RRC connection paging, RRC connection establishment, RRC connection modification, and RRC connection release), inter-Radio Access Technology (RAT) mobility, and measurement configuration for UE measurement reporting; PDCP layer functions related to header compression / decompression, security (encryption, decryption, integrity protection, integrity verification), and handover support functions; RLC layer functions related to uplink layer packet data unit (PDU) transmission, error correction via Automatic Repeat Request (ARQ), concatenation, segmentation, and reassembly of RLC service data units (SDUs), resegmentation of RLC data PDUs, and reordering of RLC data PDUs; and mapping between logical channels and transport channels, multiplexing of MAC SDUs on transport blocks (TBs), and MAC... SDU performs demultiplexing, scheduling information reporting, error correction via Hybrid Automatic Repeat Request (HARQ), priority handling, and MAC layer functions related to logical channel priority from the TB.

[0043] Transmit (TX) processor 216 and receive (RX) processor 270 implement Layer 1 functions related to various signal processing functions. Layer 1 includes a physical (PHY) layer, which may include error detection of the transport channel, forward error correction (FEC) encoding / decoding of the transport channel, interleaving, rate matching, mapping to the physical channel, modulation / demodulation of the physical channel, and multiple-input multiple-output (MIMO) antenna processing. TX processor 216 processes signal constellation mapping according to various modulation schemes (e.g., binary phase shift keying (BPSK), quadrature phase shift keying (QPSK), M-order phase shift keying (M-PSK), M-order quadrature amplitude modulation (M-QAM)). The encoded and modulated symbols can then be divided into parallel streams. Each stream can then be mapped to an orthogonal frequency division multiplexing (OFDM) subcarrier, multiplexed with a reference signal (e.g., a pilot) in the time and / or frequency domains, and then combined using an inverse fast Fourier transform (IFFT) to generate a physical channel carrying the time-domain OFDM symbol stream. The OFDM stream is spatially precoded to generate multiple spatial streams. Channel estimates from channel estimator 274 are used to determine the coding and modulation schemes, as well as spatial processing. The channel estimates can be derived from a reference signal and / or channel condition feedback transmitted by UE 250. Each spatial stream can then be provided to a different antenna 220 via a separate transmitter 218TX. Each transmitter 218TX can be transmitted using a corresponding spatial stream modulated radio frequency (RF) carrier.

[0044] At UE 250, each receiver 254RX receives signals through its corresponding antenna 252. Each receiver 254RX recovers the information modulated onto the RF carrier and provides the information to the receive (RX) processor 256. The TX processor 268 and RX processor 256 implement Layer 1 functions related to various signal processing functions. The RX processor 256 can perform spatial processing on the information to recover any spatial stream directed to UE 250. If there are multiple spatial streams directed to UE 250, they can be merged into a single OFDM symbol stream by the RX processor 256. The RX processor 256 then uses a Fast Fourier Transform (FFT) to transform the OFDM symbol stream from the time domain to the frequency domain. The frequency domain signal includes a separate OFDM symbol stream for each OFDM signal subcarrier. The symbols on each subcarrier, along with the reference signal, are recovered and demodulated by determining the most probable signal constellation point transmitted by base station 210. These soft decisions can be based on a channel estimate calculated by channel estimator 258. The soft decision is then decoded and deinterleaved to recover the data and control signals initially transmitted by base station 210 on the physical channel. The data and control signals are then provided to controller / processor 259, which implements Layer 3 and Layer 2 functions.

[0045] Controller / processor 259 may be associated with memory 260, which stores program code and data. Memory 260 may be referred to as computer-readable medium. In the uplink, controller / processor 259 provides demultiplexing, packet reassembly, decryption, header decompression, and control signal processing between transport and logical channels to recover IP packets from EPC 160. Controller / processor 259 is also responsible for error detection using ACK and / or NACK protocols to support HARQ operation.

[0046] Similar to the downlink transmission functions described in base station 210, controller / processor 259 provides RRC layer functions related to system information (e.g., MIB, SIBs) acquisition, RRC connection, and measurement reporting; PDCP layer functions related to header compression / decompression and security (encryption, decryption, integrity protection, integrity verification); RLC layer functions related to uplink PDU transmission, error correction via ARQ, concatenation, segmentation, and reassembly of RLC SDUs, resegmentation of RLC data PDUs, and reordering of RLC data PDUs; and MAC layer functions related to mapping between logical channels and transport channels, multiplexing of MAC SDUs on TB, demultiplexing of MAC SDUs from TB, scheduling information reporting, error correction via HARQ, priority processing, and logical channel priority.

[0047] The channel estimate derived by the channel estimator 258 from the reference signal or feedback sent by the base station 210 can be used by the TX processor 268 to select an appropriate coding and modulation scheme and facilitate spatial processing. The spatial stream generated by the TX processor 268 can be provided to different antennas 252 via individual transmitters 254TX. Each transmitter 254TX can be transmitted using the corresponding spatial stream modulated on an RF carrier. The uplink transmission is processed at the base station 210 in a manner similar to that described in relation to the receiver function of the UE 250. Each receiver 218RX receives the signal through its corresponding antenna 220. Each receiver 218RX recovers the information modulated onto the RF carrier and provides the information to the RX processor 270.

[0048] Controller / processor 275 may be associated with memory 276 storing program code and data. This memory 276 may be referred to as a computer-readable medium. In the uplink, controller / processor 275 provides demultiplexing, packet reassembly, decryption, header decompression, and control signal processing between transport and logical channels to recover IP packets from user equipment (UE) 250. IP packets from controller / processor 275 may be provided to evolved packet core (EPC) 160. Controller / processor 275 is also responsible for error detection using acknowledgment (ACK) and / or denial (NACK) protocols to support Hybrid Automatic Repeat Request (HARQ) operation.

[0049] New Radio (NR) can refer to wireless devices configured to operate under a new air interface (e.g., a non-Orthogonal Frequency Division Multiple Access (OFDMA) air interface) or a fixed transport layer (e.g., a non-Internet Protocol (IP) layer). NR can employ Orthogonal Frequency Division Multiplexing (OFDM) with a cyclic prefix (CP) on both the uplink and downlink, and can include support for half-duplex operation using Time Division Duplex (TDD). NR can include Enhanced Mobile Broadband (eMBB) services targeting wide bandwidth (e.g., above 80 MHz), millimeter wave (mmW) services targeting high carrier frequencies (e.g., 60 GHz), massive machine-type communication (mMTC) services targeting non-backward-compatible MTC technologies, and / or mission-critical services targeting ultra-reliable low-latency communication (URLLC) services.

[0050] It can support a single component carrier bandwidth of 100 MHz. In one example, an NR resource block (RB) can span 12 subcarriers with a subcarrier bandwidth of 60 kHz and a duration of 0.25 ms, or a bandwidth of 30 kHz and a duration of 0.5 ms (similarly, a 15 kHz subcarrier spacing with a duration of 1 ms represents a 50 MHz bandwidth). Each radio frame can consist of 10 subframes (10, 20, 40, or 80 NR slots) with a length of 10 ms. Each slot can indicate the link direction (i.e., downlink or uplink) for data transmission, and the link direction of each slot can be dynamically switched. Each slot can include downlink / uplink data and downlink / uplink control data. NR uplink and downlink slots can be combined as follows. Figure 5 and Figure 6 More detailed description.

[0051] The NR Radio Access Network (RAN) may include Central Units (CUs) and Distributed Units (DUs). NR Base Stations (BSs) (e.g., gNBs, 5G Node Bs, Node Bs, Transceiver Points (TRPs), Access Points (APs)) may correspond to one or more base stations. NR cells may be configured as Access Cells (ACells) or Data-Only Cells (DCells). For example, the RAN (e.g., Central Units or Distributed Units) may configure these cells. DCells may be used for carrier aggregation or dual connectivity and may not be used for initial access, cell selection / reselection, or handover. In some cases, DCells may not transmit synchronization signals (SSs), and in others they may transmit SSs. NR base stations may transmit downlink signals to User Equipment (UEs) indicating the cell type. Based on the cell type indication, UEs can communicate with NR base stations. For example, UEs can determine the NR base station for cell selection, access, handover, and / or measurement based on the indicated cell type.

[0052] Figure 3 An example logical architecture of a distributed radio access network (RAN) 300 is shown according to certain aspects of this disclosure. A 5G access node 306 may include an access node controller (ANC) 302. The ANC may be the central unit (CU) of the distributed RAN. The backhaul interface to the next-generation core network (NG-CN) 304 may terminate at the ANC. The backhaul interface to the neighboring next-generation access node (NG-AN) 310 may terminate at the ANC. The ANC may include one or more transceiver points (TRPs) 308 (also referred to as base stations, NR base stations, Node B, 5G NB, AP, or other terms). As mentioned above, TRP can be used interchangeably with "cell".

[0053] TRP 308 can be a Distributed Unit (DU). A TRP can connect to one ANC (ANC 302) or multiple ANCs (not shown). For example, for RAN sharing, Radio-as-a-Service (RaaS), and service-specific ANC deployments, a TRP can connect to multiple ANCs. A TRP can include one or more antenna ports. A TRP can be configured to provide services to user equipment individually (e.g., dynamically selected) or jointly (e.g., jointly transmitted).

[0054] The local architecture of the distributed RAN 300 can be used to illustrate the fronthaul definition. This architecture can define fronthaul solutions that support different deployment types. For example, the architecture can be based on transport network capabilities (e.g., bandwidth, latency, and / or jitter). The architecture can share features and / or components with LTE. Depending on certain aspects, the Next Generation Access Node (NG-AN) 310 can support dual connectivity with NR. The NG-AN can share a common fronthaul for both LTE and NR.

[0055] This architecture enables collaboration between and within TRPs 308. For example, collaboration can be pre-defined within a TRP and / or implemented across TRPs via ANC 302. Depending on certain aspects, inter-TRP interfaces may not be required or may not exist.

[0056] In some respects, the architecture of the distributed RAN 300 can have a dynamic configuration with logical function splitting. Packet Data Convergence Protocol (PDCP), Radio Link Control (RLC), and Media Access Control (MAC) protocols can be adaptively placed in the ANC or TRP.

[0057] Figure 4 An example physical architecture of the distributed RAN 400 is illustrated according to aspects of this disclosure. A centralized core network unit (C-CU) 402 can host core network functions. The C-CU can be centrally deployed. C-CU functions can be offloaded (e.g., to advanced wireless service (AWS)) to handle peak capacity. A centralized RAN unit (C-RU) 404 can host one or more ANC functions. Optionally, the C-RU can host core network functions locally. The C-RU can be distributed. The C-RU can be located closer to the network edge. A distributed unit (DU) 406 can host one or more TRPs. The DU can be located at the network edge and have radio frequency (RF) functions.

[0058] The traditional and common role of user devices is to provide connectivity to dedicated servers within the network, as seen in server-client services, or to facilitate peer-to-peer communication, such as telephone and video calls, through such servers. However, next-generation networks, including mobile networks, are expected to support a variety of user devices, artificial intelligence (AI) applications, and distributed computing and communication sharing.

[0059] The distributed artificial intelligence (AI) service platform described in this disclosure represents a strategic advancement in leveraging distributed computing principles within an AI and machine learning (ML) framework to realize mixed reality services using virtual personal assistants. The platform aims to optimize the distribution of computational workloads between user devices and network nodes by splitting AI / ML tasks based on real-time conditions. When a customer visits a shopping mall or store, a virtual personal assistant is assigned to assist them throughout their visit until they leave. This is achieved by intelligently distributing computational tasks between smartphones (responsible for initial and local AI / ML computations) and edge servers (performing significant computations offloaded from user devices).

[0060] This platform addresses several key business elements, differentiating it from traditional approaches. First, it positions AI as an omnichannel assistant capable of handling diverse service challenges. Second, it provides a virtual customer service assistant that accompanies customers throughout their store visit, offering ongoing support. Third, the architecture splits the role of AI / ML processing between smartphones and edge servers; smartphones handle the initial phase of AI / ML computation, subsequently offloading more intensive tasks to network devices. Fourth, edge servers handle the significant AI / ML computations offloaded from user or terminal devices. Fifth, the system reduces network traffic by minimizing uplink traffic by avoiding unnecessary information transmission from user devices to network devices.

[0061] This distributed approach offers multiple advantages for system performance and user experience. By distributing computing tasks among available resources on the network and local devices, the system achieves efficient resource sharing, improving overall system performance and scalability. The service significantly reduces uplink traffic, lowering bandwidth usage and related costs by transmitting only necessary data to network devices. Users benefit from faster processing speeds and improved device battery life, as heavy computing tasks are offloaded to edge servers when appropriate. Furthermore, this disclosure lays the foundation for future value creation opportunities for equipment manufacturers, service providers, and network providers.

[0062] The distributed AI service platform provides an integrated communication and computing (ICC) service architecture and mechanism specifically designed to deliver mixed reality services through virtual personal assistants. The platform supports domain-specific virtual personal assistant services that can be integrated with mixed reality capabilities. When the service is activated, a dedicated virtual assistant is assigned to assist the user until the task is completed, and can continuously provide personalized services based on user preferences and past interactions. The distributed AI service offers flexible deployment options, allowing for the installation of dedicated servers or computing resources within a corporate customer premises network (CPN), or leasing from network operators such as mobile network operators (MNOs) or cloud operators. This flexibility enables service providers to choose the most suitable deployment model based on their specific needs and resources.

[0063] Figure 5Illustration 500 outlines the design of a distributed AI service platform. This platform demonstrates how a single distributed AI service infrastructure can be applied across multiple sectors, including healthcare, retail, manufacturing, agriculture, food service, libraries, museums, transportation, tourism, real-time language services, and services for people with disabilities. This diversity allows for flexible and personalized assistance while keeping personal information on the user's device and service provider information on the facilities they manage, whether owned or leased. Distributed AI services can be deployed by installing dedicated servers or computing resources in a customer premises network (CPN) within an enterprise network, or leased from network operators such as mobile network operators (MNOs) or cloud operators. This distributed AI service platform creates new value creation opportunities by enabling equipment manufacturers, service providers, and network providers to participate in different stages of the distributed service delivery ecosystem.

[0064] The distributed AI service platform comprises multiple functional modules that can be flexibly deployed on user devices and the service engine. These modules are selectively activated based on real-time conditions, including device capabilities and status (e.g., low battery or low resource availability) and network conditions (e.g., low link bandwidth availability). Some modules may reside on user devices (e.g., smartphones), while others may be deployed on nearby systems (e.g., Customer Premises Equipment (CPE)) or mobile operator networks. Mobile operators (e.g., AT&T or Verizon) may host some of the computing devices involved in service delivery. By combining these distributed user devices and network resources, they work together to provide services.

[0065] The user device integrates several key functional modules to support distributed services. A service model agent, a simplified service model residing on the device, handles initial processing and contains only necessary components, while critical service resources remain in the service engine managed by the service provider. A personal assistant module provides a virtual assistant that interacts with the user, the local AI / ML module, and the service models in the service engine. The device also includes a multilingual service module containing various small ML models (such as LLM and Llava) to provide different language services based on user preferences when delivering services locally. A device analytics module continuously monitors or predicts the user device's operational status, including CPU and memory utilization, battery level, and battery charging status. A network analytics module monitors or predicts wireless channel conditions, including link bandwidth availability and achievable end-to-end throughput between the user device and the service engine. Furthermore, the device includes a speech-to-text (STT) module that converts voice audio prompts or commands into text, and a text-to-speech (TTS) module that converts text responses into audio streams.

[0066] The service engine comprises complementary modules that collaborate with user devices to deliver complete services. The service model, a comprehensive knowledge base managed by the service provider, contains all the domain-specific information needed to support the customer. The service engine's multilingual service module includes various large ML models (such as LLM and Llava) that deliver different language services based on user preferences when provided via a high-performance server. The network analytics module in the service engine interacts with corresponding modules in the user device, enabling the device to accurately monitor network conditions. The service engine also includes a TTS module that converts text responses into audio streams when processed on the server side rather than the device side.

[0067] Virtual personal assistant mixed reality services have particular application value in the retail industry, addressing challenges faced by both traditional stores and emerging unmanned stores. Unmanned stores (also known as autonomous stores or cashierless stores) have seen significant growth in the retail sector as part of the trend towards integrating automation and smart technologies. These stores heavily rely on technologies such as RFID, computer vision, the Internet of Things (IoT), and artificial intelligence (AI) to track purchases and manage inventory. Their primary consumer benefit lies in streamlining the shopping experience by eliminating checkout queues, making the shopping process more efficient. These stores collect vast amounts of consumer behavior data, which can be used to optimize inventory levels, store layout, and marketing strategies. While reducing staffing needs promises to lower operating costs, initial technology investments can be substantial.

[0068] Major tech companies are driving the unmanned store trend. Amazon, through its Amazon Go stores, and other tech giants are expanding their presence in the field, demonstrating confidence in the viability of the unmanned store model. Global adoption is occurring at varying paces across countries, with some implementing these technologies more quickly. While retail is the primary application area, the concept has expanded to food service, libraries, and transportation hubs. As technology continues to advance, unmanned systems are expected to see more complex and diverse applications in retail and other sectors. However, this trend faces significant challenges, particularly in addressing privacy concerns related to data collection and maintaining personalized, interactive customer service beyond basic purchase tracking and inventory management.

[0069] Virtual personal assistant services directly address these challenges by providing personalized interactions that are often lacking in unmanned stores. Take retail stores like supermarkets as an example: customers often struggle to find items. In stores like Costco, product locations are frequently reorganized, leading to customer frustration when searching for what they need. Customers entering the store typically require assistance from customer service representatives, but human employees are often unavailable or busy in back offices or other areas of the store.

[0070] When stores implement distributed artificial intelligence (AI) customer service systems, virtual assistants replace the need for physical customer service personnel. When a customer enters a shopping mall or store, the system can detect their presence and activate a virtual personal assistant on their device to assist throughout their visit. Alternatively, customers who wish to use a virtual assistant can proactively activate it by scanning a Near Field Communication (NFC) tag upon entering the store or shopping mall. Scanning the NFC tag will prompt the user to install an application (if not already installed), or activate the virtual assistant if the application is already installed.

[0071] For first-time visitors, a default 3D avatar appears on the user's device—whether it's smart glasses, a smartphone, or another device with a display—and greets them. This avatar can be rendered on top of the real-world view in mixed reality (MR) or displayed separately from the real-world view in augmented reality (AR). Interactions between the user and the virtual assistant can be conducted via audio, text, or both, depending on device capabilities and user preferences.

[0072] After an initial greeting such as "I am your assistant" or "You can choose a virtual avatar or create one for yourself," users can select different virtual avatars, including male, female, child, or pet figures, or customize the avatar's appearance. This customization option is particularly relevant for first-time visitors or users whose existing customer information is not stored in the system.

[0073] Customer follow-up benefits from continuity, as the same avatar can continue to serve them across multiple visits. This personalization feature enhances customer engagement and may increase the likelihood of repeat visits.

[0074] The system requests informed consent to take the customer's photo for registration and future identification. Alternatively, customers can choose other identification methods, such as phone numbers. Customers retain the option to decline and skip this registration step.

[0075] The virtual assistant can provide additional privacy options through voice prompts, such as, "Would you like me to remember you and your visit so I can better serve you next time you come?" Communications during the visit can be stored locally on the user's device for use on future visits.

[0076] When customers have questions while shopping in-store, the virtual store assistant provides comprehensive support. The assistant offers detailed product information, including manufacturer details, expiration date, price, discounts, warranty, ingredients, nutritional content, and calories per serving. It also provides operational guidance, such as assembly instructions, installation procedures, and usage methods. The system facilitates purchase-related services, including order placement and order status tracking. Furthermore, it provides product availability information, identifies the current quantity of items in the store, locates nearby stores with stock, and provides details on out-of-stock items. When authorized by the user, the virtual store assistant can provide personalized services based on individual preferences, shopping history, and other relevant factors.

[0077] The system allows customers to ask natural language questions, such as "Where can I find this product?" and receive immediate directions to the appropriate shelf. When a customer scans a product, the virtual assistant provides comprehensive information about the price, applicable discounts, sales status, out-of-stock availability, and other product details the customer may request. This service cannot operate solely on customer devices such as phones or smart glasses because the requested information is specific to each store's operations and inventory.

[0078] The system infrastructure is deployed on store servers, implemented through agents or store-owned Customer Premises Equipment (CPE). Alternatively, stores like Costco may lease server capacity from network operators, with services provided by external infrastructure. Even when customers interact with the virtual assistant on their personal devices, the underlying services may originate from nearby in-store servers, more distant mobile operator servers, or run directly on the customer's device when local resources and cached information are sufficient.

[0079] Distributed systems dynamically allocate functions based on various real-time factors. Network conditions and device capabilities determine where each function is executed. High-end smartphones, with their powerful processing capabilities, can handle more functions locally, while smart glasses with limited battery life or hardware capabilities rely more on external servers, whether local agents or mobile network infrastructure. This dynamic allocation optimizes performance while efficiently managing device resources and network bandwidth.

[0080] Figure 6 Figure 600 illustrates the deployment architecture of a virtual personal assistant mixed reality service in the retail industry. The virtual assistant service network includes user devices, product registration servers, and network services connecting user devices, service agents, and servers containing service models and service engines. User devices include smartphones, smart glasses, and smartwatches, which interact with the distributed service infrastructure. While this disclosure employs the Retrieval Augmented Generation (RAG) framework to implement the service model, its application is not limited to RAG and includes other applicable frameworks.

[0081] The service engine comprises multiple functional modules, including Speech-to-Text (STT), Text-to-Speech (TTS), a large collection of Language Models (LLMs), and sub-modules for generating and rendering 3D virtual assistant avatars. These sub-modules can be distributed across the service network, rather than concentrated on a single machine, allowing for flexible deployment based on resource availability and performance requirements. Figure 6 The server infrastructure shown can be deployed in a variety of configurations: as shown in index "1", it can be deployed within a dedicated server in a customer premises network (CPN); as shown in index "2", it can be hosted on shared computing and storage resources managed by mobile network operators (MNOs); as shown in index "3", it can be operated by fixed network operators (including DSL, fiber, cable, Wi-Fi, or satellite network operators); and as shown in index "4", it can be deployed on cloud network operator infrastructure.

[0082] To optimize performance and reliability, servers can be deployed across multiple sites or networks, achieving load balancing, improved fault tolerance, and enhanced end-to-end performance. The service model and service engine modules within the server can be deployed in different physical locations, supporting the maintenance of a single service model and the running of multiple service engine instances to support virtual assistant services. The service broker serves as the initial communication endpoint for user devices to establish connections; user devices establish contact with the service broker before connecting to the server.

[0083] User devices, whether smartphones, smart glasses, or smartwatches, serve as the initial processing center for AI / ML tasks. The device handles the initial computational phase before offloading resource-intensive operations to server infrastructure. This approach accelerates initial response time while reducing network data transmission. The device then handles lightweight AI / ML computations, including initial data collection, preprocessing steps, and basic analytical operations that do not require significant computational resources.

[0084] The service proxy serves as the primary service landing point for user equipment communications. When a virtual assistant application launches on a user device, it establishes a connection with a known proxy site. This proxy can reside within the Customer Premises Network (CPN) of the Customer Premises Equipment (CPE) or be accessed through any network connection available to the client application. Depending on the configured service operation policy, the proxy performs various functions, including redirecting the virtual assistant application to a specific service server, providing message forwarding capabilities, or implementing load balancing across multiple server instances.

[0085] Evolved Residential Gateways (eRGs) provide gateway functionality between public 5G network operators and Customer Premises Networks (CPNs) within residences, offices, or retail locations. The 5G system architecture supports network access for authorized User Equipment (UEs) and authorized non-3GPP devices within a PIN or CPN via gateway components, such as gateway-capable Personal IoT Network (PIN) elements or eRG devices. Premises Radio Access Stations (PRASs) are base stations deployed within a CPN that function similarly to standard base stations and can operate using licensed, unlicensed frequency bands, or a combination of both. The connection between the eRG and connected devices (including UEs, non-3GPP devices, or PRAS units) can utilize any applicable non-3GPP technology, such as Ethernet, fiber optic connections, or wireless LANs (WLANs).

[0086] As a network device, the server comprises service model and service engine components, performing a significant amount of artificial intelligence / machine learning (AI / ML) computation offloaded from user devices. The server infrastructure boasts powerful computing capabilities and storage capacity to handle complex processing tasks. The service model component builds and maintains an embedded database of products during the registration process, while the service engine uses information retrieved from the service model to process user queries. The server's computing power includes executing deep learning processes, performing complex analyses, and managing high storage demands that exceed the capacity of user devices.

[0087] A variety of technologies can be used to detect user devices and enable interaction within the service area; the specific choice depends on the required accuracy and user convenience. Table 1 details the specifications of these detection technologies. Bluetooth and Wi-Fi Direct technologies are widely available in consumer devices, and their effective range is determined by the radio propagation characteristics of the deployment environment.

[0088] Table 1. Technologies used for short-distance data exchange

[0089] For scenarios where the application is only activated when a customer actually enters the store, and accidental activation by passersby is avoided, accurate location detection is required. Ultra-wideband (UWB) technology provides high-precision positioning capabilities, but it requires the deployment of UWB anchor devices within the service area, significantly increasing implementation costs. Near Field Communication (NFC) and Quick Response (QR) code technologies are particularly suitable for virtual assistant activation because they require users to actively scan NFC tags or QR codes upon entering the store, clearly expressing user intent and offering ease of operation.

[0090] When a customer enters a retail location, the system initiates a series of processes to activate the virtual assistant service. Customers wishing to interact with the virtual assistant scan an NFC tag or QR code located at the store entrance. This scan triggers the system; if the app is not installed on the device, it prompts the user to install it; if it is already installed, the virtual assistant is activated directly. The system then requests the user's informed consent to take their photo for registration and future in-store identification. Alternative identification methods, such as phone numbers, are also supported, and customers can choose to decline this registration step. The virtual assistant can make requests via natural voice, such as, "Would you like me to remember you and your visit? I can better serve you next time you come."

[0091] The virtual assistant will also request user authorization to record interactions during their in-store visit. Customers can choose to store interactions in the in-store system to improve future services, save them only on their local device for personal reference and training, or not record them at all to meet their privacy needs. This meticulous control over data processing allows customers to strike a balance between personalized service and their privacy preferences.

[0092] The system supports various network connection scenarios that may occur during service operation. Similar to using public Wi-Fi in places like Starbucks, customers may or may not directly connect to the store's local network infrastructure. Message flow will dynamically adjust based on whether the user's device is connected to the store's CPN. When no connection is established, the service is provided through the user's subscribed mobile network operator. Some retail locations may not have any on-site service infrastructure and will rely entirely on services hosted by mobile network operators or cloud service providers.

[0093] This architecture supports end-to-end communication through various message flow patterns, adapting to different deployment and operational configurations. When a user launches the virtual assistant client application, their device is typically not initially connected to a CPN access point (such as in-store Wi-Fi). The client application contacts the service agent via a CPN or internet connection. The user's initial message reaches the service agent through the network operator and internet infrastructure subscribed to by the user. The agent then coordinates subsequent communication steps according to its operational configuration. When the agent functionality is directly integrated into the client application, updates are required whenever the service network configuration changes (such as adding, migrating, or removing service servers).

[0094] The network a user subscribes to refers to the mobile network operator the customer uses, such as AT&T or Verizon. When a customer enters the store, they typically do not immediately obtain in-store network credentials. If the application knows the in-store web service URL, the initial service request is routed through the customer's subscribed operator's network to a proxy server within the store. The proxy then provides network access credentials, including an authentication ID and password, to establish a local network connection. Once the user's device accepts these credentials and establishes a local connection, subsequent communication bypasses the mobile operator's network and interacts directly with the in-store service infrastructure, thereby reducing latency. This mechanism optimizes data exchange efficiency once a local area network connection is established.

[0095] Figure 7 This diagram illustrates the message flow between a centralized service broker and a customer premises network (CPN). This message flow represents one of the three main deployment scenarios for virtual personal assistant services, where the service broker acts as an intermediary between user devices and distributed service servers. The CPN represents the local network infrastructure within a retail location or enterprise facility, which can directly host service servers or provide optimized connectivity for remote service resources.

[0096] The messaging flow begins with the user device launching the virtual assistant client application. In this initial phase, the user device typically connects via its subscribed network operator (such as AT&T or Verizon), rather than directly to the store's local network infrastructure. The client application is configured with the address of a service proxy, which serves as the primary landing point for all initial service connections.

[0097] Step 1 involves the user device sending a service discovery request message to the service agent via its subscribed network operator and the internet. This message contains comprehensive device information, enabling the system to make optimal service delivery decisions. Device capability information includes hardware specifications, such as the number of CPU cores, indicating whether the device is a high-end smartphone capable of local artificial intelligence (AI) processing or a resource-constrained device such as smart glasses. The message also includes the device's current location within the store, helping to determine proximity to local service resources. Battery status information, including whether the device is charging and its current battery level (e.g., 20% or 80%), provides crucial input for workload allocation decisions. When the battery is low, the system minimizes local processing demands and relies more on network-based resources to keep the device running.

[0098] Service discovery requests also include Radio Access Technology (RAT) availability, specifying which wireless technologies the device supports, such as Wi-Fi, 4G, 5G, or Bluetooth. Each RAT option offers different bandwidth capabilities and power consumption characteristics, which influence service delivery decisions. Radio quality metrics, including signal strength and connection stability, determine the maximum data throughput achievable between the user device and the network. This information directly impacts decisions regarding data-intensive operations, such as whether to stream high-quality video avatars or use low-bandwidth text-based interactions.

[0099] Step 2 occurs when the service agent responds and provides Customer Premises Network (CPN) access credentials and connection information. The agent provides the network name, authentication credentials, and connection parameters, enabling the user device to establish a direct connection to the store's local Wi-Fi access point or other CPN infrastructure. This local connection offers lower latency and higher bandwidth compared to routing over the public internet. However, the user device retains autonomy over the connection decision and can choose to join or leave based on user preferences, privacy settings, or security considerations.

[0100] Step 3 represents the scenario where the user equipment accepts a local connection opportunity and establishes a connection with the CPN access point. This connection provides the most efficient communication path for subsequent service interactions because data is transmitted directly between the user equipment and the local service infrastructure, without needing to traverse the public internet.

[0101] Step 4 confirms the successful CPN connection when the user equipment sends an acknowledgment message to the service broker via the newly established CPN connection. This acknowledgment indicates that subsequent communication can utilize the optimized local network path instead of the longer route taken by the subscribed network operator.

[0102] Steps 5 and 6 demonstrate the standard service interaction pattern when the service server is located within the CPN. The user sends a service request through the service broker, which may contain natural language questions about the product, requests for assistance, or instructions to the virtual assistant. The service broker forwards the request to the appropriate server within the CPN network, which processes the request using its AI model and local product information database. The server generates a response and sends it back to the user's device through the service broker. Because all communication occurs within the store's network infrastructure, this local processing path provides the lowest possible latency.

[0103] Steps 7 and 8 illustrate an alternative service path when the required service server is deployed in the mobile network edge cloud rather than the local CPN. This configuration may be used for compute-intensive operations requiring more powerful servers than on-premises servers, or for accessing centralized services shared across multiple store locations. The user sends a service request to the service broker, which determines that the request should be handled by the mobile network edge server. The broker forwards the message to the server in the mobile network. When the mobile network edge server is not directly accessible from the public internet, the broker's traffic passes through the evolved residential gateway (eRG). The eRG acts as a gateway between the CPN and the 5G network infrastructure, providing authorized access to 5G network services for both 3GPP-compliant and non-3GPP devices within the CPN. Responses from the mobile network edge server are returned to the user equipment via the eRG and the service broker along the reverse path.

[0104] Figure 7 The message flow in the diagram represents the optimal latency scenario when a user device (User Device) communicates with the service infrastructure using a local CPN connection. This configuration minimizes network latency by eliminating unnecessary public network routing. However, the system design also accommodates situations where the device may choose not to utilize the available local connection. After receiving the CPN access credentials in step 2, the User Device may refuse the local connection opportunity due to privacy preferences, security policies, or user configuration settings. In this case, the device continues to route all traffic through its subscribed mobile network operator, resulting in longer network paths and higher latency. While this represents a suboptimal performance configuration, it is still an effective operating mode fully supported by the system. This flexibility reflects the platform's commitment to user autonomy and privacy preferences, allowing customers to balance performance optimization with their personal data processing preferences.

[0105] Figure 8 Figure 800 illustrates the message flow of a centralized service broker without a CPN connection. This configuration represents an alternative operating mode where user devices maintain connectivity through their subscribed network operator, rather than establishing a direct connection to the store's local network infrastructure. While this approach results in higher latency compared to a direct CPN connection, it is a fully supported configuration that accommodates user privacy preferences, security policies, or situations where automatic connection to the local network is disabled on the device.

[0106] The message flow begins in step 1, where the user device sends a service discovery request message to the service broker via its subscribed network operator and the internet. This initial communication follows the pattern in the CPN connection scenario, and the message contains comprehensive device information, including device capabilities, current location within the store, battery status, device type, Radio Access Technology (RAT) availability, radio quality metrics, and user preference configuration. These parameters enable the service broker to make informed decisions regarding service redirection and resource allocation, taking into account that all subsequent communication will traverse the public internet rather than the local network path.

[0107] In step 2, the service broker responds and provides CPN access information and credentials, offering the user equipment the opportunity to connect to the CPN access point. This achieves lower network service latency and higher throughput connections by establishing a direct path to local service resources. The system provides this option to all devices, maintaining consistency in the service discovery protocol regardless of their final connection decision.

[0108] Step 3 is a key feature of this scenario: after receiving valid credentials, the user device actively chooses not to connect to the CPN access point. The device sends a negative acknowledgment (NACK) message to the agent via its subscribed network and the internet, confirming its decision to continue routing through the public network. This choice may reflect a user's privacy preferences, a corporate security policy that prohibits automatic connections to third-party networks, or a device configuration that prioritizes cellular connections. Although this decision increases network latency due to the longer routing path, the system fully supports this choice as an effective operating mode.

[0109] Steps 4 and 5 demonstrate the service interaction when the target server is located within the CPN network. The user sends a service request, such as a natural language question about a product, to the service broker via the internet. The service broker forwards the request to the appropriate server within the CPN network, which processes the query and generates a response. The server's response is returned to the user's device via the service broker, traveling along the internet and the reverse path of the network the user has subscribed to. Although the server is located within the CPN, the communication path still traverses the public internet because the user's device chooses not to establish a local connection.

[0110] Steps 6 and 7 illustrate an alternative service fulfillment path when the required processing occurs on a server deployed in the mobile network edge cloud, rather than within the customer premises network (CPN). The user sends a service request to the service broker via the internet, and the broker determines that the specific request should be handled by the mobile network edge server based on factors such as computing needs, data locality, or load balancing. The broker forwards the message to the server in the mobile network. When the mobile network edge server is not directly accessible from the public internet (common in operator-managed infrastructure), the broker's traffic passes through an evolved residential gateway (eRG). The eRG provides the necessary gateway functionality, enabling authorized user equipment (UE) and authorized non-3GPP devices to access the 5G network and its services. The response from the mobile network server follows the reverse path, returning to the user equipment via the eRG and the service broker.

[0111] Figure 8 The message flow shown demonstrates the platform's flexibility in adapting to different user preferences and network configurations. While the lack of a direct CPN connection leads to... Figure 7 While the local connection scenario shown is less effective in terms of latency, this configuration offers significant benefits to users who prioritize network isolation or have specific security needs. The system maintains full functionality across the selected network path, demonstrating the robustness of the distributed architecture. Service delivery adapts to available connection options while maintaining a consistent user experience.

[0112] Figure 9 Figure 900 illustrates the message flow without a centralized service broker and without a CPN connection. This configuration represents the scenario with the highest latency among the three deployment options because it lacks both broker-mediated communication optimization and a local CPN connection. The message flow demonstrates how the system operates when user devices are both independent of the store's local infrastructure and not continuously mediated by a centralized broker.

[0113] Step 1 is initiated when the user equipment sends a service discovery request message to the service agent via its subscribed network operator and the Internet. This message contains comprehensive device information, including device capabilities, location, battery life, device type, radio access technology (RAT) availability, radio quality metrics, and user preference configurations. These parameters inform the agent's decision-making process regarding providing the user equipment with appropriate server information.

[0114] In step 2, the service broker responds with CPN access information and credentials, offering the user equipment the opportunity to establish a connection with the CPN access point. This connection will provide lower network service latency and higher throughput. The broker provides this option regardless of the user equipment's final connection decision.

[0115] Step 3 reflects the user equipment's decision not to connect to the CPN access point. The device sends a negative acknowledgment (NACK) message to the agent via its subscribed networks and the Internet, confirming its preference to maintain routing via public networks. This decision may stem from privacy preferences, security policies, or device configuration settings that prioritize isolation from the local infrastructure network.

[0116] Step 4 is the key difference in this scenario. The agent provides appropriate server information to the user device based on various operational factors such as load distribution, geographic location, and server availability. When multiple servers exist in a distributed infrastructure, the agent selects the one suitable for direct communication. These servers can be deployed in various locations, such as... Figure 6 As shown, this includes customer premises equipment near the store, mobile carrier networks, fixed network carrier infrastructure, or cloud platforms (such as ChatGPT, Amazon Services, or other cloud service providers). Each server has the same functional capabilities, and the system selects based on current operating conditions and server availability, rather than on functional differences.

[0117] In step 5, the user equipment sends an acknowledgment (ACK) message to confirm receipt of the server information. If the previous message was transmitted via TCP, this acknowledgment can be omitted, as TCP itself provides an acknowledgment mechanism.

[0118] Steps 6 and 7 demonstrate direct communication between the user equipment and the selected server. The user equipment sends service requests (such as natural language questions about the product) directly to the designated server via the Internet, without routing through a service proxy. This direct communication path differs from a proxy-mediated configuration. The server processes the request and returns the response directly to the user equipment, traveling along the reverse path of the Internet and the user's subscribed network operator.

[0119] Steps 8 and 9 illustrate an alternative service fulfillment path when the service server is running within a fixed network infrastructure. Similar to the previous steps, the user equipment communicates directly with the server without the intervention of a proxy, and requests and responses are transmitted via the Internet and the network subscribed to by the user.

[0120] This configuration exhibits the highest latency across the three deployment scenarios due to a combination of factors. First, the lack of proxy mediation after initial setup means each request must establish its own routing path via the public internet, failing to benefit from proxy optimization. Second, the lack of CPN connectivity forces all traffic to traverse a longer path through the user's subscribed network operator and the public internet, preventing the utilization of potentially faster local network infrastructure. While this approach introduces additional latency, it provides maximum network isolation for users prioritizing privacy and security over performance optimization. Services in this configuration originate from servers deployed on mobile, fixed, or cloud infrastructure but always reach the user's device via a public internet path.

[0121] Figure 10 Figure 1000 illustrates the design of a virtual personal assistant mixed reality service. The figure describes the system architecture between user devices (such as smartphones) and the service server infrastructure, abstracting intermediate network elements including Customer Premises Equipment (CPE), Evolved Residential Gateway (eRG), Customer Premises Network (CPN), and Mobile Network Operator (MNO) components. The design covers the preparation process for configuring the service server (including service engine and service model components), as well as the complete message flow sequence from the user's initial request to the final response delivery.

[0122] For implementation in the retail industry, the service model preparation follows a structured three-step process. In step 1, a basic knowledge base is established through product listings and information registration. The virtual assistant service requires comprehensive access to product information, necessitating the construction of a complete service model encompassing all item details. The system stores all product items and their related metadata in the inventory database, recording unique item names, categories, detailed descriptions, regular prices, discount rates, warranty terms, expiration dates, best-before dates, displayed inventory quantities, and stored inventory quantities. This data can be stored as plain text or converted into context vectors for efficient retrieval. Figure 10 The top vector database (DB) 1002 represents the product list and information registration database, serving as the main repository for text product information.

[0123] Step 2 involves registering product item images via a smartphone application interface. This process involves scanning each product item using the device's camera, simultaneously capturing a unique product item name identifier. The system embeds multiple images of each item taken from different angles and their corresponding item names into a dedicated vector database. This embedding and storage process is repeated for the entire product catalog. Figure 10 The bottom vector database 1004 represents the product project image registration database. This dual-database architecture enables efficient cross-referencing between visual product representations and their textual descriptions.

[0124] These two databases operate as complementary components of the service model. The product listing and information registration database provides detailed textual product data, while the product item image registration database enables visual product recognition through image embeddings. These databases can be referenced using product names or product indexes as keys, creating flexible query paths. While the databases could be merged into a single unified database for simplified management, maintaining them separately optimizes query performance for different types of searches. The system employs a pre-trained deep learning model to construct the embedding database from product item images and their corresponding names. OpenAI's Contrastive Language-Image Pre-Training (CLIP) model is the primary implementation choice, generating embeddings suitable for both image and text processing. CLIP's bimodal capabilities make it particularly effective at matching visual product representations with their textual item names. This model operates under an MIT license, providing flexibility for commercial deployment.

[0125] Step 3 involves fine-tuning the service model through validation and iterative improvement. After initial product item registration is complete, the system validates accuracy by scanning sample items in the registered inventory. The validation process confirms that the system correctly identifies items and retrieves accurate information. When misidentification occurs, the system updates existing embeddings or registers additional image embeddings until reliable target item recognition is achieved. Once the service model is ready and connected to the service engine, the virtual personal assistant service is ready to go.

[0126] The system dynamically distributes computation based on real-time evaluation of multiple operational factors. Network connectivity determines the available bandwidth and latency characteristics for data transmission. User device capabilities encompass processing power, available memory, and current resource utilization. Resource availability includes battery status; the system considers current battery level and whether the device is charging. Information availability on the user device affects whether a query can be resolved locally or requires network access. Based on these factors, user queries can be processed at different locations within the distributed architecture, including the user device itself or various network servers. Response delivery also adapts to available resources, with processing occurring at the optimal location under current conditions.

[0127] The platform implements three main operating scenarios to adapt to different combinations of network and device conditions. Scenario 1 addresses situations where network connectivity is available but user device resources are insufficient or the required information is unavailable locally. In this configuration, the user device minimizes local processing by streaming voice input directly to the network server. The server infrastructure handles computationally intensive tasks, including speech-to-text conversion, query processing using large language models, response generation, and text-to-speech conversion. The server then sends the final audio response back to the device, which simply plays the received audio without further processing. This approach extends device battery life and supports operation on resource-constrained devices such as smart glasses.

[0128] Scenario 2 applies to situations with poor network connectivity but where the user device has sufficient computing resources. Although the device can perform processing locally, it lacks the information needed to answer queries, thus requiring network communication. To reduce data transmission over limited network connections, the device performs speech-to-text conversion locally, transmitting only a compact text representation rather than a bandwidth-intensive audio stream. The web server processes the text query using its AI model and returns a text response, similar to a typical ChatGPT interaction. The device then performs text-to-speech conversion locally to generate audio output for the user. This approach adapts to network limitations while maintaining voice-based interaction through an efficient data transmission strategy.

[0129] Scenario 3 describes a scenario where the user device has both sufficient computing resources and local cache information, resulting in fully local processing. This typically occurs when a customer makes a repeated query that can be satisfied by a previously cached response. Before performing local processing, the device confirms two prerequisites: first, the cached data is sufficient to satisfy the current request; and second, the local computing power has sufficient battery capacity. The processing flow is entirely executed on the device. First, the user's voice query is converted from speech to text. A large local language model uses cached information to generate a text response, which is then converted from text to speech to generate audio output. The system completes the interaction by playing the synthesized audio response. This configuration eliminates network dependencies, provides fast response times, and protects user privacy through fully local processing.

[0130] Figure 11 This is Schematic diagram 1100, illustrating a scenario of a distributed artificial intelligence service platform operation. The network provides available connectivity, but user devices lack sufficient computing resources or the necessary local information. This scenario demonstrates the platform's ability to dynamically offload computing tasks from resource-constrained devices to network servers, optimizing the distribution of AI / machine learning processing within a distributed architecture.

[0131] In this configuration, the distributed AI service platform allocates processing responsibilities based on real-time assessments of network capabilities and device limitations. The network's robust connectivity enables efficient data transmission between user devices and the service infrastructure, while resource constraints or information gaps on devices necessitate server-side processing. This dynamic allocation demonstrates the platform's distributed computing architecture's capability to migrate computing tasks to the most suitable processing location based on current operating conditions.

[0132] Step 1 is initiated when a user needs information about a specific item. The user scans the item with their smartphone camera and generates a query via voice or text input. Figure 11 The illustration shows the target item being captured by the camera on the left side of the smartphone, which can be captured via the smartphone's integrated camera or smart glasses worn by the customer. This scenario assumes bimodal communication between the customer and the virtual assistant, supporting both audio and text interaction to accommodate different user preferences and device capabilities.

[0133] Step 2 involves performing a speech-to-text conversion on the smartphone, transforming the voice query into text format, and then embedding it for processing. The converted text is displayed on the smartphone screen for user confirmation. The system stores the query locally and privately in the device database, creating a history that can be optimized for future service interactions through personalized responses.

[0134] Step 3 involves the smartphone converting both the input prompt text and the captured image into embedded representations. This conversion process utilizes CLIP models or similar embedding generation techniques to create compact vector representations for the multimodal input data.

[0135] Step 4 transmits the embedded image generated by the smartphone to the service engine via the selected network connection. Embedded transmission significantly reduces network traffic compared to sending the original image. An uncompressed 1920x1080 resolution image requires approximately 6 MB of data transmission, even with JPEG compression it requires 300-600 KB, while embedded representation requires only about 2 KB. This represents a reduction of over 99% compared to uncompressed images and 85-93% compared to JPEG images, demonstrating the platform's efficiency in minimizing network bandwidth consumption.

[0136] Step 5: After receiving the embedding, the service engine retrieves the most relevant information to the user's query from the service model. The service model can be co-located with the service engine for optimal performance or deployed remotely according to infrastructure requirements. The retrieval process uses the embedding vector to identify and match products and related information, retrieving them from the vector database maintained by the service model.

[0137] Step 6 involves the service engine combining its Large Language Model (LLM) capabilities with relevant information retrieved from the service model to prepare a comprehensive response. The service engine processes this information, generates a natural language response to the user's query, and then delivers the response to the user's device. This server-side processing leverages the computing resources of the network infrastructure to compensate for the limited processing power of the user's device.

[0138] Step 7 extends response delivery by converting text-based responses to speech format. The service engine utilizes a Text-to-Speech (TTS) application programming interface (API) that supports Speech Synthesis Markup Language (SSML) to achieve precise control over speech synthesis parameters. The TTS system generates phoneme or lip-sync data simultaneously with audio output, creating synchronization information for lip-sync animation. This server-side audio generation reduces the computational burden on user devices while maintaining high-quality speech output.

[0139] Step 8 involves rendering the virtual assistant's 3D avatar, which can be performed on a server or client device depending on available resources and network conditions. The distributed artificial intelligence (AI) platform supports both configurations and dynamically selects the optimal method based on current operating parameters.

[0140] For Option 1, when the virtual assistant avatar is rendered on the server, the system must address several inherent technical considerations of server-side rendering. While this approach offloads computational demands from the client device, it introduces challenges related to network infrastructure and real-time streaming. Server-side rendering requires robust network connectivity to handle streaming data, sufficient bandwidth to transmit high-quality video streams, and sufficiently low latency to maintain responsiveness. The implementation employs streaming protocols optimized for real-time content delivery, and the server infrastructure must possess sufficient computing power to handle concurrent rendering and streaming operations.

[0141] Figure 11 The specific configuration for server-side avatar rendering is illustrated. The implementation creates a virtual assistant avatar that executes on a web server and streams to the user's device display. A WebSocket connection facilitates real-time bidirectional communication between the client and server, enabling synchronous control and feedback mechanisms. For real-time 3D avatars performing text-to-speech and synchronized lip movements, the system uses dynamic rendering rather than pre-rendered content. The server generates animations in real-time based on phoneme data generated during the TTS process, creating lip or mouth shapes (visemes) corresponding to each phoneme in the speech output.

[0142] The server-side rendering process involves multiple coordinated steps. After the TTS conversion is completed in step 7, the system executes avatar animation by mapping phonemes to corresponding facial animations. Each phoneme is associated with a specific lip shape or mouth shape (viseme) through a predefined animation mapping. The 3D rendering engine processes these mappings in real time, generating an animation sequence synchronized with the audio timeline. The rendering infrastructure uses a professional 3D engine, such as Unity or Unreal Engine, providing server-side rendering capabilities that can be controlled via a script interface.

[0143] The rendered avatar content is streamed to the client device using appropriate streaming technologies. WebRTC provides peer-to-peer real-time communication capabilities suitable for low-latency interaction. Alternatively, video streaming protocols such as Real-Time Messaging Protocol (RTMP) or HTTP Live Streaming (HLS) are used for content distribution. RTMP is optimized for real-time communication with extremely low latency, making it suitable for interactive avatar streaming. HLS uses standard HTTP for data transmission and has broad network infrastructure compatibility, but it typically has higher latency than RTMP, which may affect the quality of real-time interaction.

[0144] When configuring the server to render the avatar, the client application needs to embed a video player capable of receiving and displaying live streaming content. The client application synchronously receives the audio and video streams, and may apply timing adjustments to maintain precise alignment between the avatar's lip movements and the spoken audio. This synchronization compensates for any latency differences introduced by the audio and video in their respective transmission paths.

[0145] For option 2, when the virtual assistant avatar is rendered on the client device, computation is distributed primarily to client-side processing. This configuration performs rendering in the user's web browser or native application environment, locally creating and displaying the animated 3D avatar. The distributed AI platform strategically allocates responsibilities: servers handle computationally intensive TTS processing and phoneme generation, while client devices focus on rendering and synchronization tasks, fully leveraging the advantages of local execution.

[0146] Client-side rendering offers several advantages in a distributed architecture. Local rendering eliminates the bandwidth requirements of video streaming, reducing network traffic to only audio and synchronization data transmission. This configuration also minimizes avatar response latency because rendering occurs directly on the display device, eliminating network round-trip latency. The server utilizes the TTS API infrastructure described in step 7 to generate audio responses and phoneme synchronization data. Real-time communication between the server and client is conducted via a WebSocket connection, transmitting audio data and phoneme timing information, rather than video streaming.

[0147] The client device receives audio and phoneme data from the server and performs local avatar rendering. The browser or application loads the 3D avatar model and animation system, then processes the received phoneme data to generate corresponding facial animations. The rendering engine on the client device creates smooth transitions between lip movements and synchronizes them with the audio playback timeline. The client animates facial features while playing the received audio, maintaining precise synchronization through phoneme timing data. This distributed approach balances the computational load across the network, utilizing server resources for complex AI / machine learning (ML) processing while leveraging client devices for close-to-display rendering tasks.

[0148] Step 9 completes the interaction cycle by storing the response locally and privately in the device's database. This local storage creates a persistent record of the interaction, which can enhance future service experiences through improved personalization and response caching. The storage operation can occur at any time after the device receives the text-based response from Step 6 and runs asynchronously with the audio and visual presentation steps to optimize the user experience. This local caching mechanism helps the platform switch between network-dependent and local processing modes, as caching responses allows future queries to be satisfied even without network access.

[0149] Figure 12 Figure 1200 illustrates Scenario 2 of a distributed AI service platform operation, specifically for scenarios with poor network connectivity but where user devices possess sufficient computing resources. This scenario optimizes degraded network conditions and minimizes data transfer demands by strategically allocating processing tasks, while maintaining full service functionality. This configuration is suitable for user devices with sufficient computing resources, including CPU, GPU, and memory capacity, as well as adequate battery power to support local processing operations. Despite these local resources, the device still lacks product information to answer user queries, thus requiring communication with a network server.

[0150] The difference between Scenario 1 and Scenario 2 lies in the strategic migration of bandwidth-intensive processing tasks from the network to the user device. In Scenario 2, both 3D avatar rendering and text-to-speech conversion are performed locally on the device, such as... Figure 12 As shown. This architectural decision directly addresses situations where the network cannot support high-bandwidth video or audio streaming with acceptable latency. By performing these operations locally, the system reduces network traffic to only the necessary text-based communication, transmitting queries and responses as compact text strings rather than bandwidth-intensive multimedia streams.

[0151] Steps 1 through 6 follow the operational sequence described in Scenario 1, but with a key difference in data transmission strategy. The user initiates interaction by scanning a product item and providing a voice query; the device locally converts the voice to text. The device then generates embeddings from the text query and captured images—requiring only about 2 KB of data transmission—which are sent to the service engine, rather than the original images or audio. The service engine retrieves relevant product information from the service model and leverages its large language model capabilities to generate a text response. Notably, the server only sends text-based responses back to the user's device, avoiding the bandwidth demands of audio or video streaming that would be difficult to handle under degraded network connectivity.

[0152] Step 7 represents a crucial adaptation to network limitations, where the user device is responsible for converting the text-based response received from the server into speech format. The device employs a Text-to-Speech (TTS) application programming interface (API) that supports Speech Synthesis Markup Language (SSML), allowing for precise control over speech synthesis parameters. The TTS system generates phoneme or lip-sync data along with audio output, creating synchronization information for embodying synchronized lip-sync animation. This local processing eliminates the need to stream audio content from the server, significantly reducing network bandwidth consumption while maintaining high-quality speech output for the user.

[0153] Step 8 integrates all presentation layer processing onto the user device, including TTS manipulation, phoneme generation, 3D avatar rendering, and audio-video synchronization. The device performs a complete set of operations that would typically be distributed between the client and server under ideal network conditions. This comprehensive local processing is similar to the client-side rendering approach described in Option 2 of Scenario 1, but goes a step further by incorporating TTS and phoneme generation tasks, usually performed on the server, into the server-side execution. The device creates and animates the 3D avatar based on locally generated phoneme data, synchronizing lip movements with audio output in real time. This configuration maintains a smooth and engaging user experience even under poor network conditions, demonstrating the platform's ability to adjust computational distribution based on available resources.

[0154] Step 9 completes the interaction by storing the response locally and privately in the device database, creating a persistent record for future service enhancements. This storage operation can occur at any time after the device receives the text-based response from Step 6, independent of the presentation process. Local caching of responses enhances the platform's resilience because frequently accessed information can be obtained without network access in future queries, potentially transforming recurring queries into fully local operations, as described in Scenario 3.

[0155] This configuration demonstrates the adaptability of a distributed artificial intelligence (AI) platform, enabling it to dynamically adjust the distribution of computational workloads to accommodate network limitations while maintaining quality of service. By transmitting only compact text data over restricted network connections and performing resource-intensive multimedia processing locally, the system can operate efficiently even under challenging network conditions.

[0156] Figure 13 Figure 1300 illustrates Scenario 3 of a distributed AI service platform operation, representing a configuration where the user device has sufficient computing resources and local caching information to satisfy user queries without network dependency. This scenario demonstrates that the platform can run completely autonomously on the user device, providing the fastest response time and maximum privacy protection by eliminating all network communication during query processing.

[0157] Scenario 3 differs from Scenario 1 and Scenario 2 in that the user device operates completely independently, without any interaction with the network server, such as... Figure 13 As shown. This configuration is feasible when the user device meets two key preconditions. First, the device must contain sufficient computing resources, including the processing power of CPU and GPU cores, enough available memory to support model execution, and sufficient battery capacity to support local AI / machine learning (ML) operations. Second, the information needed to answer the user's query must be available in the device's local cache, which can come from previous interactions during the current session or from historical data retained from earlier in-store visits.

[0158] The availability of local information is typically achieved through two mechanisms. When a customer makes repeated queries for the same product during a shopping session, the initial web-based query response is cached locally, allowing subsequent identical or similar queries to be retrieved directly from the cache. Additionally, information from previous in-store visits may be retained in the device's local storage, forming a personalized knowledge base that grows over time. This accumulated local information reduces reliance on network communication, enabling more and more queries to be processed entirely on the device.

[0159] The processing sequence begins with steps 1 through 3, following the same pattern established in Scenario 1. The user initiates the interaction by scanning a product item with the device's camera and providing a voice query about the item. The smartphone performs local speech-to-text conversion, transforming the audio input into text format and displaying the converted text on the screen for user confirmation. The device then uses a locally deployed CLIP model or similar embedding generation framework to generate an embedding from the text query and the captured product image.

[0160] Step 4 marks a departure from network-dependent scenarios, where the user device retrieves relevant information directly from its local vector database using the generated embeddings. This local database contains previously cached product information, including embeddings and relevant metadata from earlier queries. The retrieval process takes place entirely in the device's memory, eliminating network latency and providing near-instantaneous access to stored information.

[0161] Step 5 continues local processing, where the user device prepares a natural language response using a locally deployed Large Language Model (LLM). The local LLM can be a smaller, optimized version, such as Llama3 8B, suitable for mobile deployment, processing the retrieved information to generate a context-appropriate response to the user's query. This local inference eliminates the need for cloud-based AI processing while maintaining response quality suitable for retail assistant scenarios.

[0162] Step 6 involves the device using a locally executed TTS API to convert the generated text response into speech format. The TTS system supports speech synthesis markup languages ​​for precise voice control, generating audio output and phoneme or lip-sync data for avatar animation synchronization. This local TTS processing is consistent with fully autonomous operation mode, preventing any network dependencies in the response delivery process.

[0163] Step 7 covers the complete avatar rendering and synchronization process, performing the same operations as described in Step 8 of Scene 2. The device performs all avatar animation tasks locally, including loading and rendering the 3D avatar model, processing phoneme data to generate corresponding facial animations, and synchronizing lip movements with the audio playback timeline. This local rendering provides smooth, responsive avatar interaction without the latency or bandwidth limitations of network rendering.

[0164] Step 8 completes the interaction cycle by storing both the query and response in the device's local database, corresponding to step 9 in scenario 2. This persistent storage expands the local knowledge base, potentially enabling future related queries to be satisfied locally without network access. The continuous accumulation of interaction history creates an increasingly capable local assistant that can autonomously handle more and more user queries.

[0165] The architecture of the distributed AI service platform achieves a significant reduction in network traffic by adopting an embedding-based representation, an advantage that is particularly evident when comparing data transmission requirements across different scenarios. The platform uses a pre-trained image embedding model to convert visual information into compact vector representations, drastically reducing the amount of data that must be transmitted when network communication is required.

[0166] The platform employs various embedding models for this purpose, with common implementations including the VisionTransformer series of models, Convolutional Neural Networks (CNNs) such as ResNet and EfficientNet, and OpenAI's Contrastive Language-Image Pre-training (CLIP) model. The CLIP model, as the primary implementation example of this disclosure, provides a concrete quantitative basis for network traffic reduction based on embedding representations.

[0167] The CLIP model architecture comprises two main components that work together to create unified embeddings for visual and textual information. The image encoder, implemented as either a visual transformer or a ResNet, processes the visual input to generate fixed-dimensional embeddings. When using the visual transformer variant, the model treats the image as a sequence of patches, while the ResNet variant employs convolutional layers for hierarchical feature extraction. The text encoder, implemented as a Transformer model, processes the text description to generate embeddings in the same dimensional space as the image embeddings, enabling direct comparison and matching of visual and textual representations.

[0168] The specific CLIP variant used in this embodiment, named ViT-B / 32, employs a Vision Transformer backbone network with specific architectural parameters. The visual encoder operates with a 768-dimensional hidden size, representing the dimension of the hidden states in the transformer layers, and contains 12 transformer layers for progressive feature refinement. The text encoder operates with a 512-dimensional hidden size, also containing 12 transformer layers. These dimensional specifications directly affect the model's representational power and the size of the generated embeddings.

[0169] The representations generated by the embedding process are extremely compact compared to the original image data. For the ViT-B / 32 model, the projection dimension is 512, meaning that each embedding contains 512 values. Each value is stored as a 32-bit floating-point number, occupying 4 bytes. The total embedding size is equal to 512 dimensions multiplied by 4 bytes per dimension, resulting in 2048 bytes, or approximately 2 kilobytes per embedding.

[0170] This compact embedded representation achieves significant data reduction compared to transmitting the original image. For example, a typical 1080p resolution image taken by a smartphone camera comprises 1920×1080 pixels. Using a standard 24-bit color depth, with 8 bits allocated to each color channel in the RGB model, the uncompressed image requires 1920×1080×3 bytes, totaling 6,220,800 bytes, approximately 6 megabytes. Even with JPEG compression, typically at a compression ratio of 10:1 to 20:1, the same image would still require 300 to 600 kilobytes for transmission.

[0171] The comparison between embedding size and image size demonstrates the platform's efficiency in minimizing network traffic. Transmitting a 2-kilobyte embeddable image instead of a 6-megabyte uncompressed image represents a 99.97% reduction in data transmission. Even compared to JPEG compressed images, embedding reduces transmission requirements by 93% to 99%. This significant reduction in data transmission requirements allows the platform to remain responsive even on bandwidth-constrained network connections, while also reducing power consumption associated with wireless data transmission in battery-powered devices.

[0172] Figure 14 A flowchart 1400 is shown for implementing a virtual personal assistant mixed reality service through integrated communication and computing. This process can be executed by a user device to provide artificial intelligence (AI) services.

[0173] In step 1402, the user equipment receives a user query for the AI ​​service.

[0174] In step 1404, the user equipment evaluates one or more conditions. These one or more conditions may include at least one of the following: local resources of the user equipment, network connectivity resources between the user equipment and the server, or availability of information on the user equipment for responding to the user's query.

[0175] In step 1406, based on the evaluation, the user equipment determines an operating mode from a plurality of operating modes for performing the AI ​​service. The plurality of operating modes may include a local-only mode and at least one split computation mode.

[0176] In step 1408, the user equipment executes the AI ​​service according to the determined operating mode to provide a response to the user query. In local-only mode, the AI ​​service can be executed entirely on the user equipment. In at least one split computing mode, at least a portion of the AI ​​service can be offloaded to the server.

[0177] In some configurations, the AI ​​service may include a virtual personal assistant mixed reality service applicable to one or more domains.

[0178] In some configurations, the user equipment may further activate the AI ​​service via a detection mechanism, which may include one of the following: scanning a near field communication (NFC) tag, scanning a quick response (QR) code, Bluetooth detection, Wi-Fi Direct, ultra-wideband (UWB), or infrared.

[0179] In some configurations, local resources of a user equipment may include one or more of the following: CPU utilization, memory utilization, battery power, or battery state of charge. Network connectivity resources may include one or more of the following: link bandwidth availability, achievable end-to-end throughput, Radio Access Technology (RAT) availability, or radio quality between the user equipment and the server.

[0180] In some configurations, the server may include one or more of the following: a dedicated server in a Customer Premises Network (CPN), a server managed by a mobile network operator, a server managed by a fixed network operator, a server managed by a satellite network operator, or a cloud-based server. The server can be selected in real time based on service requirements and resource availability.

[0181] In some configurations, assessing the availability of information may include checking on the user device's local database for cached information sufficient to respond to the user's query. The user device may then store the response in the local database for future use.

[0182] In some configurations, at least one split computation mode may be selected when an assessment determines that network connectivity is available and at least one of the following conditions is met: insufficient local resources or information is unavailable on the user's device. Performing AI services may include: transferring data derived from a user query to a server, the data containing at least one of an audio stream of the user query or an embedding of the user query; and receiving a response from the server. The response may include at least one of: (a) an audio stream and a video stream of the rendered virtual assistant; or (b) an audio stream and phoneme data for synchronizing lip movements of the virtual assistant rendered on the user's device.

[0183] In some configurations, when assessments determine that network connectivity is poor, local resources are sufficient, and information is unavailable on the user's device, at least one split computing mode can be selected. Performing AI services may include: generating an embedding based on a user query by the user's device; transmitting the embedding to a server; receiving a text response from the server; generating an audio stream by the user's device based on the text response; and rendering a virtual assistant by the user's device that synchronizes lip movements with the generated audio stream.

[0184] In some configurations, when the assessment determines that local resources are sufficient and the information is available in the local database on the user's device, a local-only mode can be selected. Executing AI services may include: retrieving data from the local database to respond to the user's query; generating a text response by the user's device using a local language model based on the retrieved data; and generating an audio stream and corresponding synchronous virtual assistant rendering by the user's device based on the text response.

[0185] In some configurations, the user equipment may further, before evaluating the one or more conditions,: send a service discovery request to the service broker containing information related to the one or more conditions; and receive from the service broker information for communicating with the server, including load balancing or redirection to one of a number of servers.

[0186] In some configurations, the user equipment may further receive credentials for the Customer Premises Network (CPN) from the service broker; and based on user preferences, connect to the CPN to communicate with the server, or refuse to connect to the CPN and communicate with the server via the subscribed mobile network.

[0187] In some configurations, the user device may further obtain the user's consent to one or more of the following: taking photos for identification purposes, storing interaction data locally on the user device, or recording interactions for future personalization.

[0188] In some configurations, the user device may further use one or more machine learning models on the user device or server to provide multilingual support for AI services, which can be selected based on user preferences.

[0189] Figure 15 A flowchart is shown for another process 1500 for realizing a virtual personal assistant mixed reality service through integrated communication and computing. This process can be executed by a user device to provide virtual assistant services.

[0190] In step 1502, the user equipment receives a user query, which includes at least one of an image of an object or voice input related to the object.

[0191] In step 1504, the user equipment uses a pre-trained model to generate an embedding vector based on the user query.

[0192] In step 1506, the user equipment transmits the embedding vector to the server via the network. The data size of the embedding vector may be significantly smaller than the data size of at least one of the image or the voice input to reduce network traffic.

[0193] In step 1508, the user equipment receives a response from the server, which is generated by the server based on the embedding vector and provides information related to the object.

[0194] In some configurations, the pre-trained model may include a contrastive language-image pre-trained (CLIP) model. The embedding vector data size is approximately 2 kilobytes.

[0195] In some configurations, the response received from the server may include the audio and video streams of the rendered virtual assistant. The user device may then play the audio stream and display the video stream of the rendered virtual assistant.

[0196] In some configurations, the response received from the server may include an audio stream and corresponding phoneme data. The user device may further render an avatar of a virtual assistant whose lip movements are synchronized with the audio stream based on the phoneme data.

[0197] In some configurations, the response received from the server may include a text response. The user device may further use a local text-to-speech (TTS) module to generate an audio stream from the text response and render an avatar of a virtual assistant whose lip movements are synchronized with the generated audio stream.

[0198] In some configurations, the user device can further select or customize an avatar for the virtual assistant service based on user input. This avatar may include one of the following: a default avatar, a male avatar, a female avatar, a child avatar, or a pet avatar.

[0199] Figure 16 A flowchart 1600 is shown, illustrating the process for preparing a service model for a virtual assistant service.

[0200] In step 1602, text information of multiple items in one or more domains is stored in a first database on the server, the text information containing at least a unique item name for each item.

[0201] In step 1604, multiple images are captured for each of the multiple projects using the user's device's camera.

[0202] In step 1606, an image embedding is generated for each captured image using a pre-trained model.

[0203] In step 1608, the image embedding and its corresponding unique project name are stored in a second database on the server.

[0204] In step 1610, the service model is fine-tuned by verifying the retrieved information for the sample items and updating the image embeddings as needed. The first and second databases can be configured to be queried by the service engine in response to user queries about the multiple items.

[0205] In some configurations, the pre-trained model may include a contrastive language-image pre-trained (CLIP) model. This one or more domains may include retail, healthcare, manufacturing, or agriculture.

[0206] It should be understood that the specific order or hierarchy of steps in the disclosed process / flowchart is merely exemplary. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process / flowchart may be rearranged. Furthermore, some steps may be combined or omitted. The appended method claims present the step elements in an exemplary order and are not limited to the specific order or hierarchy shown.

[0207] The foregoing description is intended to enable those skilled in the art to practice the aspects described herein. Various modifications to these aspects are readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to limit the aspects shown herein, but rather to impose a full scope consistent with the language of the claims, wherein a singular reference to an element does not mean “only one” unless explicitly stated otherwise, but rather “one or more.” The term “exemplary” is used herein to mean “as an example, instance, or illustration.” Any aspect described as “exemplary” is not necessarily to be construed as preferred or superior to other aspects. Unless otherwise explicitly stated, the term “some” means one or more. Combinations such as “at least one A, B, or C,” “one or more A, B, or C,” “at least one A, B, and C,” “one or more A, B, and C,” and “A, B, C, or any combination thereof” include any combination of A, B, and / or C, and may include multiple A, multiple B, or multiple C. Specifically, combinations such as “at least one A, B, or C,” “one or more A, B, or C,” “at least one A, B, and C,” “one or more A, B, and C,” and “A, B, C, or any combination thereof” can be only A, only B, only C, A and B, A and C, B and C, or A and B and C, wherein any such combination may include one or more members of A, B, or C. All structures and functions known or subsequently known to those skilled in the art that are equivalent to the elements of the aspects described herein are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not it is expressly stated in the claims. The terms “module,” “mechanism,” “element,” “device,” etc., are not substitutes for the word “means.” Therefore, no element of a claim should be construed as means plus function unless the element is expressly expressed using the phrase “means for…”.

Claims

1. A method performed by a user device for providing artificial intelligence (AI) services, comprising: Receive user queries for this AI service; Evaluate one or more conditions, wherein the one or more conditions include at least one of the following: local resources of the user equipment, network connectivity resources between the user equipment and the server, or information availability on the user equipment for responding to the user's query; Based on this assessment, an operating mode is determined from multiple operating modes for performing the artificial intelligence service, including a local-only mode and at least one split computation mode. as well as The AI ​​service is executed according to the determined operating mode to provide a response to the user's query, wherein in the local-only mode, the AI ​​service is executed entirely on the user's device, and wherein in the at least one split computing mode, at least a portion of the AI ​​service is offloaded to the server.

2. The method of claim 1, wherein the artificial intelligence service includes a virtual personal assistant mixed reality service applicable to one or more domains.

3. The method of claim 1, further comprising activating the artificial intelligence service on the user device via a detection mechanism, the detection mechanism including one of: scanning a near-field communication tag, scanning a quick response code, Bluetooth detection, Wi-Fi Direct, ultra-wideband, or infrared.

4. The method of claim 1, wherein the local resources of the user equipment include one or more of the following: CPU utilization, memory utilization, battery power or battery charging status, and wherein the network connectivity resources include one or more of the following: link bandwidth availability, achievable end-to-end throughput, wireless access technology availability or radio quality between the user equipment and the server.

5. The method of claim 1, wherein the server comprises one or more of the following: a dedicated server in a customer premises network, a server managed by a mobile network operator, a server managed by a fixed network operator, a server managed by a satellite network operator, or a cloud-based server, and wherein the server is selected in real time based on service demand and resource availability.

6. The method of claim 1, wherein assessing the availability of the information includes checking on the user device whether there is cached information in a local database sufficient to respond to the user query, and wherein the method further includes storing the response in the local database for future use.

7. The method of claim 1, wherein when the assessment determines that at least one of the following conditions is met—that the network connection is available, that the local resources are insufficient, or that the information is unavailable on the user device—the at least one split computation mode is selected, and wherein performing the artificial intelligence service includes: The data exported from the user query is transmitted to the server, including at least one of the audio stream of the user query or the embedding of the user query; as well as Receive a response from the server, wherein the response includes at least one of the following: (a) The audio and video streams of the rendered virtual assistant; or (b) Audio stream and phoneme data used to synchronize the lip movements of the virtual assistant rendered on the user's device.

8. The method of claim 1, wherein when the assessment determines that the network connection is poor, the local resources are sufficient, and the information is unavailable on the user device, the at least one split computation mode is selected, and wherein performing the artificial intelligence service includes: The embedding is generated by the user equipment based on the user's query; The embedding is transmitted to the server; Receive a text response from the server; The user device generates an audio stream from the text response; as well as A virtual assistant whose lip movements are rendered by the user device and synchronized with the generated audio stream.

9. The method of claim 1, wherein when the assessment determines that the local resources are sufficient and the information is available in a local database on the user device, the local-only mode is selected, and wherein performing the artificial intelligence service includes: Retrieve data from the local database to respond to the user's query; The user equipment generates a text response based on the retrieved data using a local language model; as well as The user device generates an audio stream and a corresponding synchronous virtual assistant rendering based on the text response.

10. The method of claim 1, further comprising, before evaluating the one or more conditions: Send a service discovery request to the service broker, the service discovery request including information related to the one or more conditions; and Receive information from the service agent for communicating with the server, including load balancing or redirection to one of multiple servers.

11. The method of claim 10, further comprising: Receive credentials for the customer's local network from the service agent; as well as Based on user preferences, connect to the customer premises network to communicate with the server, or refuse to connect to the customer premises network and communicate with the server via the subscribed mobile network.

12. The method of claim 1, further comprising: Obtain the user's consent to one or more of the following: taking photos for identification purposes, storing interaction data locally on the user's device, or recording interactions for future personalization.

13. The method of claim 1, further comprising using one or more machine learning models on the user device or the server to provide multilingual support for the artificial intelligence service, the multilingual support being selected based on user preferences.

14. A method performed by a user device for providing a virtual assistant service, comprising: Receive user queries, which include at least one of an image of an object or voice input related to the object; Use a pre-trained model to generate an embedding vector based on the user query; The embedding vector is transmitted to the server over the network, wherein the data size of the embedding vector is significantly smaller than the data size of at least one of the image or the speech input, in order to reduce network traffic; as well as The server receives a response, which is generated by the server based on the embedding vector, and provides information related to the object.

15. The method of claim 14, wherein the pre-trained model comprises a contrastive language-image pre-trained model, and wherein the data size of the embedding vector is approximately 2 kilobytes.

16. The method of claim 14, wherein the response received from the server includes an audio stream and a video stream of the rendered virtual assistant, and wherein the method further comprises: Play the audio stream and display the rendered video stream of the virtual assistant.

17. The method of claim 14, wherein the response received from the server includes an audio stream and phoneme data corresponding to the audio stream, and wherein the method further includes: The virtual assistant avatar is rendered based on the phoneme data, with lip movements synchronized with the audio stream.

18. The method of claim 14, wherein the response received from the server comprises a text response, and wherein the method further comprises: Use the local text-to-speech module to generate an audio stream from the text response; as well as An avatar of a virtual assistant that renders lip movements in sync with the generated audio stream.

19. The method of claim 14, further comprising: The virtual assistant service selects or customizes an avatar based on user input, where the avatar includes one of the following: a default avatar, a male avatar, a female avatar, a child avatar, or a pet avatar.

20. A method for preparing a service model for a virtual assistant service, comprising: The first database on the server stores text information about multiple items in one or more domains, and the text information includes at least a unique item name for each item. Use the user's device's camera to capture multiple images for each of the multiple projects; Use a pre-trained model to generate image embeddings for each captured image; The image embedding and its corresponding unique project name are stored in a second database on the server. as well as The service model is fine-tuned by validating the retrieved information from sample items and updating the image embeddings as needed, wherein the first and second databases are configured to be queried by the service engine in response to user queries about the multiple items.

21. The method of claim 20, wherein the pre-trained model comprises a contrastive language-image pre-trained model, and wherein the one or more domains comprise retail, healthcare, manufacturing, or agriculture.

22. A user device for providing artificial intelligence (AI) services, comprising: Memory; as well as At least one processor, coupled to the memory, is configured as follows: Receive user queries for this AI service; Evaluate one or more conditions, wherein the one or more conditions include at least one of the following: local resources of the user equipment, network connectivity resources between the user equipment and the server, or availability of information on the user equipment for responding to the user's query; Based on this assessment, an operating mode is determined from a plurality of operating modes for performing the artificial intelligence service, including a local-only mode and at least one split computation mode. as well as The AI ​​service is executed according to the determined operating mode to provide a response to the user's query, wherein in the local-only mode, the AI ​​service is executed entirely on the user's device, and wherein in the at least one split computing mode, at least a portion of the AI ​​service is offloaded to the server.

23. A computer-readable medium storing computer-executable code for execution by a user device to provide artificial intelligence services, comprising code for: Receive user queries for this AI service; Evaluate one or more conditions, wherein the one or more conditions include at least one of the following: local resources of the user equipment, network connectivity resources between the user equipment and the server, or availability of information on the user equipment for responding to the user's query; Based on this assessment, an operating mode is determined from a plurality of operating modes for performing the artificial intelligence service, including a local-only mode and at least one split computation mode. as well as The AI ​​service is executed according to the determined operating mode to provide a response to the user's query, wherein in the local-only mode, the AI ​​service is executed entirely on the user's device, and wherein in the at least one split computing mode, at least a portion of the AI ​​service is offloaded to the server.

24. A user device for providing virtual assistant services, comprising: Memory; as well as At least one processor, coupled to the memory, is configured as follows: Receive user queries, which include at least one of an image of an object or voice input related to the object; Use a pre-trained model to generate an embedding vector based on the user query; The embedding vector is transmitted to the server over the network, wherein the data size of the embedding vector is significantly smaller than the data size of at least one of the image or the speech input, in order to reduce network traffic; as well as Receive a response from the server, which is generated by the server based on the embedding vector and provides information related to the object.

25. A computer-readable medium storing computer-executable code for providing virtual assistant services, executed by a user device, comprising code for: Receive user queries, which include at least one of an image of an object or voice input related to the object; Use a pre-trained model to generate an embedding vector based on the user query; The embedding vector is transmitted to the server over the network, wherein the data size of the embedding vector is significantly smaller than the data size of at least one of the image or the speech input, in order to reduce network traffic; and Receive a response from the server, which is generated by the server based on the embedding vector and provides information related to the object.

26. An apparatus for preparing a service model for a virtual assistant service, comprising: Memory; as well as At least one processor, coupled to the memory, is configured as follows: The first database on the server stores text information about multiple items in one or more domains, and the text information includes at least a unique item name for each item. Use the user's device's camera to capture multiple images for each of the multiple projects; Use a pre-trained model to generate image embeddings for each captured image; The image embedding and its corresponding unique project name are stored in a second database on the server. as well as The service model is fine-tuned by validating the retrieved information from sample items and updating the image embeddings as needed, wherein the first and second databases are configured to be queried by the service engine in response to user queries about the multiple items.

27. A computer-readable medium storing computer-executable code for preparing a service model for a virtual assistant service, comprising code for: The first database on the server stores text information about multiple items in one or more domains, and the text information includes at least a unique item name for each item. Use the user's device's camera to capture multiple images for each of the multiple projects; Use a pre-trained model to generate image embeddings for each captured image; The image embedding and its corresponding unique project name are stored in a second database on the server. as well as The service model is fine-tuned by validating the retrieved information from sample items and updating the image embeddings as needed, wherein the first and second databases are configured to be queried by the service engine in response to user queries about the multiple items.

Images