Electronic device for controlling RF performance and storage medium thereof

Abstract

Disclosed is an electronic device comprising a neural processing unit (NPU), an RF circuit connected to at least one antenna, a modem for cellular communication, and a shared memory. The NPU may obtain, from the shared memory, time-sensitive input information and non-time-sensitive input information obtained from the modem and a sensor fusion module and stored in the shared memory, input the time-sensitive input information and the non-time-sensitive input information to a multi-modal deep learning algorithm and thereby generate output information for use in operating the RF circuit and the modem, and store the output information in the shared memory in order to make the RF circuit and the modem operate according to the output information.

Description

Electronic device for controlling RF performance and its storage medium

[0001] Embodiments of the present disclosure relate to an electronic device for controlling RF (radio frequency) performance and a storage medium thereof.

[0002] Thanks to the remarkable advancements in information and communication technology and semiconductor technology, the proliferation and use of various electronic devices is rapidly increasing. Electronic devices are being developed to be carried by users and to communicate with other devices using wireless communication technology. A wireless communication electronic device can refer to any device that performs a specific function based on its installed software, such as a smartphone, mobile terminal, electronic organizer, portable multimedia player, tablet PC (personal computer), notebook PC, audio / video device, desktop / laptop computer, home appliance, or in-vehicle navigation system.

[0003] In wireless communication environments such as 3G (3rd generation), 4G (4th generation), LTE (long term evolution), and / or 5G NR (5th generation new radio), various technologies such as beamforming, antenna power control, modulation and multiplexing, and DRX (discontinuous reception) can be used to secure stable communication quality, improve data transmission speed, and increase communication efficiency.

[0004] Channel estimation can estimate channel conditions by identifying the surrounding electric field using a reference signal containing a specified sequence, and compensate for the received signal based on the estimated channel information. Antenna power control can include adjusting the transmission power and signal direction using beamforming and / or precoding. Beamforming and / or precoding can reduce interference, improve communication quality, and improve energy efficiency by focusing signals on specific users, thereby reducing transmission power consumption.

[0005] Modulation and multiplexing may include modulation schemes based on quadrature phase shift keying (QPSK) and / or quadrature amplitude modulation (QAM), and multi-carrier transmission techniques such as orthogonal frequency division multiplexing (OFDM) or universal filtered multi-carrier (UFMC). DRX can reduce unnecessary receiving operations of electronic devices, thereby saving battery consumption.

[0006] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0007] Embodiments of the present disclosure may provide an electronic device and a storage medium thereof including a shared memory for direct communication between a modem and a neural processing unit (NPU).

[0008] Embodiments of the present disclosure can provide an electronic device and a storage medium thereof for controlling the performance of an RF and a modem based on a multi-modal deep learning model.

[0009] Embodiments of the present disclosure can provide an electronic device and a storage medium thereof that perform a multi-modal deep learning-based RF context-aware AI (artificial intelligence) algorithm for improving RF and modem performance of a mobile communication system.

[0010] Embodiments of the present disclosure can provide an electronic device and a storage medium thereof that collectively determine modem control parameters and RF control parameters so as to reflect various input information to a multi-modal deep learning model in real time and non-real time and improve communication quality and energy efficiency.

[0011] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0012] An electronic device according to one embodiment of the present disclosure may include a neural processing unit (NPU), an RF circuit connected to at least one antenna, a modem configured to perform cellular communication, and a shared memory configured to be accessible by the NPU, the modem, and a sensor fusion module associated with at least one sensor, and storing time-sensitive input information and non-time-sensitive input information acquired from the modem and the sensor fusion module. The NPU may be configured to: acquire the time-sensitive input information and the non-time-sensitive input information from the shared memory. The NPU may be configured to input the time-sensitive input information and the non-time-sensitive input information into a multi-modal deep learning algorithm, thereby generating output information for use in operating the RF circuit and the modem. The above NPU is configured to store the output information generated by the multi-modal deep learning algorithm that takes the time-sensitive input information and the non-time-sensitive input information as inputs in the shared memory, so that the RF circuit and the modem can operate by the output information.

[0013] In accordance with one embodiment of the present disclosure, a non-transitory computer-readable storage medium storing one or more programs may include instructions that, when executed by a neural processing unit (NPU) of an electronic device, cause the electronic device to: obtain time-sensitive input information and non-time-sensitive input information obtained from a modem for cellular communication and a sensor fusion module associated with at least one sensor and stored in a shared memory, and input the time-sensitive input information and the non-time-sensitive input information into a multi-modal deep learning algorithm to generate output information for use in operating an RF circuit connected to at least one antenna and the modem, and store the output information generated by the multi-modal deep learning algorithm that has the time-sensitive input information and the non-time-sensitive input information as inputs in the shared memory, thereby causing the RF circuit and the modem to operate by the output information.

[0014] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0015] FIG. 2 is a drawing for explaining an example of a mobile communication system according to one embodiment of the present disclosure.

[0016] FIG. 3 is a diagram for explaining a structure for performing rule-based channel estimation in a mobile communication system according to one embodiment of the present disclosure.

[0017] FIG. 4 is a diagram for explaining a structure for performing channel estimation using a multi-modal deep learning model according to one embodiment of the present disclosure.

[0018] FIG. 5 is a diagram showing the input and output of a context-aware AI solution based on multi-modal deep learning according to one embodiment of the present disclosure.

[0019] FIG. 6 is a flowchart illustrating an execution procedure of a multi-modal deep learning algorithm according to an embodiment of the present disclosure.

[0020] FIG. 7 is a diagram illustrating a transmission procedure using a multi-modal deep learning algorithm according to one embodiment of the present disclosure.

[0021] FIG. 8 is a diagram illustrating a receiving procedure using a multi-modal deep learning algorithm according to one embodiment of the present disclosure.

[0022] FIG. 9 illustrates a software (SW) layer structure for executing a multi-modal deep learning-based AI solution according to one embodiment of the present disclosure.

[0023] FIG. 10 is a diagram for explaining the operation procedure of a context-aware AI solution based on multi-modal deep learning according to one embodiment of the present disclosure.

[0024] FIG. 11a and FIG. 11b illustrate a user interface (UI) for setting up application of a multi-modal deep learning-based AI solution according to one embodiment of the present invention.

[0025] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings. In describing the embodiments of the present disclosure, detailed descriptions of related known functions or configurations will be omitted if they are determined to unnecessarily obscure the gist of the present disclosure. Furthermore, the terms described below are defined based on their functions in the embodiments of the present disclosure and may vary depending on the intentions or practices of users and operators. Therefore, their definitions should be based on the overall content of the present disclosure.

[0026] It should be noted that the technical terms used in this disclosure are used merely to describe one embodiment and are not intended to limit the present disclosure. Alternatively, unless specifically defined otherwise in this disclosure, the technical terms used in this disclosure should be interpreted as having a meaning generally understood by those skilled in the art to which this disclosure pertains, and should not be interpreted in an excessively broad or narrow sense. Alternatively, the technical terms used in this disclosure may be understood and replaced with other technical terms that are understandable to those skilled in the art. General terms used in the embodiments of this disclosure should be interpreted as defined in the dictionary or according to the context, and should not be interpreted in an excessively narrow sense.

[0027] As used herein, singular expressions may include plural expressions unless the context clearly dictates otherwise. In this disclosure, terms such as "consist of" or "include" should not necessarily be construed to include all components or operations described in the specification, and should be construed to mean that some of the components or operations may not be included, or that additional components or operations may be included.

[0028] While terms including ordinal numbers, such as "first" and "second," used herein may be used to describe various components, these components should not be limited by these terms. These terms may only be used to distinguish one component from another. For example, without departing from the scope of the present disclosure, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component."

[0029] When a component is referred to as being "connected" or "connected" to another component, it may be directly connected or connected to that other component, but there may also be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0030] Hereinafter, embodiments according to the present disclosure will be described with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers and redundant descriptions thereof will be omitted. In describing embodiments of the present disclosure, if a detailed description of a related known technology is determined to obscure the gist of the present disclosure, the detailed description thereof will be omitted. It should be noted that the attached drawings are only intended to facilitate easy understanding of embodiments of the present disclosure and should not be construed as limiting the present disclosure by the attached drawings. The present disclosure should be construed to extend to all modifications, equivalents, and substitutes other than the attached drawings.

[0031] In the present disclosure, embodiments will be described using an electronic device as an example, but the electronic device may also be referred to as a terminal, a mobile station, mobile equipment (ME), user equipment (UE), user terminal (UT), subscriber station (SS), wireless device, handheld device, or access terminal (AT). In the embodiments of the present disclosure, the electronic device may be a device having a communication function, such as, for example, a mobile phone, a personal digital assistant (PDA), a smart phone, a wireless MODEM, or a laptop.

[0032] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0033] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0034] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0035] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0036] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0037] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0038] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0039] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0040] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0041] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0042] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0043] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0044] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0045] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0046] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0047] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0048] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0049] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0050] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0051] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a transceiver (radio frequency integrated circuit)) may be additionally formed as a part of the antenna module (197).

[0052] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, a transceiver disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0053] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0054] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0055] FIG. 2 is a diagram illustrating an example of a mobile communication system according to one embodiment of the present disclosure. Here, a multiple user (MU) MIMO (multiple input multiple output) system is illustrated as an example.

[0056] Referring to FIG. 2, a base station (210) (e.g., an eNB (LTE (long-term evolution) node B) or a gNB (NR node B)) may use multiple antennas (212) (e.g., an array antenna) to transmit downlink (DL) signals or receive uplink (UL) signals to one or more user equipments (UEs) (222, 224, 226) (e.g., an electronic device (101)).

[0057] Downlink signals transmitted through a plurality of antennas (212) may reach user terminals (222, 224, 226) via a channel (200). The downlink signals may include a downlink reference signal (RS) defined as a sequence agreed upon in advance between a base station (210) and user terminals (222, 224, 226). The user terminals (222, 224, 226) may perform channel estimation based on the reference signal to determine channel estimation information indicating downlink characteristics of the channel (200), and may perform channel compensation and demodulation on data signals included in the downlink signals based on the channel estimation information, and / or adaptively adjust transmission parameters (e.g., modulation and coding scheme (MCS)) for downlink transmission.

[0058] Uplink signals transmitted from user terminals (222, 224, 226) may be received through a plurality of antennas (212) via a channel (200). The uplink signals may include an uplink reference signal (RS) defined as a sequence agreed upon in advance between a base station (210) and user terminals (222, 224, 226). The base station (220) determines channel estimation information indicating uplink characteristics of the channel (200) by performing channel estimation based on the reference signal, and performs channel compensation and demodulation on data signals included in the uplink signals based on the channel estimation information, or adaptively adjusts transmission parameters (e.g., MCS) for uplink transmission.

[0059] FIG. 3 is a diagram for explaining a structure for performing rule-based channel estimation in a mobile communication system according to one embodiment of the present disclosure.

[0060] Referring to FIG. 3, an electronic device (300) (e.g., electronic device (101)) may include an RF circuit (302) and a processor (310). The RF circuit (302) may include an RF front end (RFFE) (304) and a transceiver (306). The RFFE (304) may be configured to filter and amplify RF signals of a specified frequency band (e.g., an RF band). As an example, the RFFE (304) may filter and amplify an RF signal from the transceiver (306) and transmit it to an antenna (304a), or may amplify and filter an RF signal received via the antenna (304a) and transmit it to the transceiver (306).

[0061] The processor (310) may be comprised of at least one processing circuit, such as an application processor or a communication processor. In one embodiment, the processor (310) may include at least one of a central processing unit (CPU) (320), a digital signal processor (DSP) (316), a modulator and demodulator (MODEM) (314), a neural processing unit (NPU) (312), or a sensor fusion module (318).

[0062] The modem (314) may encode and modulate a digital signal transmitted from the DSP (316) and transmit it to the transceiver (306), or may demodulate and decode a baseband signal transmitted from the transceiver (306) and transmit it to the DSP (316). The sensor fusion module (318) may merge sensing information collected from at least one sensor and / or at least one communication chip and provide it to the CPU (320) and / or the NPU (312). In one embodiment, the CPU (320) or the NPU (316) may perform rule-based channel estimation based on information measured by the transceiver (306), the modem (314), and / or the sensor fusion module (318), thereby controlling the modem (314) and the RF circuit (302).

[0063] In one embodiment, rule-based channel estimation methods can be inaccurate in complex communication environments (e.g., 5G environments). Furthermore, they lack real-time response to user context information and RF environment changes, making it difficult to optimize antenna power control, modulation, and multiplexing. Furthermore, inefficient DRX cycle settings can lead to unnecessary power consumption and lower reception success rates.

[0064] In one embodiment, the modem (314), DSP (316), NPU (312), and CPU (320) may be configured to operate independently. For example, since the NPU (312) cannot directly communicate with the modem (314), it may exchange data with the modem (314) through an indirect interface via external memory (e.g., dynamic random access memory (DRAM)). This structure may increase system complexity and slow down data processing speed.

[0065] FIG. 4 is a diagram for explaining a structure for performing channel estimation using a multi-modal deep learning model according to one embodiment of the present disclosure.

[0066] Referring to FIG. 4, an electronic device (400) (e.g., the electronic device (101) of FIG. 1) may include an RF circuit (402) (e.g., the wireless communication module (192) of FIG. 1) and a processor (410) (e.g., the processor (120) of FIG. 1). The RF circuit (402) may include an RF front end (RFFE) (404) and a transceiver (406). The RFFE (404) may be configured to filter and amplify RF signals of a specified frequency band (e.g., an RF band). As an example, the RFFE (404) may filter and amplify an RF signal from the transceiver (406) and transmit it to an antenna (404a) (e.g., the antenna module (197) of FIG. 1), or may amplify and filter an RF signal received via the antenna (404a) and transmit it to the transceiver (406).

[0067] The processor (410) may be comprised of at least one processing circuit, such as an application processor (AP) and / or a communication processor (CP). In one embodiment, the processor (410) may include at least one of a central processing unit (CPU) (420), a digital signal processor (DSP) (416), a modem (414) configured to perform cellular communication (e.g., a cellular modem), a neural network processing unit (NPU) (412), a sensor fusion module (418), or a shared memory (422).

[0068] In one embodiment, the NPU (412) may be connected to a shared memory (422) and a memory (e.g., memory (130)) that stores instructions. The instructions, when executed by the NPU (412), may cause the electronic device (400) to control at least one of the antenna (404a), the RFFE (404), the transceiver (406), or the modem (414) based on executing a multi-modal deep learning algorithm according to embodiments of the present disclosure.

[0069] In one embodiment, a shared memory (422) may be connected between the NPU (412) and the sensor fusion module (418) and the modem (414) for direct communication between the NPU (412) and the sensor fusion module (418) and the modem (414).

[0070] The modem (414) may encode and modulate a digital signal transmitted from the DSP (416) and transmit it to the transceiver (406) according to a designated radio access technology (RAT) (e.g., at least one of 3G, 4G, LTE, or 5G NR), or may demodulate and decode a baseband signal transmitted from the transceiver (406) and transmit it to the DSP (416). The modem (414) may store information measured during the process of transmitting or receiving signals (e.g., at least one of the signal quality information (512a) or the cell environment information (512b) of FIG. 5) in the shared memory (422) so that the information can be used as input for a multi-modal deep learning model (e.g., the multi-modal deep learning model (500) of FIG. 5) by the NPU (412).

[0071] The sensor fusion module (418) merges and preprocesses (e.g., embedding, concatenation, and / or dimension reduction) sensing information collected from at least one sensor (e.g., at least one of a sensor module (176), a gyro sensor, a geomagnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a biometric sensor, a temperature sensor, an altitude sensor, a light intensity sensor, a humidity sensor, an oxygen sensor, a heart rate sensor, a blood pressure sensor, an on sensor, or an electrocardiogram sensor) and / or at least one communication chip (e.g., at least one of a global positioning system (GPS) module, a Wi-Fi chip, a Bluetooth chip, or an UWB chip)) and processes the processed sensing information (e.g., at least one of location information (514a), movement information (514b), device environment information (514c), user behavior pattern information (522a), surrounding environment information (522b), biometric information (522c), or device information (522d) of FIG. 5). It can be stored in shared memory (422) so that it can be used as input for a multi-modal deep learning model (500) by NPU (412).

[0072] In one embodiment, the NPU (422) may execute a multi-modal deep learning model (500) (e.g., a multi-modal deep learning algorithm) based on information stored in the shared memory (422) (e.g., time-sensitive input information (510) and non-time-sensitive input information (520) of FIG. 5) to generate output information (e.g., output information (530) of FIG. 5) for controlling the modem (414) and the RF circuit (402) and store the output information in the shared memory (422). In one embodiment, the multi-modal deep learning model (500) may preprocess (e.g., embedding and normalizing) various types of input information (e.g., time-sensitive input information (510) and non-time-sensitive input information (520) of FIG. 5) and use them as inputs of a context-aware AI solution.

[0073] Embodiments of the present disclosure can improve data processing speed and reduce system complexity through an architecture including a shared memory (422) for direct communication between a modem (414) and a sensor fusion module (418) and an NPU (412). Embodiments of the present disclosure can improve RF performance and communication quality and maximize energy efficiency by reflecting various input information (e.g., time-sensitive input information (510) and non-time-sensitive input information (520) of FIG. 5) in real time and comprehensively optimizing RF control parameters such as beamforming, power control, modulation and coding scheme, and DRX (e.g., output information (520) of FIG. 5) by utilizing a multi-modal deep learning-based context recognition artificial intelligence (AI) solution.

[0074] The electronic device (400) includes a shared memory (422) that enables direct communication between the modem (414), the NPU (412), and the sensor fusion module (418), thereby improving data transmission speed between the modem (414), the NPU (412), and the sensor fusion module (418) and reducing system complexity, thereby increasing efficiency. In one embodiment, the NPU (412) can execute a context-aware AI solution based on multi-modal deep learning based on input information stored in the shared memory (422).

[0075] FIG. 5 is a diagram showing the input and output of a context-aware AI solution based on multi-modal deep learning according to one embodiment of the present disclosure.

[0076] Referring to FIG. 5, the NPU (412) can execute a context-aware AI solution based on multi-modal deep learning based on input information stored in shared memory (422). Considering real-time performance, the context-aware AI solution can classify input information into two types, for example, time-sensitive input information (510) and non-time-sensitive input information (520) of FIG. 5.

[0077] In one embodiment, the time-sensitive input information (510) may include RF environment information (512) stored in real time in shared memory (422) by the modem (414), and / or time-sensitive sensing information (514) stored in real time in shared memory (422) by the sensor fusion module (418).

[0078] In one embodiment, the RF environment information (512) may include signal quality information (512a) and / or cell environment information (512b). The signal quality information (512a) may include at least one of a received signal strength indicator (RSSI), a reference signals received power (RSRP), a reference signal received quality (RSRQ), or a signal to interference and noise ratio (SINR). The cell environment information (512b) may include at least one of a cell ID (identification) (e.g., an eNB ID and / or a gNB ID), a cell radius, a cell capacity, a list of surrounding base stations, Wi-Fi interference information, or BT interference information. The Wi-Fi interference information may indicate interference by at least one Wi-Fi device (e.g., an access point (AP) or a station (STA)) detectable by the electronic device (400). The BT interference information may indicate interference by at least one BT device detectable by the electronic device (400).

[0079] In one embodiment, the time-sensitive sensing information (514) may include at least one of location information (514a), movement information (514b), or device environment information (514c). The location information (514a) may indicate the location of the electronic device (400) obtained through at least one of a GPS module, a Wi-Fi chip, a BT chip, or a UWB chip. The movement information (514b) may indicate a movement pattern (e.g., inclination, angular velocity, and / or posture) of the electronic device (400) sensed by at least one of a gyro sensor, a geomagnetic sensor, or an acceleration sensor. The device environment information (514c) may indicate at least one of a grip state, an ambient temperature, or an internal temperature (e.g., a CPU temperature and / or a battery temperature) of the electronic device (400).

[0080] Non-time sensitive input information (520) may include non-time sensitive sensing information (522) stored in the shared memory (422) by the sensor fusion module (418). Non-time sensitive sensing information (522) may include at least one of user behavior pattern information (522a), surrounding environment information (522b), biometric information (522c), or device information (522d). User behavior pattern information (522a) may indicate at least one of call patterns by time zone (e.g., time and / or day of the week), data usage patterns by time zone (e.g., time and / or day of the week), usage patterns by application (e.g., usage time, usage frequency, and / or data usage), or past movement paths. Surrounding environment information (522b) may indicate building interior structures, terrain, illumination, noise, and / or air pressure related to the surrounding environment of the electronic device (400). Biometric information (522c) may indicate the heart rate, number of steps, and / or skin temperature of a user carrying the electronic device (400). Device information (522d) may indicate the model name, chipset performance, operating system (OS) version, and / or battery status related to the electronic device (400).

[0081] In one embodiment, non-time sensitive sensing information (522) may be stored in the shared memory (422) in the form of feature vectors. The sensor fusion module (418) may acquire non-time sensitive sensing information (522) through one or more sensors, and convert the non-time sensitive sensing information (522) into sensor feature vectors through a pre-trained small-scale deep learning model (e.g., sensor fusion AI model (1006)) and store the converted sensor feature vectors in the shared memory (422).

[0082] In one embodiment, output information (530) may be generated as a result of a multi-modal deep learning model (500) based on time-sensitive input information (510) and non-time-sensitive input information (520). In one embodiment, output information (530) may include at least one of antenna power control information (532), modulation / multiplexing information (534), or DRX parameters (536).

[0083] In one embodiment, the antenna power control information (532) may include at least one of a beamforming weight vector set, a precoding weight vector set, or an envelope tracking (ET) parameter indicating beam direction and / or power. In one embodiment, the modulation / multiplexing information (534) may include a modulation index indicating a modulation scheme, such as PSK (e.g., binary phase shifting keying (BPSK), quadrature PSK (QPSK), or pi / 2-BPSK) or high-ordered QAM (e.g., 16-ary quadrature amplitude modulation (16-QAM), 64-QAM, or 256-QAM), and / or a multiplexing index indicating a multi-carrier transmission technique, such as OFDM or UFMC. The DRX parameters (536) may include at least one of a DRX cycle, a DRX inactivity timer, a DRX long cycle, or a DRX short cycle.

[0084] In one embodiment, the DRX cycle may indicate the length of a period during which the electronic device (400) monitors whether there is information transmitted from the network to the electronic device (400) on a control channel (e.g., a physical downlink control channel (PDCCH)). In one embodiment, the DRX deactivation timer may indicate a time during which the electronic device (400) deactivates a DRX operation according to a specified condition. In one embodiment, the DRX short cycle may indicate a time between periods during which the electronic device (400) monitors a control channel according to a DRX operation, and the DRX long cycle may indicate an interval between periods during which the electronic device (400) monitors a control channel according to an extended DRX (eDRX) operation.

[0085] The output information (530) of the multi-modal deep learning model (500) can be used for RF performance optimization such as antenna power control, digital modulation and coding scheme, and DRX control. The multi-modal deep learning model (500) according to the embodiment of the present disclosure can reflect RF environment changes and user context in real time and non-real time, classify sensing information that requires less real time (e.g., non-time sensitive input information (500)) to minimize sensor operation, and optimize RF control parameters (e.g., output information (530)), thereby improving RF performance of a mobile communication system and maximizing power efficiency.

[0086] Figure 6 is a flowchart illustrating the execution procedure of a multi-modal deep learning algorithm according to an embodiment of the present disclosure. Depending on the embodiments, at least one of the operations described below may be omitted, modified, or executed in a different order. In one embodiment, at least one of the operations described below may be executed by an electronic device (400) (e.g., an NPU (412)).

[0087] Referring to FIG. 6, in operation 602, the electronic device (400) (e.g., NPU (412)) may identify that a channel estimation operation is triggered according to a specified trigger condition. In one embodiment, the specified trigger condition may include a change in time-sensitive sensing information (e.g., time-sensitive sensing information (514) of FIG. 5) exceeding a preset threshold, a change in non-time-sensitive sensing information (e.g., non-time-sensitive sensing information (522) of FIG. 5) exceeding a preset threshold, and / or reaching a specified period.

[0088] In operation 604, the electronic device (400) (e.g., NPU (412)) may obtain time-sensitive input information (e.g., time-sensitive input information (510) of FIG. 5) from the shared memory (422). In operation 604, the electronic device (400) (e.g., NPU (412)) may perform preprocessing (e.g., embedding and normalization) on the time-sensitive input information to generate normalized feature vectors.

[0089] At operation 608, the electronic device (400) (e.g., NPU (412)) may obtain non-time sensitive input information (e.g., non-time sensitive input information (520) of FIG. 5) from the shared memory (422). At operation 610, the electronic device (400) (e.g., NPU (412)) may perform preprocessing (e.g., embedding, concatenation, and / or dimension reduction) on the non-time sensitive input information to extract feature vectors (e.g., sensor feature vectors) corresponding to the non-time sensitive input information.

[0090] In operation 610, the electronic device (400) (e.g., NPU (412)) may generate output information (e.g., output information (530) of FIG. 5) by executing a multi-modal deep learning-based context-aware AI solution (e.g., at least one of a transformer model, multi-attention, feature extraction, loss function calculation, or clustering) on ​​preprocessed information (e.g., normalized feature vectors) corresponding to the time-sensitive input information and preprocessed information (e.g., sensor feature vectors) corresponding to the non-time-sensitive input information.

[0091] In operation 614, the electronic device (400) (e.g., NPU (412)) may control the antenna (404a) and / or RFFE (404) by transmitting the output information (e.g., antenna power control information (532) of FIG. 5) to the antenna (404a) and / or RFFE (404) through the shared memory (422).

[0092] At operation 616, the electronic device (400) (e.g., NPU (412)) may control the DRX operation of the modem (414) by transmitting the output information (e.g., DRX parameters (536) of FIG. 5) to the modem (414) through the shared memory (422). At operation 618, the electronic device (400) (e.g., NPU (412)) may control the modulation and / or multiplexing operation of the modem (414) by transmitting the output information (e.g., modulation / multiplexing information (534) of FIG. 5) to the modem (414) through the shared memory (422).

[0093] FIG. 7 is a diagram illustrating a transmission procedure using a multi-modal deep learning algorithm according to one embodiment of the present disclosure. In one embodiment, the electronic device (400) may perform the following operations from inputting user voice data to outputting / receiving antennas.

[0094] Referring to FIG. 7, a microphone (702) (e.g., an input module (150)) may convert a user's voice signal into an electrical analog voice signal. A coder and decoder (CODEC) (704) may include an amplifier (AMP) (704a) and an analog-to-digital converter (ADC) (704b), and may amplify the analog voice signal and then convert it into a digital signal. A DSP (416) may include a source encoder (416a) and a channel encoder (416b). The source encoder (416a) may compress voice and / or image data included in the digital signal through source coding to reduce the transmission amount. The channel encoder (416b) may add an error detection and correction code to the encoded data to increase data reliability. The DSP (416) may apply an encryption algorithm to the channel-encoded data to ensure data security.

[0095] The modem (414) can perform processing of the PDCP (packet data convergence protocol) layer, the RLC (radio link control) layer, the MAC (medium access control) layer, and scrambling. The modem (414) can obtain a modulation index (712) generated by a multi-modal deep learning model by the NPU (412) from the shared memory (422), and perform modulation (e.g., QAM) based on the modulation index (712). The modem (414) can obtain a precoding weight vector (714) generated by the multi-modal deep learning model by the NPU (412) from the shared memory (422), and perform precoding (e.g., application of a precoding matrix) based on the precoding weight vector (714).

[0096] The modem (414) can obtain a multiplexing index (716) generated through a multi-modal deep learning model by the NPU (412) from the shared memory (422) and perform multiplexing (e.g., OFDM processing including inverse fast Fourier transform (IFFT) and cyclic prefix (CP) insertion) based on the multiplexing index (716).

[0097] The PDCP layer can perform IP (Internet Protocol) packet header compression and encryption. The RLC layer can perform data order control, error detection, and retransmission. The MAC layer can perform data transmission scheduling and resource allocation. Scrambling can randomize data patterns to reduce interference.

[0098] The modem (414) can map a digital signal to modulation symbols (e.g., QAM modulation symbols) through modulation. The modem (414) can process high-dimensional QAM based on the execution result of a multi-modal deep learning model by the NPU (412) (e.g., modulation index (712)).

[0099] The modem (414) can apply a precoding matrix based on a MIMO channel matrix to modulation symbols. The modem (414) can process precoding based on the execution result of a multi-modal deep learning model by the NPU (412) (e.g., a precoding weight vector (714)).

[0100] The modem (414) can multiplex and transmit modulated data from one or more users onto a single channel. The modem (414) can perform multiplexing based on OFDM or UFMC based on the execution result of a multi-modal deep learning model by the NPU (412) (e.g., multiplexing index (716)).

[0101] In one embodiment, the modem (414) may be configured to communicate directly with the NPU (412) and the sensor fusion module (418) via shared memory (422) for control of modulation, precoding, and multiplexing.

[0102] The transceiver (406) includes a mixer (406a) and an RF DAC (digital to analog converter) (406b), and can perform frequency up conversion of an analog signal converted by the DAC (406b) to a carrier frequency through the mixer (406a).

[0103] The RFFE (404) may be configured to amplify, filter, and frequency convert an RF signal between the antenna (404a) and the transceiver (406). The RFFE (404) may obtain envelope tracking parameters (718) generated through a multi-modal deep learning model by the NPU (412) from the shared memory (422) and control a power amplifier (PA) based on the envelope tracking parameters (718).

[0104] The antenna (404a) may include an array antenna for implementing a beamforming technology that focuses a signal in a specific direction. The RF circuit (404) may perform transmission beamforming by obtaining a beamforming weight vector (720) generated through a multi-modal deep learning model by the NPU (412) from the shared memory (422) and applying a phase shift according to the beamforming weight vector (720) to the antenna elements of the array antenna.

[0105] In embodiments of the present disclosure, the NPU (412) may execute a multi-modal deep learning model to analyze time-sensitive input information (510) and non-time-sensitive input information (520), and determine RF control parameters (e.g., output information (530)). In one embodiment, the shared memory (422) may store various information, such as RF data, control information, sensor fusion feature vectors, and deep learning model outputs, for data exchange and data sharing between the modem (414) and the NPU (412). The sensor fusion module (418) may merge various sensor data to generate contextual information (e.g., RF context and / or user context), and provide the information as input to the multi-modal deep learning model through the shared memory (422).

[0106] FIG. 8 is a diagram illustrating a reception procedure using a multi-modal deep learning algorithm according to one embodiment of the present disclosure. In one embodiment, the electronic device (400) may perform the following operations until transmitting an RF signal received through a reception antenna to a user.

[0107] Referring to FIG. 8, the antenna (404a) can receive an RF signal spread across space. In order to receive the RF signal through the antenna (404a), the RF circuit (404) can obtain a beamforming weight vector (812) generated by a multi-modal deep learning model by the NPU (412) from the shared memory (422), and control the reception beam direction and beam width according to the beamforming weight vector (812).

[0108] RFFE (404) can switch an RF signal received by an antenna (404a) to an LNA (low noise amplifier) ​​by an RF switch (RF switch: RF_SW), and output a power-controlled RF signal through amplification by the LNA and filtering by a band-pass filter.

[0109] The transceiver (406) can convert the RF signal into a baseband signal through a mixer (406c). The mixer (406c) can frequency down-convert the RF signal using a local oscillator (LO) signal. The frequency down-converted signal can be converted into a digital signal by an analog to digital converter (ADC) (406d) and then transmitted to a modem (414).

[0110] The modem (414) is responsible for digital signal processing and can perform DRX, de-multiplexing, demodulation, descrambling, MAC layer processing, RLC layer processing, and PDCP layer processing.

[0111] The modem (414) can separate modulated data of multiple users included in the baseband signal through demultiplexing. The modem (414) can obtain a multiplexing index (816) generated through a multi-modal deep learning model by the NPU (412) from the shared memory (422) and perform demultiplexing according to the multiplexing index (816).

[0112] The modem (414) can extract information data from the demultiplexed digital signal through demodulation. The modem (414) can obtain a modulation index (820) generated through a multi-modal deep learning model by the NPU (412) from the shared memory (422) and perform demodulation according to the modulation index (820).

[0113] The modem (414) can restore the scrambled data included in the above information data to its original form through descrambling.

[0114] The modem (414) can perform data transmission scheduling and resource allocation through MAC layer processing. The modem (414) can perform data order control, error detection, and retransmission through RLC layer processing. The modem (414) can decompress and decrypt IP packet headers through PDCP layer processing.

[0115] The modem (414) may perform a DRX operation to detect a received signal prior to multiplexing. The modem (414) may obtain DRX parameters (814) generated through a multi-modal deep learning model by the NPU (412) from the shared memory (422), and monitor a received channel (e.g., PDCCH) according to the DRX parameters (814) (e.g., DRX cycle, DRX deactivation timer, and / or DRx long / short cycle).

[0116] The modem (414) can perform inverse precoding on the modulated data after multiplexing. The modem (414) can obtain a precoding weight vector (818) generated through a multi-modal deep learning model by the NPU (412) from the shared memory (422) and perform inverse precoding (e.g., application of a precoding matrix) based on the precoding weight vector (818).

[0117] The DSP (416) can convert the digital signal transmitted from the modem (414) into an analog signal by performing channel decoding and source decoding. Channel decoding can correct errors in channel-coded data included in the digital signal. The channel decoder (416c) can perform channel decoding according to channel coding information provided from the NPU (412) through the shared memory (422). The source decoder (416d) can restore compressed voice and / or image data included in the decoded bits and transmit them to the codec (804).

[0118] The codec (804) can convert a digital signal including voice and / or image data transmitted from the DSP (416) into an analog voice signal through a digital to analog converter (DAC) (804b), and then amplify the signal with an AMP (804a) and transmit it to a speaker (802).

[0119] The NPU (412) receives time-sensitive sensing information (e.g., time-sensitive sensing information (514) of FIG. 5) and non-time-sensitive input information (e.g., non-time-sensitive sensing information (522) of FIG. 5) provided from the sensor fusion module (418), and RF environment information (e.g., RF environment information (512) of FIG. 5) provided from the modem (414) through the shared memory (422), executes a multi-modal deep learning model, and generates RF control parameters (e.g., output information (530) of FIG. 5) such as at least one of a beamforming weight vector (812), DRX parameters (814), a multiplexing index (816), a precoding weight vector (818), or a modulation index (820) as an execution result of the multi-modal deep learning model, and stores the RF control parameters in the shared memory (422).

[0120] The shared memory (422) receives and stores input information (e.g., time-sensitive input information (510) and non-time-sensitive input information (520) of FIG. 5) from the sensor fusion module (418) and the modem (414), stores RF control parameters (e.g., output information (530) of FIG. 5) generated through execution of a multi-modal deep learning model in the NPU (412), and can transmit the RF control parameters to the modem (414).

[0121] The shared memory (422) can be responsible for data exchange between the NPU (412), the modem (414), and the sensor fusion module (418). The NPU (412), the modem (414), and the sensor fusion module (418) can cooperatively operate by exchanging necessary data (e.g., time-sensitive input information (510), non-time-sensitive input information (520), and output information (530)) through the shared memory (422). Input information generated at different rates in the modem (414) or the sensor fusion module (418) can be stably stored in the shared memory (422) and used by the NPU (412).

[0122] NPU (412) can store RF control parameters (e.g., output information (530) of FIG. 5) as a result of executing a multi-modal deep learning model in shared memory (422).

[0123] The modem (414) can read the execution result of the NPU (412) from the shared memory (422) (e.g., output information (530) of FIG. 5) and adjust the reception operations (e.g., reception beamforming, DRX, demultiplexing, precoding, and / or demodulation) to improve performance.

[0124] In one embodiment, a set of beamforming weight vectors (e.g., beamforming weight vectors (720) and / or beamforming weight vectors (812)) may be configured as follows.

[0125] w = [w1, w2, ..., w M ] T

[0126] Here w irepresents a weight for the i-th antenna, and M may represent the number of antennas (e.g., transmitting antennas or receiving antennas). The electronic device (400) (e.g., NPU (412)) may determine the beamforming weight vector set by a multi-modal deep learning algorithm to increase the signal power received from the receiving antenna, improve SINR, and increase data throughput.

[0127] In one embodiment, the set of precoding weight vectors (e.g., precoding weight vectors (714) and / or precoding weight vectors (818)) may be configured as follows.

[0128] f(s, H) = W s

[0129] Here, f represents a precoding function, s represents a transmission signal vector, H represents a channel matrix, and W may represent a precoding matrix.

[0130] In one embodiment, an envelope tracking parameter (e.g., envelope tracking parameter (718)) is a transmit power level (P) for a power amplifier (PA) included in the RFFE (404). Tx ) and / or may include at least one parameter (e.g., α, β) that determines the envelope shape.

[0131] FIG. 9 illustrates a software (SW) layer structure for executing a multi-modal deep learning-based AI solution according to one embodiment of the present disclosure. In one embodiment, an electronic device (400) (e.g., a processor (410)) may include a SW layer structure described below. The processor (410) may include at least one of an application layer (900), a framework layer (910), a libraries layer (922), an RF manager layer (934), a sensor fusion manager (950), an NPU runtime execution unit (930), a DSP runtime execution unit (932), a hardware abstraction layer (HAL) layer (952), or a driver layer (954) (e.g., a kernel).

[0132] The application layer (900) may provide at least one of a communication settings user interface (UI) (902), a communication status display UI (904), a beamforming visualization (906), or a call / messaging application (908). The communication settings UI (902) may include input objects for enabling a user to change RF communication-related settings, for example, for beamforming activation and selection of a power control mode. The communication status display UI (904) may include output objects that visually display the current RF communication status (e.g., signal strength, mode, connected base station, etc.). The beamforming visualization UI (906) may include output objects that intuitively display the operating status of the beamforming technology (e.g., beam direction and / or beam width). The call / messaging application (908) may provide general communication functions, such as voice calls and / or sending and receiving messages.

[0133] The framework layer (910) may include at least one of a system control manager (912) that controls and manages the overall operation of the electronic device (400), a power manager (914) that manages power consumption, an RF framework (916) that provides RF-related functions (e.g., channel estimation, beamforming, power control, etc.), a display (DP) framework (918), or a sensor framework (920) that collects and processes data from one or more sensors (e.g., at least one of a gyro sensor, a geomagnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a biometric sensor, a temperature sensor, an altitude sensor, a light intensity sensor, a humidity sensor, an oxygen sensor, a heart rate sensor, a blood pressure sensor, an on sensor, an electrocardiogram sensor, a GPS module, a Wi-Fi chip, a Bluetooth chip, or a UWB chip).

[0134] The library layer (922) may include at least one of a DSP library (DSP_LIB) (924) related to signal processing by a DSP (416), an AI-RF library (AI-RF_LIB) (926) related to RF-related artificial intelligence algorithms (e.g., channel estimation, beamforming optimization, etc.), or a modem library (MODEM_LIB) (928) supporting mathematical operations and matrix operations by a modem (414).

[0135] The RF manager layer (934) may be responsible for comprehensively managing RF-related functions. The RF manager layer (934) may include at least one of a beamforming manager (936) that controls and manages beamforming technology, a paging / DRX manager (940) that controls paging and DRX operations, an RF communication manager (944) that processes RF communication protocols (e.g., RRC layer, MAC layer, etc.), a network status monitor (948) that monitors network status (e.g., signal strength, signal quality, etc.), an envelope tracking (ET) power manager (938), a modulation manager (942), or a channel estimation manager (946).

[0136] The sensor fusion manager (950) can fuse various sensor data to extract meaningful information (e.g., sensor feature vectors).

[0137] The NPU runtime execution unit (930) can provide a runtime environment for execution and inference of a multi-modal deep learning model.

[0138] The HAL layer (952) may provide an interface between a software structure (e.g., at least one of the application layer (900), the framework layer (910), the library layer (922), the RF manager layer (934), the sensor fusion manager (950), the NPU runtime executable (940), or the DSP runtime executable (932)) and a hardware structure (e.g., at least one of the RF circuit (402), the shared memory (422), the sensor fusion module (418), the modem (414), the DRAM (422a), or the display (956)).

[0139] The driver layer (954) may control a hardware structure (e.g., at least one of the RF circuit (402), the shared memory (422), the sensor fusion module (418), the modem (414), the DRAM (422a), or the display (956)). The processor (410) may be configured to control the shared memory (422) above the HAL layer (952).

[0140] The illustrated software structure is not limited to being included in a single processing circuit (e.g., processor (410)), and may be executed by an application processor or a communication processor. For example, the application layer (900) may be executed by the application processor, the framework layer (910) and the RF manager layer (934) may be executed by the communication processor, and the NPU runtime execution unit (930) may be executed by the NPU (412).

[0141] In one embodiment, when communication-related settings are changed through the application layer (900), the framework layer (910) can change the policy according to the changed settings. The RF manager layer (934) can transfer input information (e.g., time-sensitive input information (510) and non-time-sensitive input information (520) of FIG. 5) acquired from the RF circuit (402), the modem (414), and the sensor fusion module (418) to the NPU runtime execution unit (930) through the library layer (922) according to the changed policy. The RF manager layer (934) can control RF hardware (e.g., RF circuit (402) and / or modem (414)) using the execution result of the multi-modal deep learning model provided from the NPU runtime execution unit (930) (e.g., output information (930) of FIG. 5).

[0142] FIG. 10 is a diagram illustrating the operational procedures of a context-aware AI solution based on multi-modal deep learning according to one embodiment of the present disclosure. Depending on the embodiments, at least one of the operations described below may be omitted, modified, or executed in a different order. In one embodiment, at least one of the operations described below may be executed by an electronic device (400) (e.g., at least one of an NPU (412), a modem (414), and / or a sensor fusion module (418).

[0143] Referring to FIG. 10, in operation 1000, the electronic device (400) (e.g., the modem (414) and / or the sensor fusion module (418)) may identify that a channel estimation operation is triggered. In one embodiment, the electronic device (400) (e.g., the modem (414)) may continuously monitor RF signal quality (e.g., RSSI, SNR, and / or CSI (channel state indicator)) and cell interference information, and may determine whether the channel estimation operation is triggered based on a specified trigger condition.

[0144] In one embodiment, the trigger condition may include that the amount of change in input information (e.g., time-sensitive sensing information (514) or non-time-sensitive sensing information (522)) exceeds a specified threshold. In one embodiment, the electronic device (400) (e.g., modem (414)) may determine that the trigger condition is satisfied when the amount of change in signal quality information (512a) and / or cell environment information (512b) measured for the received signal exceeds a preset threshold. In one embodiment, the electronic device (400) (e.g., sensor fusion module (418)) may determine that the trigger condition is satisfied when the amount of change in at least one of location information (514a), movement information (514b), or device environment information (514c) collected through at least one sensor and / or at least one communication chip exceeds a preset threshold.

[0145] In one embodiment, the electronic device (400) (e.g., the modem (414) and / or the sensor fusion module (418)) may determine that the trigger condition is satisfied at specified time intervals to perform periodic channel estimation.

[0146] In one embodiment, the electronic device (400) (e.g., the modem (414)) may acquire RF environment information (512) (e.g., signal quality information (512a) and / or cell environment information (512b)) related to a current channel by performing channel estimation based on the satisfaction of the trigger condition, and may store the acquired RF environment information (512) in the shared memory (422). In one embodiment, the electronic device (400) (e.g., the sensor fusion module (418)) may collect time-sensitive sensing information (514) (e.g., at least one of location information (514a), motion information (514b), or device environment information (514c)) through at least one sensor based on the satisfaction of the trigger condition, and may store the collected time-sensitive sensing information (514) in the shared memory.

[0147] In one embodiment, the electronic device (400) (e.g., modem (414) and / or sensor fusion module (418)) may update the trigger condition after the output information (e.g., output information (530) of FIG. 5) is generated through the initial execution of the multi-modal deep learning algorithm by the NPU (412).

[0148] In operation 1002, the electronic device (400) (e.g., NPU (412)) can read out time-sensitive input information (510) (e.g., at least one of signal quality information (512a), cell environment information (512b), location information (514a), motion information (514b), or device environment information (514c)) from the shared memory (422), and perform embedding and normalization on the read time-sensitive input information (510). Feature vectors corresponding to the time-sensitive input information (510) generated through the embedding and normalization can be provided as inputs to a multi-modal deep learning model.

[0149] In operation 1004, the electronic device (400) (e.g., the sensor fusion module (418)) may perform embedding on non-time sensitive sensing information (522) (e.g., at least one of user behavior pattern information (522a), surrounding environment information (522b), biometric information (522c), or device information (522d)) input directly or through shared memory (422) from at least one sensor and / or at least one communication chip. In operation 1006, the electronic device (400) (e.g., the sensor fusion module (418)) may perform concatenation and / or dimensionality reduction on feature vectors generated through the embedding by executing a pre-learned sensor fusion AI model. In operation 1008, the electronic device (400) (e.g., sensor fusion module (418)) may store sensor feature vectors generated through execution of the sensor fusion AI model in a shared memory (422) so that they can be used as inputs for a multi-modal deep learning algorithm by the NPU (412).

[0150] In operation 1010, the electronic device (400) (e.g., NPU (412)) executes a multi-modal deep learning algorithm that takes time-sensitive input information (510) and non-time-sensitive sensing information (522) as inputs, thereby recognizing an RF context corresponding to the time-sensitive input information (510) and non-time-sensitive sensing information (522), and generating output information (e.g., output information (530) of FIG. 5) for controlling the antenna (404a), RFFE (404), and modem (414) based on the RF context.

[0151] In one embodiment, the multi-modal deep learning algorithm can extract feature vectors from time-sensitive input information (510) and non-time-sensitive sensing information (522) by executing at least one of a transformer model, a multi-head attention model, a multi-layer perceptron (MLP) model, or a CNN model. In one embodiment, the multi-modal deep learning algorithm can calculate a loss function to reduce a difference between predicted values ​​and measured values ​​based on the feature vectors, and train a model of the multi-modal deep learning algorithm using the loss function. In one embodiment, the multi-modal deep learning algorithm can recognize an RF context by clustering input information having similar features.

[0152] In one embodiment, output information (530) generated through the multi-modal deep learning algorithm (e.g., at least one of antenna power control information (532), DRX parameters (536), or modulation / multiplexing information (534)) may be transmitted to at least one of the modem (414), the antenna (404a), or the RFFE (404). The antenna power control information (532) may be transmitted to the antenna (404a) to appropriately control antenna transmission power, thereby reducing unnecessary power consumption and improving communication quality. The DRX parameters (536) may be transmitted to the modem (414) to enable the modem (414) to reduce unnecessary receiving operations of the electronic device (400), thereby reducing battery consumption. Modulation / multiplexing information (534) is transmitted to the modem (414), causing the modem (414) to operate with an appropriate modulation scheme (e.g., BPSK, QPSK, pi / 2-BPSK, 16QAM, 64-QAM, 256-QAM, or 1024-QAM) according to channel conditions, thereby increasing data throughput and reducing transmission error rate, thereby improving data reliability.

[0153] In one embodiment, the NPU (412) can store output information (530) generated through the multi-modal deep learning algorithm in the shared memory (422). The modem (414) can read the output information (530) stored in the shared memory (422) and control a transmission procedure or a reception procedure.

[0154] FIG. 11a and FIG. 11b illustrate a user interface (UI) for setting up application of a multi-modal deep learning-based AI solution according to one embodiment of the present invention.

[0155] Referring to FIG. 11A, the electronic device (400) may display a settings screen (1110) based on a user input. The settings screen (1110) may include an input object (1112) (e.g., “data network method SIM1”) for specifying a data network method corresponding to a specified subscription identification module (SIM) (e.g., SIM1).

[0156] Referring to FIG. 11B, the electronic device (400) may display an input object (1114) for selecting a mode (e.g., “6G priority mode (AI)”) that enables application of a multi-modal deep learning-based AI solution based on a user input (e.g., a touch) to the input object (1112). Based on the user input (e.g., a touch) to the input object (1114), the electronic device (400) may trigger execution of a multi-modal deep learning-based RF context-aware AI algorithm that utilizes input information (e.g., time-sensitive input information (510) and non-time-sensitive input information (520)) collected by the modem (414) and the sensor fusion module (418). The electronic device (400) may improve communication quality by customizing RF and modem operations, such as beamforming, power control, and modulation scheme, through the AI ​​algorithm. The above multi-modal deep learning algorithm can be implemented as an on-device AI solution to maintain data security.

[0157] In one embodiment, the electronic device (400) may operate in a mode (e.g., “6G priority mode (basic)”) that restricts the execution of functions of the AI ​​solution after securing stability of communication quality through application of the multi-modal deep learning algorithm. By executing the restricted AI solution, the electronic device (400) may provide basic 6G network settings, minimize AI functions, reduce resource usage, and reduce battery consumption.

[0158] The multi-modal deep learning model according to embodiments of the present disclosure can improve communication efficiency by improving RF performance and modem performance based on various input information (e.g., time-sensitive input information (510) and non-time-sensitive input information (520)).

[0159] An electronic device (400) according to one embodiment of the present disclosure may include a neural processing unit (NPU) (412), a radio frequency (RF) circuit (402) connected to at least one antenna (404a), a modem (414) configured to perform cellular communication, and a shared memory (422) configured to be accessible by the NPU, the modem, and a sensor fusion module (418) associated with at least one sensor, and storing time-sensitive input information (510) and non-time-sensitive input information (520) obtained from the modem and the sensor fusion module. The NPU may be configured to obtain the time-sensitive input information and the non-time-sensitive input information from the shared memory. The NPU may be configured to generate output information for use in operating the RF circuit and the modem by inputting the time-sensitive input information and the non-time-sensitive input information into a multi-modal deep learning algorithm. The above NPU is configured to store the output information generated by the multi-modal deep learning algorithm that takes the time-sensitive input information and the non-time-sensitive input information as inputs in the shared memory, so that the RF circuit and the modem can operate by the output information.

[0160] In one embodiment, the time-sensitive input information may include at least one of signal quality information including at least one of a received signal strength indicator (RSSI), a reference signals received power (RSRP), a reference signal received quality (RSRQ), or a signal to interference and noise ratio (SINR); cell environment information indicating at least one of a cell ID, a cell radius, a cell capacity, a list of surrounding base stations, Wi-Fi interference, and BT (Bluetooth) interference; location information acquired through at least one of a global positioning system (GPS) module, a Wi-Fi chip, or a BT chip; movement information determined based on at least one of a gyro sensor, a geomagnetic sensor, or an acceleration sensor; or device environment information including at least one of a grip type, an ambient temperature, or an internal temperature.

[0161] In one embodiment, at least one of the signal quality information or the cell environment information may be measured by the modem and stored in real time in the shared memory.

[0162] In one embodiment, at least one of the location information, the movement information, or the device environment information may be collected by the sensor fusion module and stored in real time in the shared memory.

[0163] In one embodiment, the non-time sensitive input information may include at least one of user activity pattern information indicating at least one of a time-based call pattern and / or data usage pattern, a usage pattern of at least one application, a movement path of the electronic device, biometric information including at least one of a heart rate and a skin temperature, surrounding environment information indicating at least one of a building, terrain, and weather, or device information indicating at least one of a model name, chipset performance, an OS version, and a battery status.

[0164] In one embodiment, at least some of the non-time sensitive input information may be acquired by the sensor fusion module and converted into feature vector data through a pre-learned sensor fusion AI model, and the converted feature vector data may be stored in the shared memory.

[0165] In one embodiment, the output information may include antenna power control information including at least one of a beamforming weight vector set related to beam direction and beam power, a precoding weight vector set, and an envelope tracking parameter, a modulation index indicating a modulation scheme and / or a multiplexing index indicating orthogonal frequency division multiplexing (OFDM) or universal filtered multi carrier (UFMC), or at least one of DRX parameters indicating at least one of a DRX (discontinuous reception) cycle, a DRX inactivity timer, and a DRX long / short cycle.

[0166] In one embodiment, the beamforming weight vector set may be transmitted to the RF circuit via the shared memory. The envelope tracking parameters may be transmitted to the RF circuit via the shared memory.

[0167] In one embodiment, at least one of the precoding weight vector set, the modulation index, the multiplexing index, or the DRX parameters may be transmitted to the modem via the shared memory.

[0168] In one embodiment, the sensor fusion module may be configured to fuse sensor data collected from the at least one sensor and the at least one communication chip to generate RF context information and provide the RF context information as an input to the multi-modal deep learning algorithm via the shared memory.

[0169] According to one embodiment of the present disclosure, a non-transitory computer-readable storage medium storing one or more programs, wherein the one or more programs, when executed by a neural processing unit (NPU) (412) of an electronic device (400), cause the electronic device to: obtain, from the shared memory (422), the time-sensitive input information (510) and the non-time-sensitive input information (520) obtained from a modem (414) for cellular communication and a sensor fusion module (418) associated with at least one sensor and stored in a shared memory (422), and input the time-sensitive input information and the non-time-sensitive input information into a multi-modal deep learning algorithm to generate output information to be used for operating an RF circuit (402) connected to at least one antenna (404a) and the modem, and store the output information generated by the multi-modal deep learning algorithm that inputs the time-sensitive input information and the non-time-sensitive input information in the shared memory, so that the RF circuit and the modem operate the It may contain instructions that act on the output information.

[0170] In one embodiment, the time-sensitive input information may include at least one of signal quality information including at least one of a received signal strength indicator (RSSI), a reference signals received power (RSRP), a reference signal received quality (RSRQ), or a signal to interference and noise ratio (SINR); cell environment information indicating at least one of a cell ID, a cell radius, a cell capacity, a list of surrounding base stations, Wi-Fi interference, and BT (Bluetooth) interference; location information acquired through at least one of a global positioning system (GPS) module, a Wi-Fi chip, or a BT chip; movement information determined based on at least one of a gyro sensor, a geomagnetic sensor, or an acceleration sensor; or device environment information including at least one of a grip type, an ambient temperature, or an internal temperature.

[0171] In one embodiment, at least one of the signal quality information or the cell environment information may be measured by the modem and stored in real time in the shared memory.

[0172] In one embodiment, at least one of the location information, the movement information, or the device environment information may be collected by the sensor fusion module and stored in real time in the shared memory.

[0173] In one embodiment, the non-time sensitive input information may include at least one of user activity pattern information indicating at least one of a time-based call pattern and / or data usage pattern, a usage pattern of at least one application, a movement path of the electronic device, biometric information including at least one of a heart rate and a skin temperature, surrounding environment information indicating at least one of a building, terrain, and weather, or device information indicating at least one of a model name, chipset performance, an OS version, and a battery status.

[0174] In one embodiment, at least some of the non-time sensitive input information may be acquired by the sensor fusion module and converted into feature vector data through a pre-learned sensor fusion AI model, and the converted feature vector data may be stored in the shared memory.

[0175] In one embodiment, the output information may include antenna power control information including at least one of a beamforming weight vector set related to beam direction and beam power, a precoding weight vector set, and an envelope tracking parameter, a modulation index indicating a modulation scheme and / or a multiplexing index indicating orthogonal frequency division multiplexing (OFDM) or universal filtered multi carrier (UFMC), or at least one of DRX parameters indicating at least one of a DRX (discontinuous reception) cycle, a DRX inactivity timer, and a DRX long / short cycle.

[0176] In one embodiment, the beamforming weight vector set may be transmitted to the RF circuit via the shared memory. The envelope tracking parameters may be transmitted to the RF circuit via the shared memory.

[0177] In one embodiment, at least one of the precoding weight vector set, the modulation index, the multiplexing index, or the DRX parameters may be transmitted to the modem via the shared memory.

[0178] In one embodiment, the sensor fusion module may be configured to fuse sensor data collected from the at least one sensor and the at least one communication chip to generate RF context information and store the RF context information in the shared memory to be used as input to the multi-modal deep learning algorithm.

[0179] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0180] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0181] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0182] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0183] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0184] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (400), neural processing unit (NPU) (412); A radio frequency (RF) circuit (402) connected to at least one antenna (404a); a modem (414) configured to perform cellular communication; and A shared memory (422) configured to be accessible by the NPU, the modem, and the sensor fusion module (418) associated with at least one sensor, and storing time-sensitive input information (510) and non-time-sensitive input information (520) obtained from the modem and the sensor fusion module, The above NPU, Obtaining the time-sensitive input information and the non-time-sensitive input information from the shared memory, By inputting the time-sensitive input information and the non-time-sensitive input information into a multi-modal deep learning algorithm, output information for use in operating the RF circuit and the modem is generated, An electronic device configured to store the output information generated by the multi-modal deep learning algorithm using the time-sensitive input information and the non-time-sensitive input information as inputs in the shared memory, so that the RF circuit and the modem operate by the output information.

2. In the first paragraph, the time-sensitive input information is: Signal quality information including at least one of RSSI (received signal strength indicator), RSRP (reference signals received power), RSRQ (reference signal received quality), or SINR (signal to interference and noise ratio); Cell environment information indicating at least one of cell ID, cell radius, cell capacity, list of surrounding base stations, Wi-Fi interference, and BT (bluetooth) interference; Location information obtained through at least one of a GPS (global positioning system) module, a Wi-Fi chip, or a BT chip; Movement information determined based on at least one of a gyro sensor, a magnetometer sensor, or an acceleration sensor; or An electronic device comprising at least one of device environmental information including at least one of a grip type, an ambient temperature, or an internal temperature.

3. An electronic device according to claim 2, wherein at least one of the signal quality information or the cell environment information is measured by the modem and stored in real time in the shared memory.

4. An electronic device according to claim 2, wherein at least one of the location information, the movement information, or the device environment information is collected by the sensor fusion module and stored in real time in the shared memory.

5. In any one of paragraphs 1 to 4, the non-time sensitive input information is: User activity pattern information indicating at least one of hourly call patterns and / or data usage patterns, usage patterns of at least one application, and movement paths of the electronic device; Biometric information including at least one of heart rate and skin temperature; Information about the surrounding environment representing at least one of buildings, terrain, or weather; or An electronic device comprising at least one of device information indicating at least one of model name, chipset performance, OS version, and battery status.

6. An electronic device according to claim 5, wherein at least some of the non-time sensitive input information is acquired by the sensor fusion module and converted into feature vector data through a pre-learned sensor fusion AI model, and the converted feature vector data is stored in the shared memory.

7. In any one of paragraphs 1 to 6, the output information is: Antenna power control information including at least one of a beamforming weight vector set, a precoding weight vector set, and an envelope tracking parameter related to beam direction and beam power; A modulation index indicating a modulation method and / or a multiplexing index indicating orthogonal frequency division multiplexing (OFDM) or universal filtered multi carrier (UFMC); or An electronic device comprising at least one of DRX parameters indicating at least one of a DRX (discontinuous reception) cycle, a DRX inactivity timer, and a DRX long / short cycle.

8. In the 7th paragraph, the beamforming weight vector set is transmitted to the RF circuit through the shared memory, and / or An electronic device wherein the envelope tracking parameters are transmitted to the RF circuit via the shared memory.

9. An electronic device according to claim 7 or 8, wherein at least one of the precoding weight vector set, the modulation index, the multiplexing index, or the DRX parameters is transmitted to the modem via the shared memory.

10. In any one of paragraphs 1 to 9, An electronic device wherein the sensor fusion module is configured to fuse sensor data collected from the at least one sensor and the at least one communication chip to generate RF context information and provide the RF context information as an input to the multi-modal deep learning algorithm through the shared memory.

11. In a non-transitory computer-readable storage medium storing one or more programs, the one or more programs, when executed by a neural processing unit (NPU) (412) of an electronic device (400), cause the electronic device to: Obtaining time-sensitive input information (510) and non-time-sensitive input information (520) from a modem (414) for cellular communication and a sensor fusion module (418) associated with at least one sensor and stored in a shared memory (422), By inputting the above time-sensitive input information and the above non-time-sensitive input information into a multi-modal deep learning algorithm, output information for use in operating the RF circuit (402) and the modem connected to at least one antenna (404a) is generated, A storage medium including instructions for storing the output information generated by the multi-modal deep learning algorithm using the time-sensitive input information and the non-time-sensitive input information as inputs in the shared memory, thereby causing the RF circuit and the modem to operate by the output information.

12. In paragraph 11, the time-sensitive input information is: Signal quality information including at least one of RSSI (received signal strength indicator), RSRP (reference signals received power), RSRQ (reference signal received quality), or SINR (signal to interference and noise ratio); Cell environment information indicating at least one of cell ID, cell radius, cell capacity, list of surrounding base stations, Wi-Fi interference, and BT (bluetooth) interference; Location information obtained through at least one of a GPS (global positioning system) module, a Wi-Fi chip, or a BT chip; Movement information determined based on at least one of a gyro sensor, a magnetometer sensor, or an acceleration sensor; or A storage medium comprising at least one of device environmental information including at least one of grip type, ambient temperature, or internal temperature.

13. In the 12th paragraph, at least one of the signal quality information or the cell environment information is measured by the modem and stored in real time in the shared memory, A storage medium in which at least one of the location information, the movement information, or the device environment information is collected by the sensor fusion module and stored in real time in the shared memory.

14. In any one of paragraphs 11 to 13, the non-time sensitive input information is: User activity pattern information indicating at least one of hourly call patterns and / or data usage patterns, usage patterns of at least one application, and movement paths of the electronic device; Biometric information including at least one of heart rate and skin temperature; Information about the surrounding environment representing at least one of buildings, terrain, or weather; or Contains at least one of the device information indicating at least one of the model name, chipset performance, OS version, and battery status; A storage medium wherein at least some of the non-time sensitive input information is acquired by the sensor fusion module and converted into feature vector data through a pre-learned sensor fusion AI model, and the converted feature vector data is stored in the shared memory.

15. In any one of paragraphs 11 to 14, the output information is: Antenna power control information including at least one of a beamforming weight vector set, a precoding weight vector set, and an envelope tracking parameter related to beam direction and beam power; A modulation index indicating a modulation method and / or a multiplexing index indicating orthogonal frequency division multiplexing (OFDM) or universal filtered multi carrier (UFMC); or Contains at least one of DRX parameters indicating at least one of a DRX (discontinuous reception) cycle, a DRX inactivity timer, and a DRX long / short cycle, The above beamforming weight vector set is transmitted to the RF circuit via the shared memory, and / or The above envelope tracking parameters are transmitted to the RF circuit through the shared memory, A storage medium, wherein at least one of the precoding weight vector set, the modulation index, the multiplexing index, or the DRX parameters is transmitted to the modem via the shared memory.

Images