Communication function verification

Abstract

The embodiment of the invention relates to a device, method and equipment for communication function verification and a computer readable storage medium. In one method, a first device receives a first performance metric for a first result of a communication function from a second device based on a first set of communication parameters. The first device receives, from the second device, a reference performance metric for a reference result of the communication function based on the set of reference communication parameters. The first device verifies the communication function based on the threshold and a first difference between the first performance metric and a reference performance metric.

Description

Technical Field

[0001] Various exemplary embodiments of the present disclosure relate generally to the field of telecommunications, and more particularly to an apparatus, method, device, and computer-readable storage medium for communication function verification. Background Art

[0002] Several technologies have been proposed to improve communication performance. For example, communication devices can adopt artificial intelligence (AI) / ML models to improve communication quality. AI / ML models can be applied to different communication functions in different scenarios. Therefore, the performance of AI / ML-based communication functions is an important issue. Public Content

[0003] In a first aspect of the present disclosure, a first device is provided. The first device includes: at least one processor; and at least one memory storing instructions, which, when executed by the at least one processor, enable the first device to: receive a first performance metric of a first result of a communication function from a second device based on a first communication parameter set; receive a reference performance metric of a reference result of the communication function from the second device based on a reference communication parameter set; and verify the communication function based on a threshold and a first difference between the first performance metric and the reference performance metric.

[0004] In a second aspect of the present disclosure, a second device is provided. The second device includes: at least one processor; and at least one memory storing instructions, wherein when the instructions are executed by the at least one processor, the second device: based on a first communication parameter set, sends a first performance metric of a first result of a communication function to a first device; and based on a reference communication parameter set, sends a reference performance metric of a reference result of the communication function to the first device.

[0005] In a third aspect of the present disclosure, a method is provided. The method includes: receiving, at a first device, a first performance metric of a first result of a communication function from a second device based on a first communication parameter set; receiving, from the second device, a reference performance metric of a reference result of the communication function based on a reference communication parameter set; and verifying the communication function based on a threshold and a first difference between the first performance metric and the reference performance metric.

[0006] In a fourth aspect of the present disclosure, a method is provided. The method includes: at a second device, based on a first communication parameter set, sending a first performance metric of a first result of a communication function to a first device; and based on a reference communication parameter set, sending a reference performance metric of a reference result of the communication function to the first device.

[0007] In a fifth aspect of the present disclosure, a first device is provided. The first device includes: a component for receiving a first performance metric of a first result of a communication function from a second device based on a first communication parameter set; a component for receiving a reference performance metric of a reference result of the communication function from the second device based on a reference communication parameter set; and a component for verifying the communication function based on a threshold and a first difference between the first performance metric and the reference performance metric.

[0008] In a sixth aspect of the present disclosure, a second device is provided. The second device includes: a component for sending a first performance metric of a first result of a communication function to the first device based on a first communication parameter set; and a component for sending a reference performance metric of a reference result of the communication function to the first device based on a reference communication parameter set.

[0009] In a seventh aspect of the present disclosure, a computer-readable medium is provided, wherein the computer-readable medium includes instructions stored thereon, and the instructions are used to cause a device to at least execute the method according to the third aspect or the fourth aspect.

[0010] It should be understood that this disclosure is not intended to identify the key features or essential features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Some example embodiments will now be described with reference to the accompanying drawings, in which:

[0012] Figure 1 An example communication environment is shown in which example embodiments of the present disclosure may be implemented;

[0013] Figure 2 An example block diagram showing communication function verification according to some example embodiments of the present disclosure is shown;

[0014] Figure 3 shows an example signaling diagram for communication function verification according to some example embodiments of the present disclosure;

[0015] Figure 4 An example process for communication function verification according to some example embodiments of the present disclosure is shown;

[0016] Figure 5 A flowchart showing a method implemented at a first device according to some example embodiments of the present disclosure is shown;

[0017] Figure 6 A flowchart showing a method implemented at a second device according to some example embodiments of the present disclosure is shown;

[0018] Figure 7shows a simplified block diagram of a device suitable for implementing an example embodiment of the present disclosure; and

[0019] Figure 8 A block diagram of an example computer-readable medium is shown according to some example embodiments of the present disclosure.

[0020] Throughout the drawings, the same or similar reference numerals refer to the same or similar elements. DETAILED DESCRIPTION

[0021] The principles of the present disclosure will now be described with reference to some example embodiments. It should be understood that these embodiments are described only for the purpose of illustration, and help those skilled in the art to understand and implement the present disclosure, without implying any limitation on the scope of the present disclosure. In addition to the methods described below, the embodiments described herein can be implemented in various ways.

[0022] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0023] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it is considered to be within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0024] It should be understood that, although the terms "first", "second", etc. in front of nouns etc. may be used in this article to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element, and they do not limit the order of the nouns. For example, without departing from the scope of the example embodiments, the first element may be referred to as the second element, and similarly, the second element may be referred to as the first element. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0025] As used herein, “at least one of: ” and “at least one of ” and similar expressions (wherein a list of two or more elements is connected by “and” or “or”) refer to at least any one of the elements, or at least any two or more of the elements, or at least all of the elements.

[0026] As used herein, unless explicitly stated, performing a step “in response to A” does not mean performing the step immediately after “A” occurs, but may include one or more intermediate steps.

[0027] The terms used herein are used only for the purpose of describing specific embodiments and are not intended to limit the example embodiments. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that when used herein, the terms "comprises", "comprising", "has", "having", "includes", and / or "including" specify the presence of features, elements, and / or parts, etc., but do not exclude the presence or addition of one or more other features, elements, parts, and / or combinations thereof.

[0028] As used in this application, the term "circuitry" may refer to one or more or all of the following: (a) hardware circuit implementation only (e.g., implementation only in analog and / or digital circuits), as well as (b) a combination of hardware circuitry and software such as (where applicable): (i) a combination of analog and / or digital hardware circuitry and software / firmware, and (ii) any portion of a hardware processor (including a digital signal processor) with software, software and memory that work together to enable a device such as a mobile phone or server to perform various functions; and (c) A hardware circuit and / or processor that requires software (eg, firmware) for operation, such as a microprocessor or portion of a microprocessor, but the software may not be present when not required for operation.

[0029] This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of only a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to a particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0030] As used herein, the term "communication network" refers to a network that complies with any suitable communication standard, for example, New Radio (NR), Long Term Evolution (LTE), Advanced LTE (LTE-A), Wideband Code Division Multiple Access (WCDMA), High Speed ​​Packet Access (HSPA), Narrowband Internet of Things (NB-IoT), etc. In addition, the communication between the terminal device and the network device in the communication network can be performed according to any suitable generation communication protocol, including but not limited to the first generation (1G), second generation (2G), 2.5G, 2.75G, third generation (3G), fourth generation (4G), 4.5G, fifth generation (5G), sixth generation (6G) communication protocol and / or any other protocol currently known or developed in the future. The embodiments of the present disclosure can be applied to various communication systems. In view of the rapid development of communication, there are of course future types of communication technologies and systems that can implement the present disclosure. It should not be considered that the scope of the present disclosure is limited to the above-mentioned system.

[0031] As used herein, the term "network device" refers to a node in a communication network via which a terminal device accesses the network and receives services from it. A network device may refer to a base station (BS) or an access point (AP), such as a Node B (NodeB or NB), an evolved Node B (eNodeB or eNB), a NR NB (also referred to as a gNB), a remote radio unit (RRU), a radio head (RH), a remote radio head (RRH), a repeater, an integrated access and backhaul (IAB) node, a low-power node such as a femto, a micro), a non-terrestrial network (NTN) or a non-terrestrial network device (such as a satellite network device, a low earth orbit (LEO) satellite and a geosynchronous orbit (GEO) satellite, an aircraft network device, etc.), depending on the terminology and technology applied. In some example embodiments, a radio access network (RAN) split architecture includes a centralized unit (CU) and a distributed unit (DU) at an IAB donor node. An IAB node includes a mobile terminal (IAB-MT) portion that behaves like a UE toward a parent node, and a DU portion of an IAB node behaves like a base station toward a next-hop IAB node.

[0032] The term "terminal device" refers to any terminal device capable of wireless communication. As an example and not limitation, the terminal device may also be referred to as a communication device, a user equipment (UE), a subscriber station (SS), a portable subscriber station, a mobile station (MS) or an access terminal (AT). The terminal device may include, but is not limited to, a mobile phone, a cellular phone, a smart phone, a voice over IP (VoIP) phone, a wireless local loop phone, a tablet computer, a wearable terminal device, a personal digital assistant (PDA), a portable computer, a desktop computer, an image capture terminal device such as a digital camera, a game terminal device, a music storage and playback facility, a vehicle-mounted wireless terminal device, a wireless endpoint, a mobile station, a laptop embedded device (LEE), a laptop mounted device (LME), a USB dongle, a smart device, a wireless user equipment (CPE), an Internet of Things (IoT) device, a watch or other wearable device, a head mounted display (HMD), a vehicle, a drone, medical equipment and applications (e.g., remote surgery), industrial equipment and applications (e.g., robots and / or other wireless devices operating in industrial and / or automatic processing chain environments), consumer electronic devices, equipment operating on commercial and / or industrial wireless networks, etc. The terminal device may also correspond to the mobile terminal (MT) part of an IAB node (eg, a relay node).In the following description, the terms "terminal device", "communication device", "terminal", "user equipment" and "UE" may be used interchangeably.

[0033] As used herein, the terms "resource", "transmission resource", "resource block", "physical resource block" (PRB), "uplink resource" or "downlink resource" may refer to any resource used to perform communication (e.g., communication between a terminal device and a network device), such as a resource in the time domain, a resource in the frequency domain, a resource in the spatial domain, a resource in the code domain, or any other combination of time domain, frequency domain, spatial domain and / or code domain resources that enable communication. In the following, unless explicitly stated, resources in the frequency domain and the time domain will be used as examples of transmission resources for describing some example embodiments of the present disclosure. It should be noted that the example embodiments of the present disclosure are equally applicable to other resources in other domains.

[0034] As used herein, the term "AI / ML model" may refer to a data-driven algorithm that applies AI / ML techniques to generate a set of outputs based on a set of inputs. In the context of the present disclosure, the term "AI / ML model" may be used interchangeably with the terms "model," "AI model," and "ML model." The term "AI / ML" may be used interchangeably with the terms "AI" and "ML."

[0035] As used herein, the term "generalization" refers to the ability of a model to adapt appropriately to new, previously unseen data. New or previously unseen data may be drawn from a distribution different from the distribution used to create or train the model. In other words, generalization examines how well the model, after being trained on the training set, can digest new data (mostly corresponding to new environments / scenarios) and make correct predictions (for unseen / new environments or scenarios).

[0036] As mentioned above, AI / ML for the New Radio (NR) air interface is now being studied in the 3rd Generation Partnership Project (3GPP). One of the objectives of the study is to cover the interoperability and testability aspects of the newly defined AI / ML-enabled features. In this regard, studies on use cases may include, for example, requirements and test frameworks to validate AI / ML-based performance enhancements, taking into account the need and implications for the definition of AI / ML processing capabilities, ensuring that UEs and gNBs with AI / ML meet or exceed existing minimum requirements (if applicable).

[0037] If the model is trained too well on the training data, it will not be able to generalize. In such a case, the model will end up making incorrect predictions when given new data. This will render the model invalid, even if it was able to make correct predictions on the training data set. Such a model that is trained too well can be called overfitting.

[0038] Several mechanisms have been proposed to avoid the overfitting problem and maintain good generalization ability of the model, such as dropOut, batch normalization, early stopping, etc. For example, the following scenarios can be considered to verify the generalization performance of AI / ML models on different scenarios / configurations. An example scenario (referred to as scenario 1) can be to train the AI / ML model based on a training dataset from scenario #A / configuration #A, and then the AI / ML model performs inference / testing on a dataset from the same scenario #A / configuration #A. Another example scenario (referred to as scenario 2) can be to train the AI / ML model based on a training dataset from one scenario #A / configuration #A, and then the AI / ML model performs inference / testing on a dataset different from scenario #A / configuration #A (e.g., scenario #B / configuration #B, scenario #A / configuration #B). Another example case (referred to as case 3) may be based on training an AI / ML model based on a training data set constructed by mixing data sets from multiple scenarios / configurations (including scenario #A / configuration #A and data sets different from scenario #A / configuration #A, e.g., scenario #B / configuration #B, scenario #A / configuration #B), and then the AI / ML model performs inference / testing on a data set of a single scenario / configuration from the multiple scenarios / configurations (e.g., scenario #A / configuration #A, scenario #B / configuration #B, scenario #A / configuration #B). It should be noted that the number of multiple scenarios / configurations may be greater than two.

[0039] In some mechanisms, additional cases for generalized validation may optionally be considered. For example, a case (referred to as Case 2A) may be to train an AI / ML model based on a training dataset from one scenario #A / configuration #A, and then update the AI / ML model based on a fine-tuning dataset different from scenario #A / configuration #A (e.g., scenario #B / configuration #B, scenario #A / configuration #B). Afterwards, the AI / ML model is tested on a dataset different from scenario #A / configuration #A, e.g., subject to scenario #B / configuration #B, scenario #A / configuration #B. Improvements in fine-tuning dataset settings (e.g., size of dataset) and performance may be reported.

[0040] In brief, Case 1 focuses on a training dataset from scenario A / configuration A and performs inference on the same scenario / configuration. Case 2 focuses on a training dataset from scenario A / configuration A and performs inference on a different scenario / configuration. Case 2A investigates training on a selected scenario A / configuration A and generalizes to scenario B / configuration B via fine-tuning (e.g., transfer learning) as an evolved version of Case 2. Case 3 investigates both scenario A / configuration A and scenario B / configuration B using a mixed training dataset and tests any combination of scenario A / configuration A and scenario B / configuration B.

[0041] Several evaluation results show that Case 3 and / or Case 2A provide better performance than Case 2. In most cases / evaluations, the performance of Case 3 is degraded from that of Case 1. According to the evaluation results for scenarios with various UE distributions, Case 3 may have similar or slightly higher performance than Case 1.

[0042] For some cases, Case 2 has some performance degradation in most cases or evaluations over Case 1. In Case 2, the AI / ML can still provide better performance than the non-AI baseline Option 2 (based on measurements from beam set B) (e.g., greater than 30% of the best beam prediction unless otherwise stated).

[0043] For some cases, Case 2 has a significant performance degradation over Case 1 in most cases or evaluations. In Case 2, the AI / ML can still provide comparable or worse performance (e.g., less than 30% of the best beam prediction unless otherwise stated) than the non-AI baseline Option 2 (based on measurements from Beam Set B).

[0044] In some mechanisms, unless otherwise stated, the following are assumed in the simulation. For DL ​​Tx beam prediction, measurements from the best Rx beam are used. A fixed set B mode is used. No UE rotation is simulated. Beams can be measured regardless of their signal-to-noise ratio (SNR). No measurement error is assumed. Measurements are assumed in a single time instance (within the channel coherence time interval). No quantization is assumed for Layer 1 (L1)-Reference Signal Received Power (RSRP) measurements. No constraints are assumed on the uplink control information (UCI) payload overhead for full reporting of L1-RSRP measurements for set B for the NW side model.

[0045] These observations are applicable to both Tx beams and beam pairs. The evaluation results are from BM-Case 1, and similar observations are expected for BM-Case 1 when Set B is different from Set A.

[0046] Generalization of AI / ML models / functions becomes one of the main challenges for AI / ML-enabled use cases. In fact, if a model has been trained for given radio conditions or specific parameter settings and then encounters different radio conditions or parameter settings, the performance of the function may be severely affected.

[0047] However, no such testing mechanism exists for the validation of generalized aspects of AI / ML based beam management use cases that impact specification requirements (such as 3GPP) and impact the user equipment (UE) and test equipment (TE) interfaces. Current research focuses on network (NW) side and / or UE side model training as well as model inference for AI.

[0048] In some mechanisms, it needs to be considered whether it is necessary / feasible to introduce some form of generalization and / or scalability related requirements for different scenarios / configurations based on the Radio Access Network 1 (RAN1) protocol. It should be considered whether this can be handled implicitly in the test case definition. The intention is to ensure that the performance is still maintained in different environments / scenarios / configurations.

[0049] In some mechanisms, AI / ML requirements can be defined for situations with existing traditional performance. The traditional performance can serve as a baseline for existing use cases / processes / functionality / measurements that will be enhanced by AI / ML-based approaches.

[0050] The necessity and feasibility of defining requirements or tests to verify the generalization of AI / ML needs to be considered. AI / ML models can be one-sided models and / or two-sided models.

[0051] In some mechanisms, if the configured AI / ML function / model has been trained with the primary represented dataset, then this AI / ML function / model may experience degraded performance when different channel conditions are met in the field. To avoid performance degradation, it should be ensured that the AI / ML configured function / model has been trained with different datasets and can perform well with an acceptable tolerance margin under varying channel conditions. This should also be verified in Radio Access Network 4 (RAN4) testing to fully validate the configured AI / ML function / model.

[0052] In summary, for AI / ML supported use cases, generalized verification has also become an important topic in RAN4 discussions. However, currently there is no such test mechanism to verify the generalized aspects of AI / ML supported functions.

[0053] According to some example embodiments of the present disclosure, a solution for communication function verification is provided. In the solution, based on a first communication parameter set, a first device (such as a test device) receives a first performance metric of a first result of a communication function from a second device (such as a malicious UE or a device under test (DUT)), and based on a reference communication parameter set, receives a reference performance metric of a reference result of the communication function. The first device then verifies the communication function based on a threshold and a first difference between the first performance metric and the reference performance metric.

[0054] In this way, communication functionality, in particular generalization of communication functionality, may be verified. In some example embodiments, a threshold for verifying communication functionality may be based on a difference or similarity between the first communication parameter set and a reference communication parameter set. By using such a threshold, communication functionality verification may take into account performance metric differences and communication parameter differences between the first communication parameter set and the reference communication parameter set. In this way, generalization of AI / ML-based communication functionality or AI / ML-enabled use cases may be verified for different scenarios.

[0055] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0056] Figure 1 An example communication environment 100 is shown in which example embodiments of the present disclosure may be implemented. Figure 1 As shown, the communication network 100 may include a first device 110 and a second device 120. The first device 110 may communicate with the second device 120. The AI / ML model may be deployed at the second device 120. The AI / ML model may be used for any suitable use case or implement any suitable function, such as, but not limited to, channel state information (CSI) feedback enhancement, beam management (BM) (also known as beam prediction), positioning, etc. It may be necessary to monitor or test the performance of the AI / ML model.

[0057] It should be understood that Figure 1 The number of second devices and first devices shown in FIG. 1 is given for illustrative purposes and does not imply any limitation. The communication network 100 may include any suitable number of second devices and first devices.

[0058] In some example embodiments, the first device 110 may include test equipment (TE), and the second device 120 may include a device under test (DUT). For example, the AI / ML model may be implemented at the second device 120. Alternatively, a portion of the AI / ML model may be implemented at the second device 120. The AI / ML model (also referred to as a "model") may provide a communication function.

[0059] As used herein, the term "communication functionality" may refer to a function or service provided by an AI / ML model. The term "communication functionality" may also be referred to as an "AI / ML-based functionality" or "AI / ML-enabled functionality" or "AI / ML-based use case". In some example embodiments, the AI / ML model at the second device 120 may need to be tested or verified. In other words, the communication functionality may need to be tested or verified. For example, the generalization of the communication functionality may need to be verified.

[0060] In some example embodiments, the first device 110 may include a network device (e.g., a gNB), and the second device 120 may include a terminal device (e.g., a UE). For example, it may be necessary to monitor the performance of an AI / ML model.

[0061] In the following, for the purpose of explanation, some example embodiments are described as the first device 110 operating as a network device and the second device 120 operating as a terminal device. However, in some example embodiments, the operations described in conjunction with the terminal device may be implemented at a network device or other device, and the operations described in conjunction with the network device may be implemented at a terminal device or other device.

[0062] In some example embodiments, if the first device 110 is a network device and the second device 120 is a terminal device, the link from the second device 120 to the first device 110 is referred to as an uplink (UL), and the link from the first device 110 to the second device 120 is referred to as a downlink (DL). In the UL, the second device 120 is a transmitting (TX) device (or transmitter), and the first device 110 is a receiving (RX) device (or receiver). In the DL, the first device 110 is a TX device (or transmitter), and the second device 120 is an RX device (or receiver).

[0063] The communication in the communication environment 100 may be implemented according to any suitable communication protocol, including but not limited to cellular communication protocols of the first generation (1G), second generation (2G), third generation (3G), fourth generation (4G), fifth generation (5G), sixth generation (6G), etc., wireless local area network communication protocols such as Institute of Electrical and Electronics Engineers (IEEE) 802.11, etc., and / or any other protocol currently known or developed in the future. In addition, the communication may utilize any suitable wireless communication technology, including but not limited to: code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), frequency division duplex (FDD), time division duplex (TDD), multiple input multiple output (MIMO), orthogonal frequency division multiple access (OFDM), discrete Fourier transform spread OFDM (DFT-s-OFDM) and / or any other technology currently known or developed in the future.

[0064] Figure 2 An example block diagram 200 for communication function verification according to some example embodiments of the present disclosure is shown. As shown, a first device 110 and a second device 120 are deployed in the block diagram 200. The first device 110 may include a deployment module 210 and a verification module 220. The deployment module 210 may be configured to run or simulate communication parameters (e.g., conditions and / or configurations) to deploy a scenario. For example, a communication scenario 230 may be determined or simulated by the deployment module 210.

[0065] The communication scenarios 230 may include one or more reference scenarios 240 and one or more other scenarios 250. The one or more reference scenarios 240 may be determined by the deployment module 210 based on one or more reference communication parameter sets. The one or more other scenarios 250 (also referred to as candidate scenarios) may be determined by the deployment module 210 based on one or more (other) communication parameter sets (also referred to as candidate communication parameters). Figure 3 Details about the reference communication parameters and the candidate communication parameters are described.

[0066] In some example embodiments, the verification module 220 may be configured to verify a communication function 260 associated with the second device 120. For example, the verification module 220 may verify a generalization of the communication function 260. The communication function 260 may be performed by an AI / ML model implemented at the second device 120. In the example shown, the communication function 260 may include AI / ML-based beam management (BM), AI / ML-based CSI feedback enhancement, AI / ML-based positioning, etc. The result of the communication function device 260 may be an inference 270 of the AI / ML model.

[0067] By using the block diagram 200 , reference conditions and configurations may be tested by using the first device 110 , and accuracy targets for selected key performance indicators (KPIs), such as beam prediction accuracy, may be verified.

[0068] Then, different other scenarios 250 may be tested, which may be combinations of different conditions and / or configurations. The scenarios 250 may be a list of predefined scenarios in the specification, or may be configured by the first device 110. The first device 110 may validate each scenario with a corresponding threshold value (also referred to as a tolerance margin) associated with the selected scenario. If the performance metrics (also referred to as performance results) for all of these various scenarios meet the corresponding threshold values ​​for these scenarios (i.e., within the corresponding tolerance margins), then the generalized aspect may be considered validated. Figure 3 Describes details about the threshold or tolerance margin.

[0069] It should be understood that the present disclosure is not necessarily specific to beam management or beam prediction, which are used here as examples, but can also be used for other use cases such as CSI feedback enhancement, positioning, etc.

[0070] Already referenced Figure 2 An example embodiment of communication function verification is described below with reference to Figure 3 and Figure 4 These embodiments are described in detail.

[0071] Reference now Figure 3 , which shows a signaling diagram 300 for communication function confirmation according to some example embodiments of the present disclosure. Figure 3 As shown, the signaling diagram 300 involves the first device 110 and the second device 120. The signaling diagram 300 may be applicable to any suitable AI / ML use case or any suitable communication function. For example, various communication functions may include, but are not limited to, CSI feedback enhancement, beam management, positioning, etc. Some example embodiments are described below using beam prediction as an example. However, it should be understood that this is only an example without any limitation, and the concepts described with respect to beam prediction may be applied to another AI / ML-based communication function or use case.

[0072] For the purpose of discussion, refer to Figure 1 To describe the signaling diagram 300. Figure 3 As shown, the second device 120 sends (325) a reference performance metric to the first device 110. The first device 110 receives (330) the reference performance metric. As used herein, the term "performance metric" or "performance result" may refer to a metric used to evaluate the results of a communication function. For example, the performance metric may be a KPI, such as prediction accuracy, etc.

[0073] In some example embodiments, the first device 110 may determine (315) a reference communication parameter set. The first device 110 may determine (320) or execute a reference deployment scenario (also referred to as a reference scenario) for the communication function based on the reference communication parameter set.

[0074] The first device 110 may then send information related to the reference deployment scenario to the second device 120. As used herein, the information related to the reference deployment scenario may be referred to as "third information". As an example, the third information may include a request for a reference performance metric. In some example embodiments, the second device 120 may send (325) the reference performance metric to the first device 110 in response to the third information. The first device 110 may verify (335) the accuracy target based on the received (330) reference performance metric.

[0075] In some example embodiments, the first device 110 may determine (315) a reference communication parameter set based on one or more predefined reference communication parameter sets. Alternatively or in addition, the first device 110 may determine (315) a reference communication parameter set based on information from the second device 120. For example, the second device 120 may send (305) second information related to the reference communication parameter set to the first device 110. As used herein, information related to the reference communication parameter set may be referred to as "second information". The first device 110 may receive (310) the second information. The first device 110 may determine (315) a reference communication parameter set based on the second information.

[0076] By way of example, the second information may include supported communication functions and / or lifecycle management (LCM) based on a model identification (ID). The second information may also include applicable conditions associated with the supported communication functions. The applicable conditions may be included in a reference communication parameter set. As used herein, the term "applicable conditions" may refer to conditions for the applicability of a communication function. For example, the second device 120 may notify the network of applicable conditions for the use of AI / ML-based communication functions. In this case, the reference conditions may also be indicated as applicable conditions.

[0077] In some example embodiments, second device 120 may send (305) supported functionality / LCM based on model ID (including its applicable conditions) to first device 110. First device 110 may select reference communication parameters (corresponding to reference scenarios) based on supported functionality / LCM conditions based on model ID or based on predefined reference communication parameters (corresponding to predefined scenarios) in the specification.

[0078] Based on the determined (315) reference communication parameters, the first device 110 may determine (320) a reference scenario or configure a reference scenario and run a test for a performance metric using the reference scenario.

[0079] In some example embodiments, second device 120 sends (360) a performance metric of a result of the communication function to first device 110 based on a communication parameter set that is different from the reference communication parameter set. For example, second device 120 sends (360) a first performance metric of a first result of the communication function to first device 110 based on the first communication parameter set. First device 110 receives (365) the first performance metric.

[0080] In some example embodiments, the first set of communication parameters may be predefined.By way of example, the first device 110 may select the first set of communication parameters from a plurality of predefined sets of communication parameters associated with the communication function.

[0081] In some example embodiments, the first device 110 may determine (345) a deployment scenario for the communication function based on the first set of communication parameters. For example, the first device 110 may run or simulate a deployment scenario.

[0082] The first device 110 may send (350) first information related to the deployment scenario to the second device 120. As used herein, information related to the deployment scenario may be referred to as "first information". The second device 120 may receive (355) the first information. As an example, the first information may include a request for a first performance metric. That is, the first device 110 may request the second device 120 to perform inference and provide the inferred first performance metric.

[0083] In some example embodiments, second device 120 may send (360) the first performance metric to first device 110 in response to receiving (355) the first information. First device 110 may receive (365) the first performance metric. For example, second device 120 may perform inferences and provide the first performance metric and optionally the inferences (such as beam prediction accuracy or predicted beam ID) to first device 110. In some example embodiments, second device 120 may send the inferences to first device 110. First device 110 itself may determine the first performance metric based on the inferences.

[0084] As briefly discussed, the reference communication scenario may be determined based on a reference communication parameter set (320), while the deployment scenario may be determined based on another communication parameter set (such as a first communication parameter set) (345). In some example embodiments, the communication parameters may be related to radio conditions. Alternatively or additionally, in some example embodiments, the communication parameters may be related to a communication configuration associated with the communication function. For example, if the communication function is beam management, the communication configuration may be a configuration of a beam set or a configuration of a beam grid.

[0085] For example, the reference radio conditions may include, but are not limited to, additive white Gaussian noise (AWGN) propagation conditions or pedestrian scenarios in urban macro conditions, or any other radio condition scenarios that are significantly present within the scene.

[0086] The first set of communication parameters may be associated with different radio condition profiles for testing. These radio conditions may be predefined or selected. For example, these radio conditions may be selected from most current scenarios in the field or from standardized propagation modes. For example, example radio conditions may include, but are not limited to, tapped delay line (TDL)-A and / or TDL-C propagation conditions or scenarios with different speeds in urban / rural macro / micro conditions.

[0087] In some example embodiments, the first set of communication parameters may relate to a communication configuration associated with a communication function, such as a configuration or parameters of a beam set or a configuration of a beam grid. The first set of communication parameters may include different configurations / parameters defined or selected for testing, including but not limited to selection of beam set configurations from different network vendors, different beam configuration grids, and different numbers of narrow beams and wide beams.

[0088] Several example communication parameters have been described. It should be understood that the communication parameters may include any other suitable conditions or parameters. Examples of communication parameters may include, but are not limited to: urban macro (UMa) propagation conditions, urban micro (UMi) propagation conditions, 200m inter-site distance (ISD) or 500m ISD, the same deployment, different cells with different configurations / assumptions, carrier frequencies, various outdoor / indoor distributions (or UE distributions) of the second device 120 (e.g., 100% / 0%, 20% / 80%, etc.), various mobilities of the second device 120 (e.g., [3km / h], 30km / h, 60km / h, etc.), other parameters of the second device 120 (e.g., Rx beam codebook / panel of the second device 120), antenna array size of the second device 120, various gNB settings (such as DL Tx beam codebook), various beam (pair) sets A or gNB antenna array sizes, various beam (pair) sets B, and the like.

[0089] The impact of generalization on the performance of various communication functions or AI / ML use cases (each configured with different AI / ML functions / models) may depend largely on the configuration and parameter settings generated for the dataset used for training. For example, for the AI / ML beam management use case, communication parameters such as configuration may cover different beam sets / codebooks used, number of wide / narrow beams, grid of beam configurations, etc. Similarly, communication parameters may include parameter settings such as different scanning frequencies of beams, power settings, etc.

[0090] Statistically, the most current condition / configuration in the field can be found. This can help select the reference condition / configuration. Therefore, in some example embodiments, the reference communication parameters can be determined by the first device 110 based on the most current condition or configuration in the field (315).

[0091] Using the reference performance metric and the first performance metric, the first device 110 may verify (380) the communication functionality. In some example embodiments, the first device 110 may verify (380) the communication functionality based on a threshold and a first difference between the first performance metric and the reference performance metric. As used herein, the difference between a particular performance metric and the reference performance metric may be referred to as "performance degradation" or "performance difference".

[0092] In some example embodiments, the threshold may be predefined or configured. Alternatively or in addition, in some example embodiments, the first device 110 may determine (370) the threshold based on a second difference between the first communication parameter set and the reference communication parameter set. For example, the first device 110 may determine a similarity, such as an index of the similarity between the first communication parameter set and the reference communication parameter set. The index of the similarity may reflect the second difference. That is, the first device 110 may identify the similarity between the selected pair of conditions and the reference pair, thereby determining a second difference between a given pair of conditions and configurations and a reference pair of conditions and configurations (e.g., using an algorithm implemented at the first device 110). It should be understood that the first device 110 may also use other parameters (such as distance) to represent the second difference. The scope of the present disclosure is not limited in this respect.

[0093] In some example embodiments, a larger second difference or a lower similarity may result in a larger threshold value (also referred to as a tolerance margin or tolerance level). Likewise, a smaller second difference or a larger similarity may result in a smaller threshold value. The first device 110 may determine the threshold value for all combinations of conditions and configurations based on the difference between a given pair of conditions and configurations and a reference pair of conditions and configurations.

[0094] By way of example, the second device 120 may experience a degree of variation that is different from the reference conditions / configurations in the field. Therefore, in order to ensure that the second device 120 performs within a tolerable margin and that different conditions and configurations can be tested and verified in RAN4, different tolerance margins for different conditions / configurations need to be defined according to the degree to which these conditions / configurations differ from the reference conditions / configurations. In this way, verification can be more accurate and reliable.

[0095] Alternatively or in addition, in some example embodiments, the first device 110 may obtain the threshold value by retrieving the threshold value from a predefined table. The predefined table may be determined based on the difference or similarity between the plurality of communication parameter sets and the reference communication parameter. For example, the predefined threshold value may depend on the degree of difference between the selected scenario and the reference scenario. Tables 1 and 2 below show threshold values ​​for several scenarios based on different communication parameter sets. Table 1 Thresholds

[0096] In Table 1, the reference scenario corresponds to normal radio conditions and AWGN. Different thresholds may be determined for other scenarios with different conditions. Table 2 Thresholds Reference Parameters Parameter Set 1 Parameter Set 2 Reference beam set 0％ 2％ 4％ Beam Set 1 3％ 4％ 6％ Beam Set 2 5％ 6％ 8％

[0097] In Table 2, the reference scenario corresponds to the reference parameters and the reference beam set. Different thresholds may be determined for other scenarios with different parameter sets and / or different beam sets.

[0098] In some example embodiments, if first device 110 verifies (375) the first performance metric and determines that the first difference is greater than or equal to the threshold, first device 110 may determine that generalization of the communication function is invalid (380). As used herein, if the difference between a specific performance metric and a reference performance metric is greater than or equal to a corresponding threshold, the specific performance metric may be referred to as a performance metric that does not satisfy the threshold or does not satisfy a tolerance margin.

[0099] Alternatively or in addition, in some example embodiments, if the first device 110 verifies (375) the first performance metric and determines that the first difference is less than the threshold, the first device 110 may determine that the first performance metric is verified (375). As used herein, if the difference between a particular performance metric and a reference performance metric is less than a corresponding threshold, the particular performance metric may be referred to as a performance metric that satisfies the threshold or satisfies a tolerance margin.

[0100] In some example embodiments, first device 110 may also verify (375) a second performance metric of a second result of the communication function based on the second communication parameter set. For example, first device 110 may receive the second performance metric from second device 120. First device 110 may verify (375) the second performance metric based on a difference between the second performance metric and the reference performance metric and based on a second threshold associated with the second communication parameter set. First device 110 may also verify (380) the communication function based on verification (375) of the second performance metric.

[0101] In some example embodiments, loop 340 may be performed by first device 110 and second device 120. That is, first device 110 may determine (345) different deployment scenarios for different communication parameter sets. For example, in loop 340, first device 110 may select scenarios one by one from a selected / predefined scenario list and run performance tests for the selected scenarios.

[0102] The first device 110 may obtain different performance metrics corresponding to different scenarios. The first device 110 may determine (370) corresponding thresholds for the communication parameter sets. The first device 110 may verify (375) each performance metric based on the corresponding threshold. For example, the second device 120 may perform a test on all selected combinations and report the results to the first device 110 (such as a performance verification entity).

[0103] The first device 110 may validate the test based on defined tolerance margins for different combinations. If the performance metric for the selected / predefined scenario is within the tolerance margin, the generalized aspects for the AI / ML-based BM may be considered validated, otherwise it is not validated.

[0104] Based on the verification of the different performance metrics (375), the first device 110 can verify (380) the communication function. For example, if all performance metrics are verified (375), that is, all performance metrics meet the corresponding thresholds, the first device 110 can determine that the generalization of the communication function is verified (380). Otherwise, if a specific performance metric is not verified (375), the first device 110 can determine that the generalization of the communication function is invalid (380).

[0105] Alternatively, in some example embodiments, if the number of verified (375) performance metrics is greater than or equal to a predefined or configured threshold number, then the generality of the communication functionality is verified (380). For example, if a majority of the performance metrics meet the respective thresholds, then the generality of the communication functionality is verified (380).

[0106] In this way, communication functions, such as generalization of communication functions, can be verified. Through verification, it can be guaranteed that the AI / ML-based communication functions can work with acceptable accuracy for different conditions and configurations. A more robust system can thus be achieved, which can then be a field performance monitoring mechanism.

[0107] Already about Figure 2 and Figure 3 Several example embodiments for communication function verification are described. In order to better understand the solution of the present disclosure, Figure 4 Describing further example embodiments, Figure 4 An example process 400 for communication function verification according to some example embodiments of the present disclosure is shown. The process 400 may be implemented by the first device 110.

[0108] At block 410, the first device 110 may select a reference pair of radio conditions and configuration / parameter settings. Examples of reference radio conditions may be AWGN propagation conditions or pedestrian scenarios in urban macro conditions, or any other radio condition scenario that is significantly present in the field.

[0109] Examples of reference configurations may be a specific (selected / mutually agreed) set of beams, number of wide / narrow beams, beam grid configuration. Examples of reference parameter settings would be specific (selected / mutually agreed) scan frequencies and power settings for beams, etc. When reporting the conditions assigned to each supported function / model ID, the reference configuration may be identified. Thus, the reference configuration may be a specific condition corresponding to the supported function / model ID.

[0110] The principle for selecting reference conditions / configurations may be to select conditions that are typically tested in RAN4 verification (e.g. AWGN conditions) or conditions that are primarily encountered by the second device 120 in the field after deployment (e.g. slow moving UEs in urban macro conditions).

[0111] At block 420, the first device 110 may select different pairs of conditions and configurations. For example, the first device 110 may define or select different radio condition profiles for testing. This may be selected from most current scenarios in the field or from standardized propagation modes (e.g., TDL-A and / or TDL-C propagation conditions or scenarios with different speeds in urban / rural macro / micro conditions). For example, the first device 110 may define or select different configurations / parameters for testing. This may be a selection of beam set configurations from different network vendors, different beam grid configurations, and different numbers of narrow and wide beams.

[0112] In some example embodiments, there may be a predefined list of scenarios with various conditions and configurations. These scenarios may be a combination of conditions commonly tested in RAN4 (such as TDL-A, TDL-C channel conditions) with parameters changed to calibrate normal and extreme radio conditions. The list of scenarios may also be influenced by the most current scenarios in the field (such as slow moving UEs / high speed UEs in micro / macro urban / rural conditions).

[0113] In the following description, examples of types of propagation models such as AWGN, TDL-A, TDL-C with static, slow moving and high speed UEs may be used. For example, for clarity and simplicity, AWGN with static conditions may be selected as a reference condition. It should be understood that this is just an example.

[0114] At block 430, the first device 110 may define an index of similarities between the selected pair of conditions and configurations and the reference pair. For example, the first device 110 may identify similarities between the selected pair of conditions and the reference pair, thereby determining a difference between a given pair of conditions and configurations and the reference pair of conditions and configurations (e.g., using an algorithm implemented at the first device 110).

[0115] At block 440, the first device 110 may perform a test on each selected pair of conditions and configurations. Each pair has a different tolerance margin. For example, the first device 110 may define a tolerance margin for each combination of conditions and configurations based on the degree to which a given pair of conditions and configurations forms a reference pair of conditions and configurations.

[0116] If one assumes that the beam prediction accuracy target for a reference condition / configuration is 95% of correctly predicted beams, i.e., the best predicted beam matches the ground truth best beam (where "match" may mean that the RSRP of the best predicted beam is within, e.g., 1 dB of the RSRP of the ground truth best beam), then Table 1 above depicts different tolerance margins for different scenarios, depending on how close they are to the reference condition.

[0117] The alternative conditions for testing can be selected relative to the reference conditions. For example, if the model was originally trained in rural pedestrian conditions, vehicle conditions may already be extreme to it. However, if the model is used to train in rural pedestrian and vehicle conditions, urban conditions may be considered extreme to it.

[0118] Likewise, a reference beam set m is selected together with other reference parameter settings. Subsequently, various configurations may be defined in the specification with different beam sets and different parameter settings. Depending on how close these beam sets and parameters are to the reference beam set and reference parameter settings, different tolerance margins may be specified for different combinations. Table 2 above depicts different tolerance margins for different configurations, depending on how close they are to the reference configuration.

[0119] At block 450, the first device 110 may determine whether the performance degradation is less than a tolerance margin. The performance degradation may refer to a difference between a performance metric and a reference performance metric. In some example embodiments, the performance metric may be a performance KPI. For example, the performance KPI may be beam prediction accuracy.

[0120] Performance KPIs (e.g., beam prediction accuracy for reference conditions / configurations) may be defined as targets in the RAN4 specification and will be part of the performance requirement testing for AI / ML based BM use cases. To validate generalized aspects, it is not sufficient to validate only the performance KPIs for a given condition / configuration, therefore, it is necessary to test and validate AI / ML based BM use cases for various conditions and configurations.

[0121] In addition, discussions on applicable conditions are currently ongoing in RAN1, where the second device 120 can inform the network of applicable conditions for AI / ML-based solutions. In this case, the reference conditions can also be indicated as applicable conditions.

[0122] In some example embodiments, if the performance degradation is less than the tolerance margin, the first device 110 may determine that the generalization of the current set of communication parameters is verified at block 460. Otherwise, if the performance degradation is greater than or equal to the tolerance margin, the first device 110 may determine at block 470 that the generalization of the communication function of the current communication parameters is invalid.

[0123] By using process 400, AI / ML-based communication functions (such as beam management) can be tested for various other conditions / configurations. In this way, generalized aspects of AI / ML-based solutions, such as generalization of communication functions, can be verified.

[0124] Already referenced Figures 2 to 4 Several example embodiments are described. It should be understood that these example embodiments may be used in any suitable combination, or alone. It should be noted that Figures 2 to 4 The communication functions of beam management or beam prediction are shown as examples. The above concepts can be applied to other communication functions or use cases, such as CSI feedback enhancement, positioning, etc.

[0125] Figure 5 A flowchart of an example method 500 implemented at a first device according to some example embodiments of the present disclosure is shown. For the purpose of discussion, Figure 1 The perspective description method 500 of the first device 110 in FIG.

[0126] At block 510 , the first device 110 receives a first performance metric of a first result of a communication function from a second device based on a first set of communication parameters.

[0127] At block 520 , the first device 110 receives a reference performance metric of a reference result of a communication function from the second device based on the reference communication parameter set.

[0128] At block 530 , the first device 110 verifies the communication functionality based on the threshold and a first difference between the first performance metric and the reference performance metric.

[0129] In some example embodiments, method 500 includes: determining a deployment scenario of a communication function based on a first communication parameter set; sending first information related to the deployment scenario to a second device; and receiving a first performance metric in the deployment scenario from the second device.

[0130] In some example embodiments, the method 500 comprises selecting a first set of communication parameters from a plurality of predefined sets of communication parameters associated with a communication function.

[0131] In some example embodiments, the method 500 includes: receiving second information related to a reference communication parameter set from a second device; and determining the reference communication parameter set based on the second information.

[0132] In some example embodiments, method 500 includes: determining a reference deployment scenario of a communication function based on a reference communication parameter set; sending third information related to the reference deployment scenario to a second device; and receiving a reference performance metric in the reference deployment scenario from the second device.

[0133] In some example embodiments, the method 500 includes determining the threshold based on a second difference between the first set of communications parameters and a reference set of communications parameters.

[0134] In some example embodiments, the communication parameter relates to at least one of: a radio condition, a communication configuration associated with the communication function.

[0135] In some example embodiments, the method 500 further comprises: determining that generalization of the communication function is invalid based on determining that the first difference is greater than or equal to a threshold value.

[0136] In some example embodiments, method 500 further includes: based on determining that the first difference is less than the threshold, receiving a second performance metric of a second result of the communication function from the second device based on the second communication parameter set; and further verifying the communication function based on at least the second performance metric.

[0137] In some example embodiments, the communication functionality includes beam management.

[0138] Figure 6 FIG. 6 is a flowchart showing an example method 600 implemented at a second device according to some example embodiments of the present disclosure. Figure 1 The perspective description method 600 of the second device 120 in FIG.

[0139] At block 610 , the second device 120 sends a first performance metric of a first result of a communication function to the first device based on the first set of communication parameters.

[0140] At block 620 , the second device 120 sends a reference performance metric of a reference result of the communication function to the first device based on the reference communication parameter set.

[0141] In some example embodiments, method 600 further includes: receiving first information related to a deployment scenario of the communication function from the first device based on the first communication parameter set; and sending a first performance metric in the deployment scenario to the first device in response to receiving the first information.

[0142] In some example embodiments, the method 600 further comprises: sending second information related to the reference communication parameter set to the first device.

[0143] In some example embodiments, method 600 further includes: receiving third information related to a reference deployment scenario of the communication function from the first device based on a reference communication parameter set; and sending a reference performance metric in the reference deployment scenario to the first device in response to receiving the third information.

[0144] In some example embodiments, the communication functionality includes beam management.

[0145] In some example embodiments, a first device (eg, Figure 1 The first device 110 in the method 500 may include components for performing the corresponding operations of the method 500. The components may be implemented in any suitable form. For example, the components may be implemented in a circuit or a software module. The first device may be implemented as or included in Figure 1 In the first device 110.

[0146] In some example embodiments, the first device 110 includes: a component for receiving a first performance metric of a first result of a communication function from a second device based on a first communication parameter set; a component for receiving a reference performance metric of a reference result of the communication function from the second device based on a reference communication parameter set; and a component for verifying the communication function based on a threshold and a first difference between the first performance metric and the reference performance metric.

[0147] In some example embodiments, the first device further includes: a component for determining a deployment scenario of a communication function based on a first set of communication parameters; a component for sending first information related to the deployment scenario to the second device; and a component for receiving a first performance metric in the deployment scenario from the second device.

[0148] In some example embodiments, the first apparatus further comprises means for selecting the first set of communication parameters from a plurality of predefined sets of communication parameters associated with the communication function.

[0149] In some example embodiments, the first apparatus further comprises: means for receiving second information related to the reference communication parameter set from the second apparatus; and means for determining the reference communication parameter set based on the second information.

[0150] In some example embodiments, the first device also includes: a component for determining a reference deployment scenario of a communication function based on a reference communication parameter set; a component for sending third information related to the reference deployment scenario to the second device; and a component for receiving a reference performance metric in the reference deployment scenario from the second device.

[0151] In some example embodiments, the first apparatus further comprises means for determining the threshold based on a second difference between the first communication parameter set and a reference communication parameter set.

[0152] In some example embodiments, the communication parameter relates to at least one of: a radio condition, a communication configuration associated with the communication function.

[0153] In some example embodiments, the first apparatus further comprises means for determining that generalization of the communication function is invalid based on determining that the first difference is greater than or equal to a threshold value.

[0154] In some example embodiments, the first device further includes: a component for receiving a second performance metric of a second result of the communication function from the second device based on a second set of communication parameters in response to determining that the first difference is less than a threshold; and a component for further verifying the communication function based on at least the second performance metric.

[0155] In some example embodiments, the communication functionality includes beam management.

[0156] In some example embodiments, the first apparatus further comprises means for performing the method 500 or other operations in some example embodiments of the first apparatus 110. In some example embodiments, the means comprises at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the execution of the first apparatus.

[0157] In some example embodiments, a second device (eg, Figure 1 The second device 120 in the method 600 may include components for performing the corresponding operations of the method 600. The components may be implemented in any suitable form. For example, the components may be implemented in a circuit or a software module. The second device may be implemented as or included in Figure 1 In the second device 120.

[0158] In some example embodiments, the second device includes: a component for sending a first performance metric of a first result of a communication function to the first device based on a first communication parameter set; and a component for sending a reference performance metric of a reference result of the communication function to the first device based on a reference communication parameter set.

[0159] In some example embodiments, the second device further includes: a component for receiving first information related to a deployment scenario of the communication function from the first device based on the first communication parameter set; and a component for sending a first performance metric in the deployment scenario to the first device in response to receiving the first information.

[0160] In some example embodiments, the second apparatus further comprises means for sending second information related to the reference communication parameter set to the first apparatus.

[0161] In some example embodiments, the second device further includes: a component for receiving third information related to a reference deployment scenario of the communication function from the first device based on a reference communication parameter set; and a component for sending a reference performance metric in the reference deployment scenario to the first device in response to receiving the third information.

[0162] In some example embodiments, the communication functionality includes beam management.

[0163] In some example embodiments, the second apparatus further comprises means for performing the method 600 or other operations in some example embodiments of the second apparatus 120. In some example embodiments, the means comprises at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the execution of the second apparatus.

[0164] Figure 7 700 is a simplified block diagram of a device 700 suitable for implementing an example embodiment of the present disclosure. The device 700 may be provided to implement a communication device, such as Figure 1 The first device 110 or the second device 120 shown in FIG. As shown in the figure, the device 700 includes one or more processors 710, one or more memories 720 coupled to the processor 710, and one or more communication modules 740 coupled to the processor 710.

[0165] The communication module 740 is used for two-way communication. The communication module 740 has one or more communication interfaces to facilitate communication with one or more other modules or devices. The communication interface may represent any interface necessary to communicate with other network elements. In some example embodiments, the communication module 740 may include at least one antenna.

[0166] Processor 710 may be of any type suitable for the local technology network, and may include one or more of the following: a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture, as non-limiting examples. Device 700 may have multiple processors, such as application specific integrated circuit chips that are time slaved to a clock synchronized with a main processor.

[0167] The memory 720 may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, read-only memory (ROM) 724, electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disk (CD), digital video disk (DVD), optical disk, laser disk, and other magnetic storage and / or optical storage. Examples of volatile memories include, but are not limited to, random access memory (RAM) 722 and other volatile memories that do not persist during a power outage.

[0168] The computer program 730 includes computer executable instructions executed by the associated processor 710. The instructions of the program 730 may include instructions for performing the operations / actions of some example embodiments of the present disclosure. The program 730 may be stored in a memory (e.g., ROM 724). The processor 710 may perform any suitable actions and processes by loading the program 730 into the RAM 722.

[0169] The exemplary embodiments of the present disclosure may be implemented by a program 730 so that the device 700 may execute the Figures 2 to 6 Any process of the present disclosure discussed. The exemplary embodiments of the present disclosure may also be implemented by hardware or a combination of software and hardware.

[0170] In some example embodiments, the program 730 may be tangibly embodied in a computer-readable medium that may be included in the device 700 (such as in the memory 720) or in other storage devices accessible by the device 700. The device 700 may load the program 730 from the computer-readable medium to the RAM 722 for execution. In some example embodiments, the computer-readable medium may include any type of non-transitory storage medium, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc. The term "non-transitory" as used herein is a limitation of the medium itself (i.e., tangible, not a signal), not a limitation on data storage persistence (e.g., RAM versus ROM).

[0171] Figure 8 An example of a computer readable medium 800 is shown, which may be in the form of a CD, DVD or other optical storage disk. The computer readable medium 800 has a program 730 stored thereon.

[0172] Generally, various embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, and other aspects may be implemented in firmware or software, which may be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of the present disclosure are illustrated and described as block diagrams, flow charts, or using some other graphical representations, it should be understood that, as non-limiting examples, the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof.

[0173] Some example embodiments of the present disclosure also provide at least one computer program product tangibly stored on a computer-readable medium (such as, non-volatile computer-readable medium). The computer program product includes computer executable instructions, such as those computer executable instructions executed in a device on a target physical or virtual processor included in a program module to perform any method as described above. Typically, a program module includes routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The functionality of a program module can be combined or split between program modules as needed in different embodiments. Machine executable instructions for program modules can be executed in local or distributed devices. In distributed devices, program modules can be located in local and remote storage media.

[0174] The program code for executing the method of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that the program code, when executed by the processor or controller, promotes the realization of the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, as an independent software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0175] In the context of the present disclosure, computer program codes or related data may be carried by any suitable carrier to enable a device, apparatus or processor to perform the various processes and operations described above. Examples of carriers include signals, computer readable media, etc.

[0176] The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. More specific examples of computer readable storage media will include an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage component, a magnetic storage component, or any suitable combination of the foregoing.

[0177] In addition, although the operations are depicted in a specific order, this should not be understood as requiring the specific order shown or in a sequential order to perform these operations, or to perform all the described operations, to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limitations on the scope of the present disclosure, but should be interpreted as descriptions of features that may be specific to a specific embodiment. Unless explicitly stated, certain features described in the context of a separate embodiment may also be implemented in combination in a single embodiment. On the contrary, unless explicitly stated, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0178] Although the disclosure has been described in language specific to structural features and / or methodological actions, it is to be understood that the disclosure defined in the appended claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are disclosed as example forms of implementing the claims.

[0179] Some examples of this application are provided below.

[0180] Example 1. A first device comprises: at least one processor; and at least one memory storing instructions, which, when executed by the at least one processor, cause the first device to: receive a first performance metric of a first result of a communication function from a second device based on a first communication parameter set; receive a reference performance metric of a reference result of the communication function from the second device based on a reference communication parameter set; and verify the communication function based on a threshold and a first difference between the first performance metric and the reference performance metric.

[0181] Example 2. A first device according to Example 1, wherein the first device is configured to: determine a deployment scenario of the communication function based on the first communication parameter set; send first information related to the deployment scenario to the second device; and receive the first performance metric in the deployment scenario from the second device.

[0182] Example 3. The first apparatus of Example 1 or Example 2, wherein the first apparatus is caused to select the first set of communication parameters from a plurality of predefined sets of communication parameters associated with the communication function.

[0183] Example 4. The first apparatus of any one of Examples 1 to 3, wherein the first apparatus is caused to: receive second information related to the reference communication parameter set from the second apparatus; and

[0184] The reference communication parameter set is determined based on the second information.

[0185] Example 5. A first device according to any one of Examples 1 to 4, wherein the first device is configured to: determine a reference deployment scenario of the communication function based on the reference communication parameter set; send third information related to the reference deployment scenario to the second device; and receive the reference performance metric in the reference deployment scenario from the second device.

[0186] Example 6. The first apparatus of any one of Examples 1 to 5, wherein the first apparatus is caused to determine the threshold based on a second difference between the first communication parameter set and the reference communication parameter set.

[0187] Example 7. The first apparatus of any one of Examples 1 to 6, wherein the communication parameter is related to at least one of: a radio condition, a communication configuration associated with the communication function.

[0188] Example 8. The first device of any one of Examples 1 to 7, wherein the first device is configured to: determine that generalization of the communication function is invalid based on determining that the first difference is greater than or equal to the threshold.

[0189] Example 9. A first device according to any one of Examples 1 to 8, wherein the first device is configured to: receive a second performance metric of a second result of the communication function from the second device based on a second set of communication parameters in response to determining that the first difference is less than the threshold; and further verify the communication function based at least on the second performance metric.

[0190] Example 10. A first device according to any one of Examples 1 to 9, wherein the communication function includes beam management.

[0191] Example 11. A second device, comprising: at least one processor; and at least one memory storing instructions, the instructions, when executed by the at least one processor, causing the second device to: send a first performance metric of a first result of a communication function to a first device based on a first set of communication parameters; and

[0192] Based on a reference communication parameter set, a reference performance metric of a reference result of the communication function is sent to the first device.

[0193] Example 12. A second device according to Example 11, wherein the second device is configured to: receive first information related to a deployment scenario of the communication function from the first device based on the first communication parameter set; and send the first performance metric in the deployment scenario to the first device in response to receiving the first information.

[0194] Example 13. The second apparatus of Example 11 or Example 12, wherein the second apparatus is caused to: send second information related to the reference communication parameter set to the first apparatus.

[0195] Example 14. A second device according to any one of Examples 11 to 13, wherein the second device is configured to: receive third information related to a reference deployment scenario of the communication function from the first device based on the reference communication parameter set; and send the reference performance metric in the reference deployment scenario to the first device in response to receiving the third information.

[0196] Example 15. A second device according to any one of Examples 11 to 14, wherein the communication function includes beam management.

[0197] Example 16. A method comprising: at a first device, receiving a first performance metric of a first result of a communication function from a second device based on a first communication parameter set; receiving a reference performance metric of a reference result of the communication function from the second device based on a reference communication parameter set; and verifying the communication function based on a threshold and a first difference between the first performance metric and the reference performance metric.

[0198] Example 17. A method comprising: at a second device, sending a first performance metric of a first result of a communication function to a first device based on a first communication parameter set; and sending a reference performance metric of a reference result of the communication function to the first device based on a reference communication parameter set.

[0199] Example 18. A first device, comprising: a component for receiving a first performance metric of a first result of a communication function from a second device based on a first communication parameter set; a component for receiving a reference performance metric of a reference result of the communication function from the second device based on a reference communication parameter set; and a component for verifying the communication function based on a threshold and a first difference between the first performance metric and the reference performance metric.

[0200] Example 19. A second device comprising: a component for sending a first performance metric of a first result of a communication function to a first device based on a first communication parameter set; and a component for sending a reference performance metric of a reference result of the communication function to the first device based on a reference communication parameter set.

[0201] Example 20. A computer-readable medium comprising instructions stored thereon, the instructions being used to cause an apparatus to perform at least the method according to Example 16 or the method according to Example 17.

Claims

1. A first device for communication, comprising: at least one processor; as well as at least one memory storing instructions, which, when executed by the at least one processor, cause the first device to: receiving, from the second device, a first performance metric of a first result of the communication function based on the first set of communication parameters; receiving, from the second device, a reference performance metric of a reference result of the communication function based on a reference set of communication parameters; as well as The communication function is verified based on a threshold and a first difference between the first performance metric and the reference performance metric.

2. The first device according to claim 1, wherein the first device is configured to: Determining a deployment scenario of the communication function based on the first communication parameter set; sending first information related to the deployment scenario to the second device; and The first performance metric in the deployment scenario is received from the second device.

3. The first device according to claim 1 or 2, wherein the first device is configured to: The first set of communication parameters is selected from a plurality of predefined sets of communication parameters associated with the communication function.

4. The first device according to claim 1 or 2, wherein the first device is configured to: receiving, from the second device, second information related to the reference communication parameter set; and The reference communication parameter set is determined based on the second information.

5. The first device according to claim 1 or 2, wherein the first device is configured to: Determining a reference deployment scenario of the communication function based on the reference communication parameter set; sending third information related to the reference deployment scenario to the second device; as well as The reference performance metric in the reference deployment scenario is received from the second device.

6. The first device according to claim 1 or 2, wherein the first device is configured to: The threshold is determined based on a second difference between the first communication parameter set and the reference communication parameter set.

7. The first device according to claim 1 or 2, wherein the communication parameter is related to at least one of the following: Radio conditions, A communication configuration associated with the communication function.

8. The first device according to claim 1 or 2, wherein the first device is configured to: According to determining that the first difference is greater than or equal to the threshold, it is determined that generalization of the communication function is invalid.

9. The first device according to claim 1 or 2, wherein the first device is configured to: receiving, based on determining that the first difference is less than the threshold, a second performance metric of a second result of the communication function from the second device based on a second set of communication parameters; and The communication functionality is further verified based on at least the second performance metric.

10. The first device according to claim 1 or 2, wherein the communication function includes beam management.

Images