Verification program for machine learning

Abstract

In some aspects, a method is provided that includes, for example, performing, by a verifier entity, at least one secondary task using at least one result indicated in at least received information; a tag for at least one radio signal measurement associated with the at least one result is determined by the verifier entity and using at least one output of the at least one auxiliary task, or a tag for at least one radio signal measurement associated with the at least one result is determined by the verifier entity and based on the at least one output of the at least one auxiliary task. Determining that at least one radio signal measurement associated with the at least one result is to be discarded; and transmitting the tag and the at least one radio signal measurement to the first network node or the second network node to update the machine learning model for the primary task.

Description

Technical Field

[0001] The subject matter described in this article relates to machine learning within communication systems. Background Art

[0002] Artificial intelligence (AI) or machine learning (ML) may be used with various aspects of the air interface between a user equipment (UE, user device, terminal device, or user equipment implemented as a system such as an XR system (augmented reality (AR), virtual reality (VR), and / or mixed reality (MR) or a portion thereof) and a base station (such as a next-generation evolved Node B (gNB) type base station or other type of wireless access point). For example, AI / ML-based techniques may be used to enhance the air interface to provide channel state information (CSI) feedback compression, beam prediction, positioning accuracy, etc. The AI / ML techniques may be varied to support various requirements for the level of gNB-UE cooperation. AI generally refers to the ability of a processor-based device to simulate cognitive functions (such as learning, problem solving, etc.), while ML generally refers to the application of AI via, for example, an ML model. Although the terms "AI" and "ML" are often used interchangeably, for clarity and consistency of explanation, the following description refers to ML, but wherever ML is mentioned in the examples, it includes the use of AI (and vice versa, so when AI is mentioned, it includes ML). Summary of the invention

[0003] In some aspects, a method may be provided, the method comprising receiving, by a verifier entity and from a first network node, a configuration for marking at least one radio signal measurement; receiving, by the verifier entity and from a second network node, information indicating at least one result of an inference of a machine learning model for a primary task, the primary task being associated with the at least one radio signal measurement; performing, by the verifier entity, at least one auxiliary task using at least one result indicated in the received information; determining, by the verifier entity and using at least one output of the at least one auxiliary task, a label for the at least one radio signal measurement associated with the at least one result, or determining, by the verifier entity and based on at least one output of the at least one auxiliary task, that the at least one radio signal measurement associated with the at least one result is to be discarded; and transmitting the label and the at least one radio signal measurement to the first network node or the second network node to update the machine learning model for the primary task.

[0004] In some variants, one or more of the features disclosed herein including the following features may optionally be included in any feasible combination. The operations performed by the validator entity may also include modifying the result so that the output of at least one auxiliary task using at least the modified result indicates a successful test of the modified result with at least one radio signal measurement, wherein the modified result is used as a label for the at least one radio signal measurement. The at least one auxiliary task may include a machine learning task and / or a non-machine learning task. The at least one auxiliary task may include at least one task that uses at least one of the result or the at least one radio signal measurement as input. The at least one auxiliary task may include a plurality of auxiliary tasks, wherein the validator entity uses the plurality of auxiliary tasks in combination to cross-validate the result to determine a label for the at least one radio signal measurement. The configuration for at least marking at least one radio signal measurement may also include at least one of the following items: indicating whether all or part of the at least one radio signal measurement is to be marked; a configuration for receiving information indicating the result; information about at least the auxiliary task; and a configuration for transmitting to the trainer entity a set including the label and the at least one radio signal measurement. The first network node may include a trainer entity, and wherein the second network node includes at least one of a user entity or a trainer entity. The verifier entity may be comprised in or include a gNB base station, and wherein the user entity may be comprised in or include a user equipment, and wherein the trainer entity may be comprised in or include a location management function. Determining, by the verifier entity, at least one radio signal measurement to be discarded may further comprise transmitting the at least one radio signal measurement to the first network node or the second network node for a semi-supervised training procedure. The primary task may comprise one of position estimation, line of sight estimation of a signal source, or beam selection.

[0005] Depending on the desired configuration, the above aspects and features can be implemented in systems, devices, methods and / or articles. The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the following description. The features and advantages of the subject matter described herein will become clear from the description and drawings, as well as the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In the accompanying drawings,

[0007] Figure 1 depicts a block diagram with an example of a cellular system including a trainer entity, a user entity, and a verifier entity according to some embodiments;

[0008] Figure 2 depicts an example of a process for training and validating a machine learning model according to some embodiments;

[0009] Figure 3Another example of a process for training and validating a machine learning model according to some embodiments is depicted;

[0010] Figure 4 Another example of a process for training and validating a machine learning model according to some embodiments is depicted;

[0011] Figure 5 depicts an example process at a validator entity according to some embodiments;

[0012] Figure 6 depicts an example of a network node according to some example embodiments; and

[0013] Figure 7 An example of an apparatus according to some example embodiments is depicted.

[0014] Like labels are used to refer to the same or similar items in the drawings. DETAILED DESCRIPTION

[0015] With respect to the physical layer related to the Radio Access Network (RAN) implementation of ML, it may be necessary to specify a given ML model, lifecycle management of the ML model (e.g., dataset construction for training), validation and testing of the selected ML model and corresponding use cases, signaling, training and validation of the ML model, auxiliary information for the ML model, measurement and monitoring, and / or feedback / reporting. With respect to protocols on the RAN, the implementation of ML may specify a given ML model, including aspects related to capability indication, configuration and control procedures (e.g., training / inference), management of data and ML models, and / or collaboration-level specific specification impact for a given use case.

[0016] In some embodiments, ML model training may be provided that uses a validation process to validate whether the result (generated by the ML model as output) when performing the primary task with input of so-called "live" data (such as at least one radio signal measurement) corresponds to the live data input of the ML model. If validated, the live data is labeled with an associated result and transmitted to a training entity to use the result and the associated live data as training data for the ML model primary task. As described above, the live data may correspond to data, such as radio signal measurements (which may be performed, generated, collected or otherwise obtained) at a UE or another entity. To further illustrate, the live data may correspond to UE measurements of radio signal related measurements, such as a channel input response (CIR) for a given transmission / reception point (TRP). In this way, validation allows the live data to be used as training data for supervised (and / or semi-supervised) training of the ML model primary task. Thus, the training data (sets, samples) used in training may increase with additional training data, which increases the diversity and quantity of the training data, and / or may also improve the performance and / or robustness of the ML model primary task inference.

[0017] It should be understood that the data used may be real-time or live data, or may be any data that can be used for this purpose.

[0018] A primary task refers to the primary task (PT) of the ML model when the ML model performs inference. The PT may vary based on the use case. For example, inference on the PT task of the ML model may assist the UE in position estimation (e.g., determining the location of the UE), line of sight (LOS) or non-LOS classification of links to transmission / reception points (TRPs), beam selection, etc. In some embodiments, the validation process performs a non-PT (also referred to herein as a "secondary task") to validate (e.g., test) the result, and optionally or additionally test its associated live data to determine whether to label live data such as at least one radio signal measurement with the result (and the live data mapped to the result is considered validated and therefore suitable for use in the training of the primary task of the ML model).

[0019] For example, in the case where the PT is a location estimate, the training can be performed at, for example, a server or host (e.g., a central server, a central ML unit, and / or an edge server located at (or accessing) a radio access network node (such as a gNB)). For example, the central ML unit can be a 5G network data analysis function (NWDAF), which provides data analysis (including insights and actions) to enhance, for example, location estimates. To further illustrate, the central ML unit can perform training as follows. Data group collection devices can be deployed (or selected) at certain locations. An example of such a data collection device is a so-called positioning reference unit (PRU). A PRU refers to a device at a known location (e.g., a known location can provide a label for use during ML model training). The PRU can perform measurements, and these measurements can then be used to generate correction data, which can be used to refine the location of one or more other UEs (e.g., target UEs) located in the same area as the PRU, so that the positioning accuracy can be improved. In this example, the PRU server provides reference marker data for the ML model training data set. Although some of the examples in the examples involve PRUs, other types of devices or network nodes can also be used to collect data.

[0020] To further illustrate, (multiple) PRUs may make positioning measurements and report the measurements to a central ML unit. Additionally, the central ML unit may simulate positioning measurements. Additionally, the central ML unit may receive (or be provided with) positioning measurements from multiple other sources (although at least some of these other sources may not be accurate). The central ML unit may then choose to combine positioning measurements and simulated positioning measurements from different PRUs to train the positioning ML framework. The ML framework may be deployed at a network entity (also referred to herein as a "host," where the host may be a different type of machine learning process running, including a machine learning model). The host type may include or be included in a UE (e.g., the target UE), a PRU, and / or a radio access network node (such as a gNB), and / or a location management component (LMC) to enhance positioning accuracy.

[0021] For example, in the case of deploying an ML model to a UE or other entity, the problem involves ensuring that the training data (which is used to train the ML model) is diverse and available for training the ML model. However, collecting and labeling training data may burden both the training time (and the associated latency in collecting the training data) and the computational resources required for training. For example, there are limited processing resources at a network node (e.g., UE, gNB, location management function (LMF), etc.) that can measure and extract accurate training data (e.g., input data mapped to corresponding labeled output data) from new radio (e.g., 5G or new radio, NR) measurements. Some ML models for 3GPP cellular systems (e.g., 5G, etc.) may rely on input in the form of channel input response (CIR) estimates, but CIR is not a standard Tier 1 key performance indicator (KPI), so the accuracy, availability, and / or acquisition of CIR measurements may not always be straightforward. In addition, the computational and signaling overhead associated with establishing a process for generating, collecting, and / or labeling large training data may be relatively large and heavy (even when such a process is partially automated based on non-3GPP functions, digital twins, ray tracing, etc.). Additionally, radio resources for reporting large amounts of training data between the UE and the gNB are limited.

[0022] In view of the aforementioned problem about limited resources, partial (also called noisy) labeling of training data can be used, and supervised (or semi-supervised) ML learning can be implemented, in which the training of the ML model receives partially labeled training data as input. Partially labeled training data refers only to a subset (but not all) of the training data that is labeled for ML model training. For example, part of the training data may be labeled while another part is not labeled. However, there is no mechanism or process within 3GPP for handling partially labeled training data for training ML models in supervised (or semi-supervised) ML models.

[0023] In some embodiments, a process is provided for training an ML model to perform a primary task (PT) using unlabeled live data (e.g., radio signal measurements) as input. Such unlabeled data may be considered "live" in that it is data measured (or otherwise obtained) by the UE, e.g., as part of normal radio cell measurements, such as signal samples, signal quality measurements, channel measurements, or other types of data / signals obtained by the UE (e.g., as may be obtained while the UE is engaged in some other data, measurement, or positioning session). For example, the radio signal measurements may include CIR measurements obtained as part of the UE's air interface measurement operations, rather than as a specific ML directed training task at the UE.

[0024] During the learning phase, the ML model can learn to perform the primary task, and the trained ML model can be used to generate results for the primary task performed by the ML model (during the inference phase), such as positioning, compression, beam selection, etc. For example, a training data set including labeled data can be enriched by adding real-world data that has been labeled to be able to be used in ML model training due to the validation process disclosed herein to the training data. Thus, the training data is amplified, which can increase the diversity of the training data set, can improve the performance of the ML model in performing the primary task, and / or can reduce the need to collect (via a dedicated process) a large amount of labeled training data for training the ML model.

[0025] In some embodiments, a signaling exchange is provided that can be used as part of (or, e.g., used to implement) the training of an ML model for a given task, which is referred to herein as a primary task (PT). For example, the training of an ML model for the primary task can include converting unlabeled live data (e.g., collected or obtained from (current) measurements performed by a UE, or (multiple) radio signal measurements or other data provided to the UE by, e.g., another UE) into labeled real-time or live data (e.g., radio signal measurements labeled with, e.g., results), which can be used as training data to train the ML model to perform the PT. This process can occur via a network node (such as an NR entity operating as a "verifier") that verifies the results (such as the ML model output) given the input unlabeled real-time or live data. The verifier can also be located in a server or host, such as an edge cloud unit, or can be available or operably coupled to any device, such as a user equipment (UE) via a radio connection, or it can be part of a system controlled by a user equipment. The verifier can be a logical unit.

[0026] The tag may thus indicate a validation of (live) data (e.g., radio signal measurement data samples) input to the ML model and an associated result of the inference of the PT of the ML model, wherein the validation may be performed with a non-PT (such as an auxiliary task). The non-PT (or auxiliary task) may include at least one ML model-related task and / or at least one non-ML task.

[0027] In some of the examples described herein, the signaling exchange is described with respect to at least one trainer entity, a user entity, and a verifier entity. For example, a trainer entity (also referred to herein as a new radio primary task trainer, NR-PT trainer, or simply "trainer") refers to an NR (e.g., 5G) entity that trains an ML model using labeled training data to solve a given PT. Although the trainer can be implemented at various NR entities, in some of the examples described herein, the trainer may include or be included in other nodes, functions, or entities in a gNB-type base station, a location management function (LMF), and / or a radio access network, a 5G core network, and / or other locations (e.g., as a cloud service). A user entity (also referred to herein as a new radio primary task user, NR-PT user, or simply "user") refers to an NR entity that uses a trained ML model on so-called live data (e.g., radio signal measurements collected, measured, or provided to a user) and produces a solution (also referred to as a result of a PT). Although a user may be implemented at various NR entities, in some examples described herein, a user may include or be included in a UE, although a user may also include or be included in a gNB-type base station, a transmit / receive point (TRP), a location management function (LMF), and / or other nodes, functions, or entities in a radio access network, a 5G core network, and / or other locations (e.g., as a cloud service). In addition, a verifier entity (also referred to herein as a New Radio Primary Task Verifier, an NR-PT Verifier, or simply a "verifier") refers to an NR entity that receives (e.g., obtains) results from at least an NR-PT user.

[0028] In some embodiments, the verifier entity uses the results (e.g., the results of the inference output of the ML model by executing the PT) to solve non-PT or auxiliary tasks. For example, the non-PT can verify the accuracy of the ML model results. If the non-PT is successfully solved (or, for example, the live data combined with the results generates a result that the verifier considers to be correct), the verifier entity verifies the results and marks the results as verified (e.g., by indicating that the results are verified and / or mapping the live data input to the results), so that the results mapped to the live data input (e.g., radio signal measurements) can be passed to the training data for training the PT of the ML model. However, if the non-PT is not successfully solved (e.g., the live data combined with the results generates a non-PT result that is considered incorrect), the verifier entity can iterate by modifying the results to determine whether the non-PT can be successfully solved. If successfully solved, the modified results can be used as a label for the live data input (e.g., radio signal measurements). In this way, the verifier entity can mark unlabeled live data with its (or modified) results, so that the live data is marked with the results or the modified results. If the verifier cannot successfully verify the results with live data, the live data (e.g., (multiple) radio signal measurements) may be discarded and / or sent to another node where the live data (not yet labeled or verified) may be useful for a semi-supervised training procedure.

[0029] Figure 1 A block diagram is depicted having an example of a process between a trainer entity 150 (labeled NR-PT Trainer, or simply “Trainer”), a user entity 152 (labeled NR-PT User, or simply “User”), and a verifier entity 154 (labeled NR-PT Verifier, or simply “Verifier”) in accordance with some embodiments. Figure 1 The example process involves the primary task of position estimation (PT) where an ML model is being trained for inference, but Figure 1 The process can be used for other types of 5G-related primary tasks for ML models for other use cases, such as line-of-sight (LOS) / non-line-of-sight classification, beam selection, etc.

[0030] At 102A, according to some embodiments, the trainer entity 150 may train an ML model for a task, such as a primary task (PT), using the first set of training data 105. The ML model may be trained to perform PT (in this example, "position estimation", although the PT of the ML model may be another type of task, such as beam selection, LOS / NLOS selection, classifier, regressor, etc.). In addition, the ML model (which is Figure 1The training data 105 may be labeled as training data, whereby inputs are mapped to outputs labeled with desired results, such that the training phase may learn ML model parameters to produce outputs / results given the inputs.

[0031] At 106A, the ML model (e.g., PT model v.0) may be deployed to an NR entity, such as user entity 152 (labeled as NR-PT user). For example, the ML model trained at 102A may be deployed (e.g., sent, transmitted, etc.) by deploying parameters (e.g., weights, etc.) of the ML model and / or deploying metadata (e.g., other information that enables use) of the ML model. Figure 1 The trainer 150 is depicted deploying the ML model to the user entity 152 at 106C, but other devices such as an ML orchestrator may also deploy the ML model.

[0032] At 106B, the user entity 152 may use the live data 108, such as one or more radio signal measurements, as input to the ML model and may generate results given the input. Figure 1 In the example of , the result 106C (which is the inferred output of the PT of the ML model) may include at least the result. Alternatively or additionally, the result may include the input (or the location or identity of the input), such as the live data (e.g., radio signal measurements) that generated the result at 106C. The live data (at Figure 1 Live data measurements (LDMs) may correspond to live data measured, generated, collected, reported, etc. by a user entity. For example, a user entity may perform CIR measurements per TRP as part of another (e.g., non-ML session) session of the user entity. Examples of live data include channel input response (CIR) estimates of a TRP, channel state information reference signal (CSI-RS) measurements, reference signal received power (RSRP), angle of arrival (AoA), time of arrival (ToA), and / or other measurements made, generated, or received by a user entity relative to an air interface.

[0033] However, it should be understood that the data (samples) used are not necessarily live or real-time data samples, and for simplicity, the abbreviation LDM is even used in the examples. Any data suitable for this purpose is applicable. In radio communications, measurement data (one or more samples) are obtained by measuring radio signals.

[0034] In some embodiments, the trainer entity 150 may instruct the user entity 152 to store live data used as input to the ML model, such as live data measurements 108, and may instruct the user entity to store the live data measurements together with, for example, a timestamp or other indication to locate or identify the live data measurements. For example, one or more live data measurements provided as input to the ML model 106B may generate corresponding results for the PT of the ML model. In this example, each timestamp may be mapped to a live data measurement input and a corresponding result, so that the LDM, results, and timestamps are stored over a period of time in this example.

[0035] In some embodiments, the user entity 152 may deliver (e.g., send, transmit, provide, etc.) a combination of a (live) data measurement set (LDM) 108, a result, and / or a timestamp to the verifier entity 154 at 106C. When received, the verifier entity may then perform tasks such as non-PT (also referred to herein as auxiliary tasks) to verify the result (e.g., radio signal measurement) mapped to the corresponding LDM input at 160. As previously described, the delivery may include a result, a LDM, and a mapping set of the result, or a mapping set of the LDM, the result, and a timestamp. For example, at 106C, the user entity 152 may send the LDM mapped to the corresponding result and a timestamp to the verifier entity 154. Alternatively or additionally, at 106C, the LDM portion is not provided to the verifier entity (e.g., an indication of the corresponding LDM input is provided to the result to enable identification and / or retrieval of the LDM input). Alternatively or additionally, the user entity may only deliver the LDM and the mapping result to the verifier entity without a timestamp.

[0036] In some embodiments, the verifier entity 154 may verify the result of a given input such as an LDM at 160. For example, the verifier entity may test the result of a given LDM using a non-PT (or auxiliary task) that is used to test or verify the result of a given LDM input. If the test succeeds, this indicates that the ML model correctly inferred the result of the given LDM, so the verifier entity may add a label to the LDM (e.g., by adding the result to the LDM input and / or indicating that the result of the LDM input is verified).

[0037] However, if the non-PT (or auxiliary task) fails to verify the result given the input LDM, the verifier entity 154 may consider the result at 106C to be incorrect, and thus the verifier entity will not mark or verify the LDM and corresponding result.

[0038] Alternatively or additionally, when a non-PT (or auxiliary task) fails, the verifier entity 154 may modify the results by, for example, labeling the LDM samples with a different result, such as in the case where the result is a binary result (e.g., if the PT is a binary classification task, the labels mapped to the LDM input samples may be opposite to that indicated by the result in 106C).

[0039] Alternatively or additionally, when the non-PT (or auxiliary task) fails, the verifier entity 154 may iteratively modify the initial result provided at 106C until the verification test at 160 indicates success. The modified result may then be used as a label for the LDM input. For example, the verifier entity may iteratively (e.g., repeatedly) apply the selected function f to the result provided at 106C to generate the modified result, and then verify the modified result via the non-PT at 160. If the modified result verifies successfully, the modified result is used as a label for the LDM input.

[0040] Alternatively or additionally, when non-PT fails and / or the validator entity is unable to modify (and / or verify) the original result provided at 106C, the validator entity may discard a set or pair of results, such as LDM and PT. Alternatively or additionally, when non-PT fails and / or the validator entity is unable to modify (and / or verify) the original result provided at 106C, the validator entity may forward the LDM (live or real-time data) to enable another node to use the unlabeled LDM in semi-supervised learning of the ML model.

[0041] At 164, the validation results and / or result re-labeling 162 at 160 may be used by the validator entity 154 to form a labeled training data set, which may be delivered or used alone or in combination with the training data 105 to train the PT inference of the ML model. For example, at 170A, the LDM input and labeled result pairs may be delivered to the trainer entity 150 so that the validated LDM input and labeled result pairs / data may be used as input (along with other training data, such as the training data 105) to train (or retrain) the ML model at 102B. Alternatively or additionally, this validated data (LDM, validated results) may be added to the original training data 105 to increase the diversity and quantity of the training data. To further illustrate, the LDM and its labeled results (e.g., LDM1, result 1; LDM2, opposite result 2; and LDM3, modified result 3) may be forwarded at 170A for use in the training of the ML model and / or sent to the trainer 150 to augment the training data 105 with additional training data. Alternatively or additionally, unverified LDMs may also be forwarded to the trainer entity 150 to enable the trainer to attempt to identify labels, for example in a semi-supervised training procedure.

[0042] At 102B, the trainer entity 150 may train another version (e.g., v.1) of the ML model for PT inference using the new training data (which was received at 170A) with the initial training data 105. The trainer entity may replace the ML model (e.g., v.0) with the updated version of the ML model (e.g., PT model v.1) for subsequent use by the user entity 152 to perform PT.

[0043] The non-PT (auxiliary task) may be an ML auxiliary (or enabling) task (or function) that verifies the LDM and results (or functions), and / or a non-ML-based task (or function) that verifies the LDM and results. The non-PT that the verifier entity 154 uses for verification at 160 may be a task (e.g., a program) that uses as input the results generated by the user entity and at least one of the following: (1) at least one of the inputs to the LDM used by the ML model (PT model v.0); (2) other types of measurements made on the same entity and providing as output (one of the outputs) a metric that can be used directly or indirectly to verify the validity of the ML model inference; or a combination of (1) and (2). In addition, the verifier entity 154 may use several different non-PTs in combination, in which case cross-validation may also be used in the verification and labeling of one or more data samples (in many real-world applications, a large number of data samples are required to achieve the target performance accuracy).

[0044] As previously described, the ML model can be trained to perform various primary tasks (PTs) associated with the air interface, and the trained ML model can then be used in the inference phase to perform the PTs. For example, the primary task (PT) of the ML model can be a line of sight (LOS) and / or non-line of sight classification task. For example, in this case, the trainer entity 150 may include or be included in a UE, a TRP, or an LMF (or a location management component in the RAN); the user entity 152 may include or be included in a UE (e.g., for DL ​​positioning) or a TRP (e.g., for UL positioning). The live data measurements may include radio signal measurements, such as CIR estimates per TRP. The result may include a LOS binary flag per TRP (e.g., the LOS flag is equal to 1 when there is LOS toward the corresponding TRP, or the LOS flag is 0 when there is no LOS toward the corresponding TRP). For example, the verifier entity 154 may correspond to an NG RAN entity (e.g., a gNB) that manages non-PTs, in which case the non-PTs are beam selection tasks. Beam selection may use reported results (e.g., LOS flags / indicators per signal source, such as TRPs) along with other auxiliary information to select (and / or predict) beam links between transmitters. The verifier entity 154 may perform verification (using, for example, a non-PT / auxiliary task) by comparing predicted beam pairs (e.g., predicted using radio signal measurements and / or results) with expected or optimal beam pairs (e.g., selected by an exhaustive search over all beam pairs) and their signal levels to verify or refute results associated with a sample LDM (e.g., CIR per TRP) (the LOS flag is equal to 1 when there is LOS towards the corresponding TRP).

[0045] Alternatively or additionally, the PT may be a position estimate. In this example, the trainer entity 150 may include or be included in the UE or LMC and / or LMF; the user entity 152 may include or be included in the UE (for UE-based positioning) or LMF (in UE-assisted positioning). In this example, the LDM may include radio signal measurements, such as CIR per TRP, and the results may include the (x,y) position of the UE. The verifier entity may include or be included in a non-PT NGRAN element that addresses beam management. For example, beam selection may use reported results (e.g., UE position) and / or may use other auxiliary information to select (and / or predict) beam links between transmitters. For example, the verifier entity may compare a predicted beam pair (e.g., predicted using radio signal measurements and / or results) with an expected or optimal beam pair (selected by, for example, an exhaustive search of all beam pairs) and its signal level to verify or refute the results associated with the sample. For example, when the measured signal level with the selected beam pair is less than a threshold loss amount (e.g., 3dB) compared to the signal level with the expected / optimal beam pair, the result can be verified (using, for example, a non-PT or auxiliary task); otherwise the result is not verified (e.g., is refuted).

[0046] Alternatively or additionally, the PT may be beam selection. In this example, for the ML model v.0 deployed in the UE (or split between the UE and the NG-RAN), the trainer entity may include or be included in the NG-RAN. The user entity may include or be included in the UE (e.g., when the PT model v.0 is deployed only in the UE). The LDM may compare radio signal measurements such as CSI-RS measurements and beam reference signal received power (RSRP) measurements with the results corresponding to the beam ID. The verifier entity may include or be included in the NG-RAN and use a positioning algorithm / function running in the LMF as a non-PT. The positioning algorithm may use, for example, CIR, time of arrival (ToA), angle of arrival (AoA) and / or LoS / NLoS as input. Based on the UE location information (and / or with TRP / gNB location and / or radio environment map), the verifier entity may estimate (using, for example, non-PT or auxiliary tasks) the best beam selection (e.g., beam ID) for the UE (or UE-gNB beam pair).

[0047] Figure 2 Another example of a process including a trainer entity 150 (e.g., an NR-PT trainer), a user entity 152 (e.g., an NR-PT user), and a verifier entity 154 (e.g., an NR verifier) ​​is depicted in accordance with some embodiments. Figure 2In the example of , the primary task (PT) involves line of sight (LOS) and / or non-line of sight (NLOS) classification that may be used for beam selection. In the examples described below, the trainer entity may be included in the LMF or LMC, the user entity may be included in the UE, and the verifier entity may be included in a base station, such as a gNB, although other network nodes / entities may be implemented, as well as the trainer, user and / or verifier entities.

[0048] At 1, according to some embodiments, a trainer entity (or simply trainer) 150 may collect a first set of training data (S1, one or more samples) and train an ML model (e.g., PT model v.0) to perform its primary task (PT). For example, the trainer may start training the ML model using the first set of training data (S1) to form an initial version of the ML model, such as PT model v.0, which may include LOS detector data (e.g., CIR per TRP and LOS binary flag per TRP) available at (or accessible by) the trainer. The trainer may include or be included in the LMF. Alternatively or additionally, the LMF may provide training data in the form of LOS (and / or NLOS) detector data.

[0049] At 2, according to some embodiments, the trainer 150 may deploy (e.g., send, transmit, provide, make accessible, etc.) an ML model trained by the trainer 150 to a user entity 152 (e.g., an NR-PT user or simply a user). In addition, the deployment of the ML model may include auxiliary information. For example, the auxiliary information may indicate to the user 152 (e.g., which may be included in the UE) the configuration of the report that the user (e.g., UE) should send as feedback to the verifier entity (NR verifier, or simply a verifier, which may be included in the gNB). The configuration provided by the auxiliary information may include one or more of the following: a minimum number (e.g., amount) of results before reporting to the verifier; what LDM (if any) to report to the verifier; and when to send the report to the verifier (and / or through which interface). Although Figure 2 A trainer deployment is depicted, but other entities may deploy the trained ML model to a user entity. In addition, to deploy the ML model, parameters (e.g., weights of the ML model, etc.) are provided (e.g., sent, transmitted, etc.) to the user entity. To further illustrate, the trainer (which may be included in the LMF or LMC) may transmit assistance data to the user via, for example, an LPP (LTE Positioning Protocol) message (or information element) that includes a configuration of a report that the user should send to the validator.

[0050] At 3, according to some embodiments, the trainer 150 may also configure the verifier 154. For example, the message transmitted (or sent) at 3 may include a configuration for marking at least at least one radio signal measurement. For example, the message at 3 may include auxiliary information, and the auxiliary information may include a configuration for marking at least at least one radio signal measurement. This configuration for marking at least at least one radio signal measurement may include one or more of the following: a request to verify the results reported from one or more users (e.g., NR-PT users) and / or a request to report the results of the ML model verification process to the trainer (e.g., NR-PT trainer). To further illustrate, the trainer (which may include LMF (or LMC)) may send an NR Positioning Protocol A (NRPPa) measurement report to the verifier (e.g., gNB) at 3, but other types of messages may also be used to report the input LDM and verified results. In this example, the NR Positioning Protocol A (NRPPa) measurement report may indicate a request to accept and verify the result report from the NR-PT user list and / or a request to report the results of the verification process to the NR-PT trainer. Alternatively or additionally, the configuration may indicate a configuration for receiving information indicating the result, such as a format. Alternatively or additionally, the configuration may include information about at least an auxiliary task, such as an indication of which auxiliary task the verifier entity should use when verifying the result. Alternatively or additionally, the configuration may include a configuration for transmitting a set including a label and at least one radio signal measurement, such as a trainer entity, to the trainer entity to enable the trainer entity to use the set to train the PT of the ML model. Alternatively or additionally, the configuration may include a list of network nodes, such as user entities or other types of entities, which the verifier entity should (or is allowed to) use to receive the results and associated radio signal measurements for verification. Alternatively or additionally, the trainer entity may include a first network node, which may be associated with the main task of training the ML model and / or providing training data for the training. Alternatively or additionally, the verifier entity may be included in or include a gNB base station. Alternatively or additionally, the user entity may be included in or include a user device. Alternatively or additionally, the trainer entity may be included in or include a location management function. Alternatively or additionally, the main tasks inferred by the ML model may include position estimation, line of sight estimation / detection of signal sources, and / or beam selection.

[0051] At 4, according to some embodiments, user 152 may execute an ML model (e.g., PT model v.0) using the received input data, and the ML model generates an output, such as a result. For example, the input data may include an LDM, such as at least one radio signal measurement (e.g., CIR per TRP, etc.), and the corresponding result may include a LOS flag to a TRP.

[0052] At 5, according to some embodiments, user 152 may map input data to results of the ML model (which are caused by the input data). For example, an input LDM value (CIR per TRP) may be mapped to a corresponding result, such as a LOS flag. Alternatively or additionally, user 152 may apply a timestamp to the set of mapped input data and results.

[0053] At 6, in accordance with some embodiments, the user 152 may report (e.g., transmit, send, provide, etc.) a mapping group (e.g., input LDM value(s), LOS flag(s), and / or timestamp(s)) to the verifier 154, as indicated by the message at 3. For example, the verifier may receive information indicating the result of the ML model inference and an associated indication of the input LDM (which outputs the result) from the user. The indication may include a timestamp (mapped to or associated with the result and the associated input LDM), an identifier enabling identification of the associated input LDM, and / or the input LDM itself. The report may be carried by an uplink (UL) channel, such as a UL shared channel (SCH), a UL short data transfer (SDT), or via RRC signaling.

[0054] At 7, the verifier 154 can use the information received at 6 (e.g., the results and associated input LDMs) to verify the results of a given input LDM using a non-PT (such as an auxiliary task). In the example of 7, the auxiliary task includes using the results of a given input LDM to select the best TX-RX beam pair (which can be identified by a beam index (BI) between a corresponding signal source (TRP) and a UE). For example, if the selected TX-RX beam pair matches the expected beam pair, the LOS result is verified. Otherwise, the LOS result will be modified (e.g., set to an opposite value) and re-verified. In this example, the expected beam pair is a beam pair with a signal-to-interference-plus-noise ratio (SINR) above a given threshold. Alternatively or additionally, the expected beam pair can be identified via an exhaustive search for beam pairs. To further illustrate, the verifier entity can receive information indicating an inference result of a machine learning model for a primary task, the primary task being associated with at least one radio signal measurement. The verifier entity 154 can receive the inference result of the primary task of the ML model given an associated input of at least one radio signal measurement. The result may be received together with the associated at least one radio signal measurement, or the result may be received together with an indication of the identity of the at least one radio signal measurement to enable the verifier entity to access or obtain the at least one radio signal measurement. In addition, the result may be received as an indication to enable the verifier entity to access or obtain the result.

[0055] At 8, when a non-PT (or auxiliary task) is used to verify the result, the live data measurement reported at 6 and verified at 7, such as a radio signal measurement (e.g., CIR per TRP), is labeled with the corresponding result. In other words, the output of the auxiliary task can be used to determine a label of at least one radio signal measurement associated with the result of the ML model. Given the verification, the input LDM (e.g., CIR per TRP) is labeled as a result (e.g., LOS flag 1) that has been verified using the non-PT / auxiliary task, so the input LDM (e.g., CIR per TRP) and the result (e.g., LOS flag 1) can be passed to the training data and / or used during PT training of the ML model. Alternatively or additionally, the LDM-result pair can include a flag or indication of the verification (e.g., LOS flag 1 verified).

[0056] At 9, the verifier 154 can send the labeled LDM to the trainer 150 (S2). For example, the verifier can send the input LDM (e.g., CIR per TRP) and the label results (which have been verified) to the trainer so that the input LDM and the verified results can be used to train the ML model and / or enhance the training data set. The report at 9 can be sent via an NR Positioning Protocol A (NRPPa) measurement report, but other types of messages can also be used to report the input LDM and the verified results.

[0057] At 10, the trainer 150 may include the labeled LDM training data (S2) provided by the validator 154 at 9 with the original training data (S1), so that the aggregated training data (S1 and S2) can be used to train the ML model (e.g., PT model v.0) to form another ML model (e.g., PT model version v.1). For example, the updated or second ML model (e.g., PT model version v.1) may be as follows Figure 2 The illustrated embodiments are further trained and / or provided to an entity, such as a user entity, for use in performing corresponding PT tasks (e.g., LOS and / or NLOS classification) during an inference phase of an updated or second ML model.

[0058] Figure 3 Depicted is another example of a process between a trainer 150 (labeled NR-PT Trainer), a user 152 (labeled NRPT User), and a verifier 154 in accordance with some embodiments. Figure 3 Similar in some ways to Figure 2 , but in Figure 3During the process, the result (which in this example is LOS NLOS) is reported by the user entity (NR-PT user) and used by the verifier (NR verifier). This can reduce the signaling overhead between the NR-PT user and the NR verifier (message 7) and the NR verifier and the NR-PT trainer (message 9), but may require the NR verifier to have a way to obtain other channel information required to run beam selection (in addition to the already reported LOS flag).

[0059] For example, refer to Figure 3 , at 2, the trainer 150 can be deployed to the user entity 152 (the ML model trained by the trainer 150). And the deployment of the ML model may include auxiliary information provided to the user entity. Figure 2 Instead, the reports (and configuration of the reports) are specific to the trainer 150 .

[0060] refer to Figure 3 At 6, user 152 may report the mapped group (e.g., input LDM value(s), LOS flag(s), and / or timestamp(s)) to trainer 150, which may be indicated (or commanded) by auxiliary information provided by the trainer to the user. The report may be carried by an uplink (UL) channel, such as a UL shared channel (SCH), a UL short data transfer (SDT), or via radio resource control (RRC) signaling.

[0061] refer to Figure 3 At 7, the trainer 150 may then report the results, such as the LOS flag per TRP, to the verifier 154. Figure 3 At 9 , the validator 154 may report the verified results (eg, verified LOS marking) to the trainer 150 .

[0062] refer to Figure 3 , at 10, the trainer 150 can then use Figure 3 The verified results provided at step 9 mark the LDM input (e.g., CIR per TRP). Then, Figure 3 In 11, an input LDM (e.g., CIR per TRP) and a label including a verified result (e.g., verified LOS flag 1) can be used to train an ML model, which uses the original training data (S1) and a new labeled dataset (S2) of the input LDM (e.g., CIR per TRP) and a label including a verified result (e.g., verified LOS flag). When in 11 ( Figure 3 ) When the ML model is trained using the enhanced training sets of S1 and S2, another version of the ML model (e.g., PT model version v.1) can be generated. PT model version v.1 can be Figure 3The illustrated embodiment is further trained and / or provided to an entity, such as a user entity, for performing LOS / NLOS classification during an inference phase of an updated or second ML model.

[0063] Figure 4 Depicted is another example of a process between a trainer 150 (labeled NR-PT Trainer), a user 152 (labeled NR-PT User), and a verifier 154 in accordance with some embodiments. Figure 4 Similar in some ways to Figure 3 , but in Figure 4 During the process, the UE position is estimated instead of Figure 3 The LOS / NLOS classification PT is the main task (PT) of the ML model (PT model v.0). Figure 4 , at 4 to 10, the input LDM includes the CIR per TRP, and the result includes the location of the user (e.g., x, y). Thus, for example, at 7 ( Figure 4 ) is the user's location (e.g., x, y), and the verified result reported at 9 ( FIG. 9 ) is the user's verified x, y location. And, at 10 ( Figure 4 ) uses the verification result at the (x,y) position.

[0064] Figure 5 An example of a process that may be performed at a validator entity according to some embodiments is depicted.

[0065] At 550, according to some embodiments, the verifier entity may receive a configuration for at least marking at least one radio signal measurement from the first network node. For example, the verifier entity (such as the verifier entity 154) may receive the configuration as an indication of whether to mark all or part of the at least one radio signal measurement. For example, referring to Figures 2 to 4, at 3, the configuration may indicate whether the radio signal measurements (such as LDM or other types of measurements) provided to the verifier entity are tagged with results so that the set of radio signal measurements and the results can be used for at least one primary task (PT) of training the ML model. Alternatively or additionally, the configuration may indicate a configuration, such as a format, for receiving information indicating the results. Alternatively or additionally, the configuration may include information about at least an auxiliary task, such as an indication of which auxiliary task the verifier entity should use when verifying the results. Alternatively or additionally, the configuration may include a configuration for transmitting a set including a label and at least one radio signal measurement to, for example, a trainer entity, for example, to enable the trainer entity to use the set to train the PT of the ML model. Alternatively or additionally, the configuration may include a list of network nodes, such as user entities or other types of entities, which the verifier entity should (or is allowed to) receive the results and associated radio signal measurements for verification using at least one auxiliary task. Alternatively or additionally, the first network node may include a trainer entity, which may be associated with the primary task of training the ML model and / or providing training data for the training. Alternatively or additionally, the authenticator entity may be comprised in or including the gNB base station.

[0066] At 552, according to some embodiments, the validator entity may receive from the second network node information indicating at least one result of an inference of a machine learning model of a primary task (PT), the primary task being associated with at least one radio signal measurement. Figure 1 In 106C, Figure 2 In 6 places and Figure 3 to Figure 4 In the example at 7, a verifier entity (such as verifier entity 154) may receive an inference result of the primary task of the ML model given an associated input of at least one radio signal measurement. As described above, the result may be received together with the associated at least one radio signal measurement, or the result may be received together with an indication, such as a pointer, address, timestamp, or other indicator of the identity of the at least one radio signal measurement, to enable the verifier entity to access or obtain the at least one radio signal measurement. Likewise, the result may also be received as an indication (e.g., as a pointer, address, or other indicator of the identity of the result) to enable the verifier entity to access or obtain the result. Alternatively or additionally, the first network node may include a trainer entity, which may be associated with the primary task of training the ML model and / or providing training data for such training. The second network node may include a user entity (e.g., such as Figure 2 6) or a trainer entity (e.g., Figure 3 as shown in 7).

[0067] At 554, according to some embodiments, the validator entity may use the at least one result indicated in the received information to perform at least one auxiliary task (non-PT task). Figure 1 In 160, Figure 2 in 7 and Figure 3 to Figure 4 In the example of 8, a verifier entity such as verifier entity 154 can use at least the result (which is indicated or received at 552) to perform (e.g., execute, practice computer program code) at least one auxiliary task. To further illustrate, the at least one auxiliary task can be an ML task and / or a non-ML task, which can be used to verify (e.g., test) at least one radio signal measurement (e.g., the at least one radio signal measurement does cause the result of the PT inference of the ML model). Figure 2 In the example of 7, the auxiliary task includes selecting the best TX-RX beam pair (which can be identified by a beam index (BI)) between the corresponding signal source (TRP) and the UE using the results of at least one radio signal measurement of a given input.

[0068] At 556, according to some embodiments, the verifier entity may use at least one output of the at least one auxiliary task to determine a tag for the at least one radio signal measurement associated with the at least one result, or determine based on the output of the at least one auxiliary task that the at least one radio signal measurement associated with the at least one result is to be discarded. Furthermore, the at least one auxiliary task may include at least one task having an input of at least one of the result and / or the at least one radio signal measurement. Again referring to Figure 2 In the example at 7, the verifier may use the auxiliary task to determine a label by, for example, mapping the result to at least one radio signal measurement. For example, if the selected TX-RX beam pair (which is selected using the result of the at least one radio signal measurement and the ML model) matches the expected beam pair determined by using the auxiliary task, the at least one radio signal measurement is marked with a LOS result. Alternatively or additionally, the result may be modified so that (or, for example, until) the output of at least one auxiliary task using at least the modified result indicates a successful test (e.g., a match) of the modified result. As described above, for example, the verifier entity may iteratively (e.g., repeatedly) apply the selected function f to the result (or perform a search) to generate the modified result, and then verify the modified result via the auxiliary task. If the verification of the modified result is successful, the modified result is used as a label for the at least one radio signal measurement (which may be transmitted at 55 8). At least one auxiliary task may include at least one task having an input of at least one of the result and / or at least one radio signal measurement.

[0069] Alternatively or additionally, the at least one auxiliary task may include multiple auxiliary tasks used by the verifier entity. In this case, the verifier entity may use the multiple auxiliary tasks in combination to verify (e.g., cross-validate) the results to determine the signature of the at least one radio signal measurement.

[0070] Alternatively or additionally, in case the validator entity determines to discard at least one radio signal measurement, the validator entity may transmit the at least one radio signal measurement to another network node, such as the first network node or the second network node, to enable use in a semi-supervised training procedure.

[0071] At 558, the validator entity may transmit the tag and at least one radio signal measurement to the first network node or the second network node to train a machine learning model for the primary task. Figure 1 In the 170A example, Figures 2 to 3 In the example of 9 , the verifier entity 154 may transmit the at least one radio signal measurement tagged with the result.

[0072] Alternatively or additionally, the user entity may be comprised in or comprise a user device.Alternatively or additionally, the trainer entity may be comprised in or comprise a location management function.Alternatively or additionally, the main tasks of ML model inference may include location estimation, line of sight estimation / detection of signal sources, and / or beam selection.

[0073] Figure 6 A block diagram of a network node 600 according to some example embodiments is depicted. The network node 600 may include or be included in one or more network-side nodes or functions (e.g., gNB, eNB, DU, TRP, LMF, LMC, etc.).

[0074] According to some example embodiments, network node 600 may include a network interface 602, a processor 620, and a memory 604. Network interface 602 may include a wired and / or wireless transceiver to enable access to other nodes including base stations, other network nodes, the Internet, other networks, and / or other nodes. Memory 604 may include volatile and / or non-volatile memory including program code that, when executed by at least one processor 620, provides, among other things, the processes disclosed herein with respect to a trainer entity, a verifier, and the like.

[0075] Figure 7A block diagram of an apparatus 10 according to some example embodiments is shown. The apparatus 10 may include or be included in a user device, such as a user equipment (e.g., user entity, PRU, etc.). In general, various embodiments of the user equipment 204 may include a cellular phone such as a smart phone, a tablet computer, a personal digital assistant (PDA) with wireless communication capabilities, a portable computer with wireless communication capabilities, an image capture device with wireless communication capabilities such as a digital camera, a gaming device with wireless communication capabilities, a music storage and playback appliance with wireless communication capabilities, an Internet appliance that allows wireless Internet access and browsing, a tablet computer with wireless communication capabilities, and a portable unit or terminal incorporating a combination of such functions, and further for vehicles such as cars and / or trucks and aircraft such as manned or unmanned aircraft, and a portable unit and terminal incorporating a combination of such functions. The user equipment may include or be included in an IoT device, an Industrial IoT (IIoT) device, etc. In the case of an IoT device or an IToT device, for example, when compared to a smartphone, the UE may be configured to operate with fewer resources (e.g., in terms of power, processing speed, memory, etc.).

[0076] The device 10 may include at least one antenna 12 in communication with a transmitter 14 and a receiver 16. Alternatively, the transmitting antenna and the receiving antenna may be separate. The device 10 may also include a processor 20 configured to provide signals to the transmitter and receive signals from the receiver, respectively, and to control the operation of the device. The processor 20 may be configured to control the operation of the transmitter and the receiver by sending control signaling to the transmitter and the receiver via electrical conductors. Similarly, the processor 20 may be configured to control other elements of the device 10 by implementing control signaling via electrical conductors connecting the processor 20 to other elements (such as a display or memory). For example, the processor 20 may be embodied in a variety of ways, including circuitry, at least one processing core, one or more microprocessors with an accompanying digital signal processor, one or more processors without an accompanying digital signal processor, one or more coprocessors, one or more multi-core processors, one or more controllers, processing circuitry, one or more computers, various other processing elements including integrated circuits (e.g., application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc.), or some combination thereof. Therefore, although in Figure 7 20 is shown as a single processor, but in some example embodiments, processor 20 may include multiple processors or processing cores.

[0077] The apparatus 10 is capable of operating with one or more air interface standards, communication protocols, modulation types, access types, etc. The signals sent and received by the processor 20 may include signaling information in accordance with the air interface standard of the applicable cellular system and / or any number of different wired or wireless network technologies, including but not limited to Wi-Fi, wireless local area network (WLAN) technologies such as Institute of Electrical and Electronics Engineers (IEEE) 802.11, 802.16, 802.3, ADSL, DOCSIS, etc. In addition, the signals may include voice data, user-generated data, user-requested data, etc.

[0078] For example, the device 10 and / or the cellular modem therein can operate according to various first generation (1G) communication protocols, second generation (2G or 2.5G) communication protocols, third generation (3G) communication protocols, fourth generation (4G) communication protocols, fifth generation (5G) communication protocols, sixth generation (6G) communication protocols, Internet Protocol Multimedia Subsystem (IMS) communication protocols (e.g., Session Initiation Protocol (SIP)), etc. In addition, for example, the device 10 can operate according to 3G wireless communication protocols, such as Universal Mobile Telecommunications System (UMTS), Code Division Multiple Access 2000 (CDMA2000), Wideband Code Division Multiple Access (WCDMA), Time Division Synchronous Code Division Multiple Access (TD-SCDMA), etc. The device 10 can also operate according to 3.9G wireless communication protocols, such as Long Term Evolution (LTE), Evolved Universal Terrestrial Radio Access Network (E-UTRAN), etc. In addition, for example, the device 10 can operate according to 4G wireless communication protocols (such as Advanced LTE, 5G, etc.) and similar wireless communication protocols that may be developed later.

[0079] It should be understood that the processor 20 may include circuit devices for implementing the audio / video and logic functions of the device 10. For example, the processor 20 may include a digital signal processor device, a microprocessor device, an analog-to-digital converter, a digital-to-analog converter, etc. The control and signal processing functions of the device 10 may be allocated between these devices according to their respective capabilities. The processor 20 may also include an internal voice encoder (VC) 20a, an internal data modem (DM) 20b, etc. In addition, the processor 20 may include the function of operating one or more software programs, which may be stored in the memory. In general, the processor 20 and the stored software instructions may be configured to cause the device 10 to perform actions. For example, the processor 20 is capable of operating a connection program such as a web browser. The connection program may allow the device 10 to transmit and receive network content, such as location-based content, according to a protocol (such as a wireless application protocol WAP, a hypertext transfer protocol HTTP, etc.).

[0080] The device 10 may also include a user interface, including, for example, an earpiece or speaker 24, a ringer 22, a microphone 26, a display 28, a user input interface, etc., which may be operably coupled to the processor 20. As described above, the display 28 may include a touch-sensitive display, where a user may touch and / or gesture to make selections, enter values, etc. The processor 20 may also include a user interface circuit device configured to control at least some functions of one or more elements of the user interface, such as the speaker 24, the ringer 22, the microphone 26, the display 28, etc. The processor 20 and / or the user interface circuit device including the processor 20 may be configured to control one or more functions of one or more elements of the user interface through computer program instructions, such as software and / or firmware stored on a memory accessible to the processor 20, such as the volatile memory 40, the non-volatile memory 42, etc. The device 10 may include a battery for powering various circuits associated with the mobile terminal, for example, a circuit that provides mechanical vibration as a detectable output. The user input interface may include devices that allow the apparatus 20 to receive data, such as a keypad 30 (which may be a virtual keyboard presented on the display 28 or an externally coupled keyboard) and / or other input devices.

[0081] like Figure 7 As shown, the device 10 may also include one or more mechanisms for sharing and / or obtaining data. For example, the device 10 may include a short-range radio frequency (RF) transceiver and / or interrogator 64, so that data can be shared with and / or obtained from electronic devices based on RF technology. The device 10 may include other short-range transceivers, such as infrared (IR) transceivers 66, Bluetooth TM Wireless technology Bluetooth TM (BT) transceiver 68, wireless universal serial bus (USB) transceiver 70, Bluetooth TM Low energy transceiver, ZigBee transceiver, ANT transceiver, cellular device to device transceiver, wireless LAN link transceiver and / or any other short range radio technology. The device 10, and in particular the short range transceiver, can transmit data to and / or receive data from electronic devices in the vicinity of the device, for example within 10 meters. The device 10 including a Wi-Fi or wireless LAN modem can also transmit and / or receive data from electronic devices according to various wireless networking technologies, including 6LoWpan, Wi-Fi, Wi-Fi low power, WLAN technology such as IEEE 802.11 technology, IEEE802.15 technology, IEEE 802.16 technology, etc.

[0082] The device 10 may include a memory, such as a subscriber identity module (SIM) 38, a removable user identity module (R-UIM), an eUICC, a UICC, a U-SIM, etc., which may store information elements related to a mobile subscriber. In addition to the SIM, the device 10 may also include other removable and / or fixed memory. The device 10 may include a volatile memory 40 and / or a non-volatile memory 42. For example, the volatile memory 40 may include a random access memory (RAM), including dynamic and / or static RAM, on-chip or off-chip cache memory, etc. The non-volatile memory 42 may be embedded and / or removable, and may include, for example, a read-only memory, a flash memory, a magnetic storage device, such as a hard disk, a floppy disk drive, a tape, an optical drive and / or a medium, a non-volatile random access memory (NVRAM), etc. Similar to the volatile memory 40, the non-volatile memory 42 may include a cache area for temporarily storing data. At least a portion of the volatile and / or non-volatile memory may be embedded in the processor 20. The memory may store one or more software programs, instructions, pieces of information, data, etc., which may be used by the apparatus for performing the operations disclosed herein.

[0083] The memory may include an identifier capable of uniquely identifying the device 10, such as an International Mobile Equipment Identity (IMEI). The memory may include an identifier capable of uniquely identifying the device 10, such as an International Mobile Equipment Identity (IMEI). In an example embodiment, the processor 20 may be configured using computer code stored at the memory 40 and / or 42 to provide the operations disclosed herein with respect to a UE, such as a user entity.

[0084] Some of the embodiments disclosed herein may be implemented with software, hardware, application logic, or a combination of software, hardware, and application logic. For example, the software, application logic, and / or hardware may reside on the memory 40, the control device 20, or an electronic component. In some example embodiments, the application logic, software, or instruction set is maintained on any of a variety of conventional computer-readable media. In the context of this document, a "computer-readable storage medium" may be any non-transitory medium that can contain, store, communicate, propagate, or transmit instructions for use by or in conjunction with an instruction execution system, device, or apparatus (such as a computer or data processor circuit device); a computer-readable medium may include a non-transitory computer-readable storage medium, which may be any medium that can contain or store instructions for use by or in conjunction with an instruction execution system, device, or apparatus (such as a computer).

[0085] Without in any way limiting the scope, interpretation, or application of the appended claims, the technical effects of one or more of the example embodiments disclosed herein may include a process for increasing the amount of ML training data and a network-assisted self-labeling process for increasing the amount and / or diversity of training data for training ML models.

[0086] Depending on the desired configuration, the subject matter described herein may be embodied in a system, apparatus, method, and / or article. For example, the base station and user equipment (or one or more components thereof) and / or the processes described herein may be implemented using one or more of the following: a processor that executes program code, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an embedded processor, a field programmable gate array (FPGA), and / or a combination thereof. These different implementations may include implementations in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be dedicated or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to a storage system, at least one input device, and at least one output device. These computer programs (also referred to as programs, software, software applications, applications, components, program codes, or codes) include machine instructions for programmable processors and may be implemented in high-level procedures and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the term "computer-readable medium" refers to any computer program product, machine-readable medium, computer-readable storage medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions. Similarly, systems that may include a processor and a memory coupled to the processor are also described herein. The memory may include one or more programs that cause the processor to perform or implement one or more of the operations described herein.

[0087] Although several variations are described in detail above, other modifications or additions are possible. In particular, further features and / or variations may be provided in addition to those set forth herein. In addition, the above implementations may be directed to various combinations and subcombinations of the disclosed features and / or combinations and subcombinations of several further features disclosed above. Other embodiments may be within the scope of the appended claims.

[0088] If desired, the different functions discussed herein may be performed in a different order and / or simultaneously with each other. In addition, if desired, one or more of the above functions may be optional or may be combined. Although various aspects of some of the embodiments are listed in the independent claims, other aspects of some of the embodiments include other combinations of features from the described embodiments and / or dependent claims with features of the independent claims, and are not just combinations explicitly listed in the claims. It should also be noted herein that although example embodiments are described above, these descriptions should not be considered restrictive. On the contrary, several variations and modifications may be made without departing from the scope of some of the embodiments defined in the appended claims. Other embodiments may be within the scope of the appended claims. The term "based on" includes "at least based on". Unless otherwise indicated, the phrase "such as" is used to mean "for example".

Claims

1. A method, include: receiving, by the validator entity and from the first network node, a configuration for marking at least at least one radio signal measurement; receiving, by the validator entity and from a second network node, information indicating at least one result of inference of a machine learning model for a primary task, the primary task being associated with the at least one radio signal measurement; performing, by the validator entity, at least one auxiliary task using at least the at least one result indicated in the received information; determining, by the verifier entity and using at least one output of the at least one auxiliary task, a tag for the at least one radio signal measurement associated with the at least one result, or determining, by the validator entity and based on the at least one output of the at least one auxiliary task, that the at least radio signal measurement associated with the at least one result is to be discarded; as well as The tag and the at least one radio signal measurement are transmitted to the first network node or the second network node to update the machine learning model for the primary task.

2. The method according to claim 1, wherein the verification entity performs the include: Modify the at least one result so that the output of the at least one auxiliary task using at least the modified result indicates a successful test of the modified result using the at least one radio signal measurement, wherein the modified result serves as the label for the at least one radio signal measurement.

3. The method according to any one of claims 1 to 2, wherein the at least one auxiliary task comprises at least one machine learning task and / or at least one non-machine learning task.

4. The method according to any one of claims 1 to 3, wherein the at least one auxiliary task comprises at least one task using the at least one result and / or the at least one radio signal measurement as input.

5. The method according to any one of claims 1 to 4, wherein the at least one auxiliary task comprises a plurality of auxiliary tasks, and wherein the plurality of auxiliary tasks are used in combination by the verifier entity to verify the at least one result for determining the tag used for the at least one radio signal measurement.

6. The method according to any one of claims 1 to 5, wherein the configuration for marking at least at least one radio signal measurement further comprises at least one of the following: an indication of whether the at least one radio signal measurement is to be marked; ； configuration for receiving said information indicative of said result; information about the at least auxiliary task; and a configuration for transmitting to the trainer entity a set comprising the tag and the at least one radio signal measurement.

7. A method according to any one of claims 1 to 6, wherein the validator entity is included in a network node (the first network node or the second network node), a server, a host or a system controlled by a user device, or includes the network node (the first network node or the second network node), the server, the host or the system controlled by the user device.

8. The method according to any one of claims 1 to 7, wherein the at least one radio signal measurement is determined to be discarded by the verifier entity. include: The at least one radio signal measurement is transmitted to the first network node or the second network node for use in a semi-supervised training procedure.

9. The method according to any one of claims 1 to 8, wherein the main task is include: Position estimation, line of sight estimation / detection of signal sources, or beam selection.

10. A device, include: at least one processor; as well as at least one memory including computer program code, the at least one memory and the computer program code being configured to, with the at least one processor, cause the apparatus to at least: receiving, from the first network node, a configuration for marking at least at least one radio signal measurement; receiving, from a second network node, information indicating at least one result of inference of a machine learning model for a primary task, the primary task being associated with the at least one radio signal measurement; performing at least one auxiliary task using the at least one result indicated in the received information; using at least one output of the at least one auxiliary task to determine a label for the at least one radio signal measurement associated with the at least one result, or determining, based on the at least one output of the at least one auxiliary task, that the at least radio signal measurement associated with the at least one result is to be discarded; as well as The tag and the at least one radio signal measurement are transmitted to the first network node or the second network node to update the machine learning model for the primary task.

11. An apparatus according to claim 10, wherein the apparatus is further caused to at least: modify the at least one result so that the output indication of the at least one auxiliary task using at least the modified result indicates a successful test of the modified result using the at least one radio signal measurement, wherein the modified result is used as the label for the at least one radio signal measurement.

12. An apparatus according to any one of claims 10 to 11, wherein the at least one auxiliary task comprises at least one machine learning task and / or at least one non-machine learning task.

13. The apparatus according to any one of claims 10 to 12, wherein the at least one auxiliary task comprises at least one task using at least one of the at least one result or the at least one radio signal measurement as input.

14. An apparatus according to any one of claims 10 to 13, wherein the at least one auxiliary task comprises a plurality of auxiliary tasks, and wherein the plurality of auxiliary tasks are used in combination by the verifier entity to verify the result to determine the tag used for the at least one radio signal measurement.

15. The apparatus according to any one of claims 10 to 14, wherein the configuration for marking at least at least one radio signal measurement further comprises at least one of the following: an indication of whether the at least one radio signal measurement is to be marked; ； configuration for receiving said information indicative of said result; information about the at least auxiliary task; and a configuration for transmitting to the trainer entity a set comprising the tag and the at least one radio signal measurement.

16. An apparatus according to any one of claims 10 to 15, wherein the validator entity is included in a network node (the first network node or the second network node or the third network node), a server, a host or a system controlled by a user device, or includes the network node (the first network node or the second network node or the third network node), the server, the host or the system controlled by the user device.

17. The apparatus according to any one of claims 10 to 16, wherein the at least one radio signal measurement is to be discarded or include: The at least one radio signal measurement is transmitted to the first network node or the second network node for use in a semi-supervised training procedure.

18. An apparatus according to any one of claims 10 to 17, wherein the primary task comprises one of position estimation, line of sight estimation / detection of a signal source, or beam selection.

19. A device, include: Means for: receiving, by the validator entity and from the first network node, a configuration for at least marking at least one radio signal measurement; receiving, by the validator entity and from a second network node, information indicating a result of an inference of a machine learning model for a primary task, the primary task being associated with the at least one radio signal measurement; performing, by the validator entity, at least one auxiliary task using at least the result indicated in the received information; determining, by the verifier entity and using the output of the at least one auxiliary task, a tag for the at least one radio signal measurement associated with the result, or determining, by the validator entity and based on the output of the at least one auxiliary task, that the at least radio signal measurement associated with the result is to be discarded; as well as The tag and the at least one radio signal measurement are transmitted to the first network node or the second network node to update the machine learning model for the primary task.

20. The device according to claim 19, further comprising: include: Means for performing any of the functions according to any one of claims 2 to 9.

Images