On-device machine learning model testing

Abstract

The present invention proposes a system and method for testing on-device machine learning models, such as network models, based on access to normal input and output channels, where the structure of the model is designed such that the model will generate distinctly different outputs for a given set of test inputs only as long as its internal structure remains as desired. The model is made available for such testing by training the model to the appropriate structure and / or via model pre-training with training inputs that are designed to cause the model to generate distinctly different outputs when subsequently presented with a specific set of "test" inputs. If the model's structure remains within acceptable limits, the device will generate predictable outputs when tested. If not, the device is forced to revert to the pre-trained model.

Description

[Technical field]

[0001] The present invention relates to the field of embedded machine learning models (e.g., neural network models) and testing of those models on connected devices over communication networks, such as, but not limited to, fifth generation (5G) cellular communication systems. [Background technology]

[0002] Hardware-level machine learning (machine learning model) implementations are increasingly being developed across multiple fields. In imaging processes, examples include on-chip artificial intelligence (AI) image sensors, and various systems for outputting at least partially classified images from integrated imaging and neural network hardware. In networking, recent examples have shown neural networks running entirely on network interface cards. A common feature of these implementations is that the observed output is, at least to some extent, the result of inference rather than raw data. An increasingly diverse range of functions related to (public) communications networks are being handled by on-device neural network models running on terminal devices such as user equipment (UE) devices.

[0003] "Deep learning" is the name used for neural networks that consist of several layers ("stacked neural networks"). The layers are made up of nodes. Nodes are where the computations take place, loosely mimicking neurons in the human brain that fire when they receive enough stimulation. Nodes combine (e.g., multiply) input data with a set of coefficients or weights that either amplify or attenuate the input data, thus assigning it importance with respect to the task the algorithm is trying to learn. These input-weight products are summed, and then the sum is passed through the so-called activation function of the node to determine whether and to what extent the signal should proceed through the network to affect the final result, e.g., the act of classification. If the signal passes, the neuron is "activated".

[0004] Recent developments have focused on algorithmic approaches for resource allocation rather than machine learning based approaches, e.g., for device-to-device (D2D) communications (i.e., sidelink). As an example, 5G New Radio (NR) systems use both licensed and unlicensed spectrum for D2D communications, where UEs may perform resource allocation independent of next generation Node B (gNB) or other network control functions. This results in a requirement for UEs communicating on the sidelink to perform resource allocation functions and maintain compliance with relevant standards.

[0005] At a general level, on-device model training, which allows models to adapt to their observed usage context, offers significant performance advantages in several domains. A widely deployed application of on-device model training is federated learning, in which devices collaborate to train parts of a larger model without compromising privacy by not allowing a data sample to leave the device that captured it.

[0006] Despite the large performance gains offered, on-device model training significantly changes the structure of the deployed neural network over time in unpredictable ways, which can also be due to adversarial attacks, such as poisoning or adversarial attacks, which creates a (currently unmet) requirement for robust and frictionless model testing.

[0007] Moreover, testing and validation of such neural network models by an external observer becomes problematic, especially when embodied as or deeply integrated into embedded hardware functions or into the lowest layers of a communication stack, because only the physical input channels (or inputs close to the physical input channels) and the usual observed outputs are available (as opposed to software-level access to the model parameters), and testing should be designed to be "frictionless," i.e., not requiring user input or significant use of on-device processing, energy or other resources.

[0008] If a neural network model is used by a device to perform networking tasks that occur outside of the normal (wireless network) coverage area, compliance with the standard is affected if it is not tested periodically. Additionally, if the output of the model is itself used as input (and possibly training data) for another model, e.g., in federated learning, testing of the first model is required to avoid data corruption of the second model. Such periodic testing is inconvenient if the devices require performing additional tasks dedicated only to self-testing, rather than their normal inference tasks, since the device hardware is not optimized for testing, and wastes resources for widely deployed models. Summary of the Invention [Problem to be solved by the invention]

[0009] It is an object of the present invention to provide an effective solution for testing on-device neural network models. [Means for solving the problem]

[0010] This object is achieved by an apparatus as claimed in claim 1, a device as claimed in claim 12, a system as claimed in claim 13, a method as claimed in claim 15, and a computer program product.

[0011] According to a first aspect, there is provided an apparatus for remote testing of an on-device machine learning model (e.g., an on-device neural network model) operated by a device, comprising: training input generation means for designing training inputs configured to provide a predicted output of the device in response to the test inputs; a test unit for applying training input information and test input information to the device and obtaining output information generated by the device in response to the test input information; and a model evaluation means for comparing the obtained output information with the predicted output to evaluate the on-device machine learning model. An apparatus is provided comprising:

[0012] According to a second aspect, there is provided a method for remotely testing an on-device machine learning model (e.g., an on-device neural network model) operated by a device, the method comprising: designing training input information, the training input information configured to provide a predicted output of the device in response to the test input information; providing training input information and test input information to the device; obtaining output information generated by the device in response to test input information; comparing the obtained output information with a predicted output to evaluate the on-device machine learning model; A method is provided comprising:

[0013] According to a third aspect, there is provided a first device comprising an on-device machine learning model operated by the first device, the first device being configured to determine that training of the on-device machine learning model is required in response to reaching at least one of a set of criteria consisting of a new deployment of the first device or the on-device machine learning model, a new use of the on-device machine learning model, a carrier or owner policy requirement, and a predefined time elapsed since the last training, to signal a requirement for training to a second network device comprising an apparatus according to the first aspect via a standard communication protocol and to receive testing and / or training information from the second device.

[0014] According to a fourth aspect, there is provided a system comprising at least one first device according to the third aspect and at least one second device having an apparatus according to the first aspect.

[0015] Finally, according to a fifth aspect there is provided a computer program product comprising code means for generating the steps of the method of the second aspect when run on a processor of a device.

[0016] In an option of these aspects, the training information is based on input information derived from known usage of the on-device machine learning model and the type or class of the on-device machine learning model.

[0017] Thus, an on-device machine learning model (e.g., a neural network model) can be configured to generate a predicted output in response to a test input due to prior exposure to training input data. In other words, the training input information is selected to make the on-device machine learning model suitable for subsequent testing. Thus, the on-device machine learning model can be tested using only network-side inputs and outputs, without requiring software access to the model (whether the model is embedded in hardware, software, or a mixture thereof). Appropriate input information and / or parameters can be designed and distributed to act as "training" and / or "test" inputs for the on-device model given the model usage context and communication scenario.

[0018] Additionally, a policy can be developed to determine which devices require testing, e.g., triggered by a certain condition. An updated model can then be reported by the device under test if the output of the test system is positive, or at least has a given quality level.

[0019] Thus, a first entity is enabled to make a neural network model or other machine learning module running on a second entity subject to and provide external testing, even if the first entity only has access to the model's normal inputs and outputs.

[0020] Thus, model testing can be accomplished by accessing only normal inputs and outputs, and furthermore, minimal or no computation or software effort is required from the device under test since the device under test only needs to perform its normal operations or other functions, thus conserving resources on the device under test.

[0021] Furthermore, in some embodiments, the training data may be designed such that the response to testing gives at least some information (e.g., statistical information) about the structure of the entire model, not just the particular trajectory that triggered the observed output. In some embodiments, the output may include information about the internal state of the network, not just its activity. Thus, neural network models or other machine learning models operating on any data type may be tested with network-side inputs, or testing may be triggered remotely by the system, for example, by mixing their training data with network-accessible parameters.

[0022] The devices or network devices referred to in this specification are configured to perform functions other than networking functions. This applies to non-networking applications (e.g., cameras and projectors) or applications where a test unit is co-located with the device.

[0023] According to the first option, the test unit: The training input information is provided by transmitting it to the device and the output information is obtained by receiving it via a transceiver; or The training input information is provided by interfacing with a hardware detection unit on the device, and the training input information is obtained by analyzing the output of the hardware detection unit on the device.

[0024] Thus, on-device testing is controlled by a remote network device or a co-located connected device.

[0025] According to the first option or the second option combined with any of the above first to fifth aspects, a policy is deployed (e.g., by the device) that is triggered by at least one predetermined condition to determine which devices require testing. Thus, it can be ensured that on-device machine learning models are tested periodically to ensure proper operation of the device.

[0026] According to the third option combined with the first option or the second option or any of the above first to fifth aspects, a hardware level input is designed to trigger a certain state of the on-device machine learning model to cause the device to generate an output (e.g., a networking output) as output information. Thus, the on-device machine learning model can be tested via the device's normal hardware level input by only checking the device's generated output.

[0027] According to a fourth option, which may be combined with any of the first to third options or any of the first to fifth aspects, the test input information corresponds to at least a subset of the training input information, and the model evaluation is based on the accuracy of the response of the on-board machine learning model. This provides the advantage that no specific test input information is required and deviations or non-compliance of the model during operation can be verified by checking the accuracy of the response of at least a subset of the original training input information.

[0028] According to a fifth option, which may be combined with any of the first to fourth options or any of the above first to fifth aspects, an external input database is provided for storing test input information, training input information and input information, the input information being associated with a list of types of on-device machine learning models, and the test input information being associated with a predicted response of the network device. Thus, the required test input information and the predicted response of the device under test can be obtained by simply accessing the external input database using the type of the on-device machine learning model under test.

[0029] According to a sixth option, which may be combined with any of the first to fifth options or any of the above first to fifth aspects, the on-device machine learning model is made available for testing by applying model pre-training using a mixed data vocabulary that includes network-accessible parameters mixed with true training data of an intended functionality of the on-device machine learning model. Thus, testing of the on-device machine learning model may be achieved by checking the intended functionality in the output of the device under test.

[0030] According to a seventh option, which may be combined with any of the first to sixth options or any of the first to fifth aspects, a radio frequency control algorithm is provided that is capable of modifying at least one transmission characteristic of the transmission of a test transceiver for controlling an on-device machine learning model by using the test transceiver as networking hardware of a test unit. Thus, the on-device machine learning model may be tested by using the networking hardware to modify the transmission characteristic.

[0031] According to an eighth option, which may be combined with any of the first to seventh options or any of the first to fifth aspects, a test timing algorithm for determining when a device requires testing and a status database are provided in which the results of model tests and policies regarding actions to be taken for failed tests are stored, thereby ensuring that on-device machine learning models are periodically tested and left in a proper state of operation.

[0032] According to a ninth option, which may be combined with any of the first to eighth options or any of the first to fifth aspects, training of the on-device machine learning model is performed (e.g., by a training input generation means): Designing modifications to a portion of the known model inputs to cause one or more backdoor activations to be present in the on-device machine learning model; providing data tagging (e.g., radioactivity data tagging) by adding tagged data to the training set to enable statistical identification of specific training input information; Using the input information as test input information to enable testing by checking whether, within given limits, the same output is obtained as during the original training, and / or The device is configured to apply federated learning to train an on-device machine learning model.

[0033] Thus, various options for efficient (network-based) testing of on-device machine learning models may be provided for various applications. As will be explained later, in the context of the present invention, radioactive data tagging is intended to be understood as a process of modifying training inputs with tags, such that the presence of the tags causes persistent changes to the internal structure of the trained model (beyond those caused by untagged data), which can be later discovered by statistical analysis of the model output.

[0034] According to a tenth option, which may be combined with any of the first to ninth options or any of the first to fifth aspects, test input information or a command to trigger a local test is distributed (e.g., by a device) in a unicast, multicast or broadcast channel. Thus, efficient testing of on-device machine learning models on two or more (network) devices under test can be achieved via a single communication channel.

[0035] According to an eleventh option, which may be combined with any of the first to tenth options or any of the first to fifth aspects, an on-device machine learning model is distributed to the first device, and the test input information includes information used to create the on-device machine learning model and the predicted output. Thus, a pre-trained machine learning model may be distributed to the (network) device, such that efficient model testing may be achieved without a pre-training or pre-use stage.

[0036] It should be noted that the above apparatus may be implemented based on the configuration of a discrete hardware circuit, an integrated chip, or a chip module having discrete hardware components, or based on a signal processing device or chip controlled by software routines or programs stored in a memory, written to a computer readable medium, or downloaded from a network such as the Internet.

[0037] It is to be understood that the apparatus according to claim 1, the device according to claim 12, the system according to claim 13, the method according to claim 16 and the computer program product according to claim 17 have similar and / or identical preferred embodiments, in particular as defined in the dependent claims.

[0038] It is to be understood that a preferred embodiment of the invention can also be any combination with the dependent claims or the respective independent claims of the above embodiments.

[0039] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief description of the drawings]

[0040] [Figure 1] FIG. 1 is a schematic diagram illustrating a system architecture according to various embodiments. [Diagram 2] 1 is a schematic process flow diagram of a procedure for training and testing an on-device neural network model according to one embodiment. [Diagram 3] 1 is a schematic flow diagram of a neural network model testing process according to one embodiment. [Figure 4] 1 is a schematic flow diagram of a pre-use phase of a model testing procedure according to one embodiment. [Diagram 5] 1 is a schematic flow diagram of a usage phase of a model testing procedure according to one embodiment. [Figure 6] FIG. 2 is a schematic diagram illustrating a training phase according to one embodiment. [Figure 7] 1 is a schematic diagram illustrating stages of use according to one embodiment. [Figure 8] FIG. 1 is a schematic diagram illustrating a method for deploying a trained model after testing according to an embodiment of the present invention. [Figure 9] FIG. 13 is a schematic diagram illustrating another method for deploying a trained model after testing according to another embodiment of the present invention. [Figure 10] 1 is a schematic diagram illustrating stages of use according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] Here, an embodiment of the present invention will be described based on a cellular communication network environment such as 5G, however, the present invention can also be used in conjunction with other wireless technologies (e.g., IEEE 802.11 / Wi-Fi or IEEE 802.15.4 / Ultra Wideband (UWB)) in which AI model-enhanced wireless communication is provided or may be implemented.

[0042] Throughout this disclosure, the abbreviation "gNB" (5G terminology) or "BS" (base station) is intended to mean an access device such as a cellular base station or a WiFi access point or a UWB PAN coordinator. The gNB is composed of a centralized control plane unit (gNB-CU-CP), multiple centralized user plane units (gNB-CU-UP) and / or multiple distributed units (gNB-DU). The gNB is part of the radio access network (RAN) that provides an interface to functions in the core network (CN). The RAN is part of a wireless communication network. The RAN implements a radio access technology (RAT). Conceptually, the RAN resides between communication devices such as mobile phones, computers, or any remote-controlled machine and provides their connection to the CN. The CN is the backbone of the communication network that provides a number of services to customers interconnected through the RAN. More specifically, the CN directs communication streams through the communication network and possibly other networks.

[0043] Furthermore, the terms "base station" (BS) and "network" are used synonymously in this disclosure. This means, for example, that when a "network" is described as performing a particular operation, the operation is performed by a CN function of a wireless communication network or by one or more base stations that are part of such wireless communication network, and vice versa. This may also mean that some of the functions are performed by a CN function of a wireless communication network and some of the functions are performed by a base station.

[0044] Please note that throughout this disclosure, only blocks, components and / or devices related to the proposed data distribution functionality are shown in the accompanying drawings. Other blocks are omitted for reasons of brevity. Moreover, blocks designated by the same reference numbers are intended to have the same or at least similar functions, and therefore their functions will not be described again later.

[0045] Neural networks can be controlled from the outside by inserting and detecting so-called "backdoors". These backdoors generally consist of specific activations in the network that are known to be excited by inputs of known form, and the backdoors are usually entered by presenting the network with a small percentage of modified training inputs at training time. A useful feature of such techniques is that only a small percentage (e.g., 1%) of all normal training inputs need to be modified, and the accuracy of the network on its original task remains unchanged or nearly unchanged, yet can remain externally detectable during inference via the presentation of known inputs that cause characteristic changes to the output.

[0046] As a more detailed example, neural network structures can be externally structured by "radioactive data tagging", which allows models containing specific training inputs to be identified with high statistical certainty. A useful feature of this technique is that it provides information about whether a particular class of training input influences the network's inference on a given set of inputs. Further details can be found, for example, in Sablayrolles et al., "Radioactive data: tracing through training".

[0047] On-device model training, such as federated learning, offers significant performance gains, especially for distributed hardware such as UEs or gNBs. In federated learning, each hardware unit has a model that evolves independently. Each individual hardware unit may report its individual model to a management unit, which may aggregate the individual models into a new combined model that encompasses the learning of all participating hardware units, which is again redistributed to the hardware units. However, on-device models will deviate unpredictably over time if allowed to train independently, and thus require testing.

[0048] The following embodiments enable the implementation of a network-based test system for an on-board neural network model provided in a terminal device (e.g., UE) of a wireless communication system (e.g., 5G or other cellular or WiFi or UWB communication system), where the on-device neural network model (embedded in hardware or inaccessible) can be made subject to external testing by accessing only the normal input and output channels. This can be achieved by designing the model structure through training such that the model produces distinctly different outputs for a given set of test inputs, as long as its internal structure remains in a desired state.

[0049] Neural network models can be made available for testing via model pre-training using a mixed data vocabulary that includes network-accessible parameters mixed with true training data for the model's intended function. The "training" inputs are designed to train the model into a suitable structure and cause it to produce distinctly different outputs (e.g., specific misclassifications) when later presented with a particular set of "test" inputs.

[0050] In some instances, the training inputs are selected to produce results where the model still relies on the original training inputs for inference, while in other instances a particular class is determined where inference results consistent with this class indicate an acceptable level of model stability. This pertains to "radioactive tagging" embodiments where the tagged data is in the training set.

[0051] If the model structure remains within acceptable limits, the device will produce predictable outputs when tested, otherwise the network is forced to revert (e.g., reset) to the pre-trained model.

[0052] The test and training inputs take the same physical form and are provided by network-accessible behavior, such as deliberate resource selection in time-frequency space.

[0053] FIG. 1 illustrates a schematic of a system architecture according to various embodiments.

[0054] The system includes a device under test (DUT) 30, i.e., a device on which a neural network model (DUT mode) runs, which may require testing where direct software-side access to the model is limited in some manner (e.g., by hardware architecture or resource constraints) and which generates networking signals for communication over a publicly licensed telecommunications network (e.g., a 5G network). Exemplary types of DUTs 30 include, in various embodiments, UEs (such as smartphones, Internet of Things (IoT) nodes, connected cameras, connected microphones, connected medical devices, etc.), networking equipment (routers, gateways, repeaters, etc.), or network infrastructure equipment (e.g., a base station (e.g., a next generation Node B (gNB) or another type of access device)).

[0055] Inputs to the DUT model of DUT 30 may be provided via the hardware of DUT 30 itself (i.e., hardware-level inputs) instead of algorithmically via software or computer code (i.e., software-level inputs). Hardware-level inputs are therefore distinct from inputs that are generated entirely in software, such that they may traverse the normal input chain of DUT 30 (e.g., RF front end → end → neural network) to generate a response.

[0056] In one example, a DUT model running on a networking device (e.g., network infrastructure equipment or UE) performs networking tasks such as resource allocation (e.g., providing specific parameters in the radio frequency (RF) domain, including, for example, the selection of a specific frequency channel for a specific period of time), as observed by the DUT 30 via its normal RF hardware.

[0057] In another example, a DUT model running on a networking device performs positioning tasks such as determining the location of the device based on received wireless signals observed by the DUT 30 through its conventional RF hardware.

[0058] In another example, a DUT model running on the connected device performs an image recognition task, such as determining the presence of a pattern based on video images captured by the DUT 30 via its conventional video hardware.

[0059] In another example, a DUT model running on a connected medical device performs a recognition task aimed at detecting a given medical condition or health-related event, such as arrhythmia or a fall, based on physiological vital signs via hardware sensing inputs adapted by the DUT 30.

[0060] Furthermore, the system comprises a signal design system (SDS) 10 configured to design or create suitable training and / or test inputs that can be provided from one entity in the network (e.g., gNB) to train and then test a neural network model running on another entity (e.g., UE). The design of the training inputs depends on the type of embodiment. They can be, for example, "radioactive tagging" inputs as described in Sablayrolles et al., "Radioactive data: tracing through training."

[0061] The signal design system 10 comprises as a subcomponent a training input design system (Tr-IDS) 102 for designing hardware level inputs (training inputs) for model training and state manipulation. The training input design system 102 is itself embodied as a neural network, which operates on suitable data, e.g. known (normal) model inputs 60, which are responsible for the normal functioning of the non-test model (e.g. following predefined criteria / limits and / or generating known outputs of the model), and generates slight modifications to the data. An example of a suitable data type for the known model inputs 60 are RF channel characteristics that are predicted to be typical model inputs for a DUT model performing an RF resource allocation task.

[0062] In one example, the DUT model recognizes the selection of a particular frequency channel by a communication partner for a particular period of time as a known model input, which may also refer to the selection of any other communication parameters, such as channel decoding (e.g., by using low-density parity-check codes (LDCP)), massive multiple-input multiple-output (MIMO), non-orthogonal multiple access, resource allocation, or security, as described, for example, in Amanda Ly et al., “A Review of Deep Learning in 5G Research: Channel Coding, Massive MIMO, Multiple Access, Resource Allocation, and Network Security.”

[0063] The training input design system 102 operates either directly on the known model inputs 60 or on a transformed representation such as a scalogram (i.e., a visual representation of the wavelet transform with axes of time, scale and coefficient values, similar to a spectrogram). The training input design system 102 also includes a first feature extraction stage in which some representative features of the received inputs are extracted.

[0064] Neural networks embodying the training input design system 102 may employ, for example, “radioactive” data markers, where small modifications to a training set cause particular statistical patterns in the resulting output (e.g., as described in Sablayrolles et al., “Radioactive data: tracing through training”), or adversarial input generators and “backdoor” insertion techniques, where spurious activations are trained into the model using small perturbations to the training set (e.g., as described in Ilia Shumailov et al., “Towards Certifiable Adversarial Sample Detection” or Liao, C et al., “Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbation”), or federated learning techniques to allow training inputs to be designed based on locally stored data (e.g., as described in H. Brendan McMahan et al., “Communication-Efficient Learning of Deep Networks from Decentralized Data” or Yawen Wu et al., “Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Filtering”). The network is structured according to machine learning techniques for neural networks, including those described in “NLP Pruning.”

[0065] In alternative embodiments, the training input design system 102 may be embodied as a fixed non-neural network algorithm or a manual process. In one example, a fixed correction may be included algorithmically or even manually in every scalogram "image" to cause a predictable drop in classification accuracy for a particular class by hiding a known percentage of the data. This may then be externally tested. A very simple example is, say, 25% of the image is simply occluded, and classification using known ML techniques drops by some percentage that can first be measured and then used as the predicted output. This level is measured and modified across multiple classes by occluding more or less parts of the image.

[0066] Additionally, the signal design system 10 includes, as a sub-component, a test input design system (Te-IDS) 104 for designing hardware-level inputs (test inputs) that trigger specific DUT model states to cause the DUT 30 to generate networking outputs (DUT responses).

[0067] If the test data corresponds to the training data itself and a check is performed on the accuracy of the DUT model or the predicted output for a given training sample, the functionality of the test input design system 104 is covered by the training input design system 102. Thus, the training inputs can be used as test inputs.

[0068] An embodiment of the test input design system 104 can be similar to an embodiment of the training input design system, although the test inputs need not resemble known expected inputs.

[0069] Optionally, a subset of the training inputs may be used as test inputs.

[0070] Moreover, the signal design system 10 comprises an external input database (eIDB) 106, which can be accessed by all other components of the system, and in which at least one of test inputs, training inputs, known model inputs indexed to a list of DUT model types, and test inputs indexed to expected DUT responses can be stored. Different subcomponents of the signal design system operate on different network devices or systems, for example, the Tr-IDS 102 operates on a network infrastructure device with training capabilities (e.g., in the factory where the DUTs are manufactured) and the (Te-IDS) 104 operates in the core network functions of a wireless communication system.

[0071] Additionally, the system of FIG. 1 includes a model communication system (MCS) 20 for communicating models and / or test inputs and / or training inputs to the DUT 30 or another DUT, and for receiving DUT responses and other inputs from the DUT 30.

[0072] The model communication system 20 comprises a radio frequency control algorithm (RF-CA) 202 for controlling the ML system in the DUT 30 by using a test unit 204 (e.g., a test transceiver (Te-TRX) as networking hardware capable of varying inputs such as time, frequency, coding and / or other characteristics of the transmission of the test transceiver).

[0073] Suitable implementations of the test transceiver 204 include standard wireless or wired networking infrastructure such as a gNB, a gateway, a router, a UE device, etc.

[0074] In an alternative embodiment, the DUT 30 acts as a test transceiver for another DUT, in which case the DUT 30 interfaces with the RF control algorithm 202 via the data plane. This requires that each DUT has a copy of its test inputs. If the DUT 30 needs to be tested by another DUT, the DUT 30 transmits its test data to the other DUT that requires the test service. The other DUT then transmits its test inputs to the DUT 30 through the data plane of the communication network. Thereby, distributed operation of the proposed model test system can be achieved.

[0075] Optionally, a match between the expected output and the generated output is determined by another DUT by accessing or receiving data stored in the external input database 106.

[0076] Additionally, the model communication system 20 includes a Test Timing Algorithm (Te-TA) 206 for determining when a DUT requires testing, and a DUT Status Database (DUT-SDB) 210 in which policies regarding the results of DUT testing and actions to be taken for failed tests are stored. The policies may be defined centrally, for example, by a networking standard, or may be set by a network operator or similar entity. Furthermore, policies may be deployed in response to test failures via the network sending a status flag or other indication to the DUT 30.

[0077] Additionally, the model communication system 20 includes a policy enforcement module (PEM) 208 that enforces a policy from a DUT status database 210 when the DUT fails a test. The policy enforcement module 208 communicates with the DUT 30 via the test transceiver 204 or otherwise over a broader communication network, such as a 5G network.

[0078] As a further option, the system of FIG. 1 includes at least one of a third party device (3PD) 40 and a second order neural network model (2MD) 50.

[0079] The third party device 40 is a terminal device (e.g., a UE such as a smartphone) that provides data inputs representing the known model inputs 60 to the signal design system 10 over a network. In one example, this is a UE on the network that communicates with the DUT 30 as part of its normal functionality. The third party device 40 communicates with the network via the model communication system 20 and with the DUT 30 by normal network (e.g., 5G) communications.

[0080] The second order neural network model 50 runs on a device, such as a gNB or other access device, that receives training input from the output of the DUT model of the DUT 30. The second order neural network model 50 communicates with the network via the model communication system 20.

[0081] As a further option, the system of Fig. 1 is such that the DUT 30 and the model communication system 20 are co-located on the same device. The interface between the DUT 30 and the model communication system 20 is a hardware interface rather than a networking interface, so that the model communication system 20 can provide appropriate hardware-based inputs to the DUT 30, for example if the DUT 30 is a connected device such as a camera, microphone or connected medical device. In this case, the test unit 204 is configured to provide training input information by interfacing with a hardware sensing unit on the connected device (as described in relation to Fig. 3 below) and to obtain output information by analyzing the output of the hardware sensing unit on the connected DUT 30.

[0082] In the following, the interaction between the components of the system of FIG. 1 for testing an on-board model will be explained with reference to FIG.

[0083] Figure 2 shows a schematic process flow diagram of a procedure for neural network model training and testing according to one embodiment. In Figure 2, components / blocks of the architecture of Figure 1 involved in the procedure are shown but will not be described again.

[0084] The process is designed to enable testing of neural networks in scenarios where testing personnel or functions have limited access to the known inputs and outputs of the DUT 30 (e.g., due to hardware implementation of the DUT model, as described above, or resource constraints), such that, for example, if a first entity desires to train and / or test a neural network model running on a second entity, the training can be performed over a public telecommunications network (e.g., a 5G network) through specification of a modified training procedure.

[0085] After the initiation of the procedure (indicated in FIG. 2 by the top left block “STA”), a pre-use phase begins in which the signal design system 10 is used to design or create test inputs (Te-I) and / or training inputs (Tr-I) for a particular DUT model or type of DUT 30. To accomplish this, the signal design system 10 accesses known model inputs 60 for the DUT model type (DUT-T) from the external input database 106.

[0086] Thus, the training input design system 102 designs the training inputs using known model inputs 60 of the DUT 30 accessed from an external input database 106. The known model inputs 60 are typically derived from the DUT operating environment (NOE). Several known machine learning techniques, such as radioactive data tags or adversarial input generators, are suitable for generating modifications to the input signals. Depending on the technique used, the data on which the operation is performed is either directly the input signal or an image representation of the input signal.

[0087] In one example, the training input design system 102 operates directly on signal data representing known model inputs 60. One example is the design of vector operations that can be applied to the raw data that cause features extracted from the data to be "moved" in predictable directions within a suitable representation space (e.g., as described in Sablayrolles et al., "Radioactive data: tracing through training").

[0088] A model is thereby created having radioactive data markers so that, later, when the model is fed with the designed test data, a predetermined model output is derived as the answer.

[0089] Additionally, the test input design system 104 designs test inputs that match the form of the training inputs so that they provide the desired test functionality according to the modified training that the DUT model received.

[0090] The training inputs and the matched test inputs are then stored in the external input database 106 indexed to the associated type of DUT model and the system is ready for use.

[0091] A subsequent usage phase begins whereby the DUT 30 indicates to the network, via the model communication system 20, that training (or a suitable pre-trained model) is required.

[0092] More specifically, the DUT 30 determines that training or testing is required due to reaching one of several conditions including, for example, at least one of a new development of the DUT or DUT model, a new use for an existing DUT model, a carrier or owner policy requirement, or the amount of time elapsed since the last training.

[0093] The DUT 30 signals the requirement for training (or trained model) or testing to the network via standard communication protocols, e.g., via a training-needed flag (Tr-F) or test-needed flag (Te-F), e.g., to the RF control algorithm 202 of the model communication system 20. Additionally, the DUT 30 signals a specific DUT identity (DUT-ID) to the test timing algorithm 206. Alternatively, the requirement for training or testing is triggered by an internal timer of the model communication system 20 or the signal design system 10 (e.g., based on information that a certain maximum time has elapsed since the last training), or by a trigger received from a core network function (e.g., a network data analysis function (NWDAF)) or an application function indicating that (re)training is required.

[0094] In response, the network provides training inputs (or a trained model) to the DUT 30 via the test transceiver 204 of the model communication system 20 .

[0095] The model communications system 20 provides the training inputs by using an RF control algorithm 202 that retrieves the required test inputs from the external input database 106 and modifies a test transceiver 204 to provide the required RF characteristics to the DUT 30 .

[0096] The DUT 30 signals, via the model communication system 20, that it has received the training input.

[0097] The network tests the correct reception and application of the training inputs by sending one or more test inputs immediately after training or model deployment. This testing phase can be performed multiple times, triggered by the network or by the DUT 30.

[0098] The network provides test inputs to the DUT 30 via the test transceiver 204 in the same manner as the training inputs.

[0099] In response, a DUT response (DUT-RESP) is received via an action of the DUT 30 (eg, selection of a particular channel, etc.), which is read by the model communication system 20 via, for example, the test transceiver 204.

[0100] The received DUT responses are compared to those expected given the previously transmitted test inputs by referencing the external input database 106. If the DUT responses match the expected responses, the DUT 30 is considered ready for use and its status (DUT-S) is logged in the DUT status database 210. If there is a mismatch, the network provides new training inputs (or a new trained model) to the DUT 30 via the model communication system 20.

[0101] The DUT 30 is used normally, and the DUT model collects additional known model inputs through its use in a normal operating environment. At intervals, such as when the DUT 30 is within range of a network (e.g., a gNB cell) or according to a policy (P), the test timing algorithm 206 can be used to determine whether testing is required. The test timing is determined based on several factors in various embodiments (as described below) and / or the test timing is determined based on the state of multiple DUTs.

[0102] For example, in a large network, it may be more efficient to distribute test data or commands to trigger local tests in a multicast or broadcast channel, which may be an option to minimize communication overhead or to speed up proper operation of all devices.

[0103] If the test timing algorithm determines that a test is required, the network transmits the test inputs as described above. If the test passes, the DUT status is updated in the DUT status database 210. If the DUT 30 performs a local test, the DUT 30 only reports a test fail or pass message depending on the previously deployed policy. If the test fails, the model communication system 20 retrieves the corresponding policy (P) from the DUT status database 210 and enforces the policy via the policy enforcement module 208 that enforces the policy (EP) on the DUT 30 and / or the procedure ends (END).

[0104] Optionally, according to a pre-deployment policy, the DUT 30 may need to upload the locally trained model to the system only if it passes the test according to certain criteria (e.g., the number of false positives or false negatives when classifying a certain data pattern is not higher than a threshold), thereby reducing communication overhead, e.g., for federated learning, when the purpose of uploading the locally trained model includes, e.g., improving the global model.

[0105] In another embodiment, additional entities are provided that enable collection of known model input data and / or provision of test and / or training inputs from one DUT or another non-directly controlled UE to a second DUT (rather than sending tests directly from the network to the DUT), whereby non-network-controlled model testing (e.g., UE-to-UE testing) can be achieved.

[0106] In a further embodiment, the test procedure is modified to support the use of device model outputs for training a second model (e.g., for localization use cases).

[0107] In a further embodiment, to ensure that the DUT does not accept training and / or test inputs and / or policy inputs and / or responses to transmit its locally trained model from any device, the DUT verifies whether the model communication system 20 is authorized (by the network) to perform such a task. For this purpose, the DUT is provided with authentication information to verify that the input / message comes from a legitimate source, or the DUT requests the network to confirm that the input / message comes from a legitimate source.

[0108] FIG. 3 illustrates a flow diagram of a neural network model testing process according to one embodiment.

[0109] In step S301, a new device is deployed that includes neural network functionality (with limited access to an on-device copy of the model, e.g., due to being embodied as dedicated hardware or resource or other constraints). Alternatively, an existing device is upgraded to a new model version. The device then indicates its requirement for training and / or demand for pre-trained model weights to the network (e.g., a serving access device), e.g., by sending a training request (Tr-REQ). In another example, the indication of its requirement for training by the device is enforced based on public standards or system policies.

[0110] In response to the instruction, the network provides the device with a current set of model weights, or in an embodiment, training inputs (RX M-Wt / Tr-I) associated with its intended model use, in step S302. These weights or training inputs are calculated, for example, for a given device or class of devices and stored in a network-side database. Or, in another example, the training inputs or model weights have been modified prior to transmission so that they are known to induce the existence of a particular condition when they are received, and are made subject to external testing.

[0111] When model weights (rather than raw training inputs) are broadcast, these weights have themselves been derived in the training process using training inputs that have been modified in the same way as the raw training inputs.

[0112] The training inputs or model weights need not represent all of the training data the model receives, as long as they cover the important functions to a sufficient degree to allow for the desired testing function.

[0113] In a subsequent step S303, the device indicates to the network that the model weights have been received or that training is complete (Tr-CPL), the latter being determined algorithmically, for example, by calculation of a loss function and its progress towards an optimum or plateau.

[0114] Then, in step S304, the network or another device or the device itself provides a test input (RX Te-I) to verify that the model has been trained correctly. In one example, the test input (i.e., test input data) is similarly calculated and stored in the network database as shown above.

[0115] In step S305, a test input causes the device to generate a test output (Te-O), which is measured by the network or another device or the device itself and then, in step S306, compared to an expected value derived from a database (CP Te-O / Exp-V).

[0116] In step S713, it is checked whether the test output matches the expected value (Te-O=Exp-V?).

[0117] If no response (test output) is received by the network or an incorrect response (test output) is received and no match is determined in step S307, the procedure branches to step S308 where a reset (RST) of the device is performed, for example by rolling the device back to a deployed version of the model or another predefined model configuration.

[0118] Otherwise, if the test outputs match the expected values, the procedure continues with step S309 and the model is considered to be developed at this point (MD DEPL). The device stores this initial calibration or communicates its structure to the network.

[0119] Thereafter, in step S310, the device performs its normal function (NF) and the model collects further data from its use over time to act as additional training data.

[0120] In an interrupt step S311 (e.g., triggered by a timer function or other type of trigger function), normal function is interrupted at a selected point in time defined by a trigger function, e.g., by network or policy, and the device is retested and reevaluated by returning to step S304 when the trigger is activated and repeating steps S304 to S310.

[0121] According to an alternative, more comprehensive embodiment, the neural network model is distributed to the device. In this way, the model does not need to be trained in the device. In this case, the signaled input may include the model (training data) and any data used in creating the predicted output. Then, at a time selected, for example, by the network or policy, the device retests itself by checking that the test output of the distributed neural network model for a given test input matches the predicted output within the expected association.

[0122] This approach of the above general embodiment may be used initially, and then augmented by an additional step, for example, where devices are provided with training materials so that they create models themselves (fitting the current description). The network unicasts or multicasts or broadcasts the training data and prediction output to test the local trained model. Alternatively, in machine learning methods such as federated learning (see, for example, section 7 of 3GPP specification TR22.874), the local model is uploaded to the system with which it is combined. In addition, a step that may be included is the distribution of a policy stating that the local trained model should only be uploaded if the test phase achieves a minimum performance level.

[0123] Thereby, the proposed system may enable the on-device model to undergo external testing (eg testing by a second entity, which may be a network) or self-testing.

[0124] For example, in the context of devices performing network security and some network monitoring, it is only required that the second device generates a message sequence that triggers a given response by the DUT. However, in other applications, e.g., applications using cameras as an example, this may not be so simple. Here, a projector may be used and a camera may record the image generated by the projector. For such applications and devices such as cameras, the device may include additional sensing hardware (e.g., hardware sensing unit) that mimics the signal otherwise generated by the camera sensor (e.g., CMOS sensor). For example, a deep learning accelerator may be used as hardware acceleration for the computation of deep convolutional neural networks, e.g., in embedded systems. The deep learning accelerator is adapted to calculate the inference / prediction of all layers of the deep neural network. Within a single layer, the input data may be computed in parallel by multiple computation units. This only requires the model (e.g., deep neural network architecture, parameters, scheduling) binarized by the deep learning optimizer and the input image provided by the camera sensor. In such deep learning accelerators, the neural network may be deeply embedded in the hardware and may develop locally based on the data being processed. To test the state of the neural network, it is necessary to include test hardware in the deep learning accelerator that is capable of acquiring test signals and creating inputs similar to those provided by the camera sensor. For example, if the camera sensor contains N×M sensor units, each of which has a given output, such as a given voltage from 0 to Vmax, the test hardware should acquire a test signal, such as an image of N'×N' pixels, and convert it into N×M input voltages corresponding to each of the N×M sensors of the camera sensor.For example, if a pixel in the test image can have any graytone g that can take any value from white (1) to black (0), then a given value g of a given pixel can be mapped to a voltage value g*Vmax of N / N'×M / M' sensor units.

[0125] In the following, more detailed embodiments of the pre-use and specification stages described above in relation to FIG. 2 will be explained on the basis of specific examples with reference to FIGS. 4 and 5, respectively.

[0126] FIG. 4 illustrates a flow diagram of the pre-use phase of a model testing procedure according to one embodiment.

[0127] As already mentioned above, in the pre-use stage, the signal design system is used to design test inputs and / or training inputs for a particular DUT model or DUT type.

[0128] To achieve this, in step S401, the model type and form of the known model input is accessed from a database (MD-T / F).

[0129] More specifically, the signal design system accesses known model inputs 60 for the DUT model type from an external input database.

[0130] Then, in step S402, the training input design system of the signal design system designs training inputs (Tr-I) using known model inputs of the DUT accessed from an external input database, for example, via slight modifications to the known model inputs that cause predictable changes to the structure. Several known machine learning techniques, such as radioactive data tags or adversarial input generators, are suitable for generating modifications to the signal data. Depending on the technique used, the data on which the operation is performed is either directly the signal or an image representation of the signal.

[0131] In one example, the training input design system operates directly on signal data representing known model inputs. One example is the design of vector operations that can be applied to the raw data that "move" features extracted from the data in a predictable direction within a suitable representation space (e.g., as described in Sablayrolles et al., "Radioactive data: tracing through training"). In this case, the data flow can be: signal → features → watermark generation → modified features → modified signal.

[0132] Optionally, in some cases it is advantageous to operate on image data, e.g. time series RF signal data or image data or medical data, rather than on the signal data itself. This is the case, for example, when watermarking techniques that rely on extensive image transformations are used. In this case, the known model inputs are converted by the signal design system into image form (e.g. as scalograms), resulting in a data flow: signal → image representation of signal → features from image → watermark generation → watermarked image → modified signal (extracted by reversing the image generation means steps).

[0133] In one embodiment, the training inputs are designed by using the "radioactive" data marker approach as described above, which operates by creating slight modifications of some of the features that represent the known model inputs. In this case, the training inputs resemble the known model inputs with appropriate slight modifications that allow a predictable statistical variation of the loss function to be assigned by the DUT model to a certain class of test inputs, so that when presented with several test inputs within this class, the output statistics of the DUT model provide information on the use of these training inputs for inference. The test inputs in this case are very similar to the known model inputs used for normal operation.

[0134] Thus, the idea is to create a model with radioactive data markers such that later, when the model is fed test data, a predetermined answer is derived.

[0135] In the specific case of the radioactivity approach, a test failure results in a change in the model's output statistics after several tests, making it clear that the original network-fed training data is no longer being used, or is not being used primarily for inference. The precise threshold is application dependent, and also depends, for example, on the proportion of the training set that is radioactively "tagged".

[0136] The advantage of this approach is that the DUT response provides information about the entire DUT model. For example, if there are multiple classes, it is possible to "tag" instances of all classes during training and test whether the model still uses the training data of all classes using one test sequence. This is based on the assumption that if the model still uses the prescribed training data, the output remains correct. This is suitable for DUT types where test access is less time or resource constrained, for example, smartphones that are often within range of an access device (e.g., gNB).

[0137] In a further embodiment, the training inputs are designed by using a "backdoor" insertion technique (e.g., those in Liao, C. et al., "Backdoor Embedding in Convolutional Neural Network Models via Invisible Perturbation," or related literature) to design subtle modifications to a small percentage of the known model inputs to cause one or more backdoor activations to be present in the DUT model. In this case, the training inputs resemble the known model inputs with appropriate subtle modifications to cause a high classification probability to be assigned by the network to a particular matched test input.

[0138] The matched test inputs are designed to be unlikely or impossible to observe in normal DUT model usage. For example, the test inputs may include RF parameter settings by the tester that are reserved in the standard for this purpose and therefore are not otherwise seen by the DUT, so that the DUT never encounters the test inputs during normal usage. For example, the test inputs may indicate vital signs that are not achievable in a healthy person, such as a heart rate of 400. Thereby, only one test input needs to be displayed during the test time, which is more suitable for DUT models with more limited test access, such as when the DUT is an IoT node whose resources are constrained.

[0139] In a further embodiment, the training inputs are designed by global modifications to known model inputs to train specific "classes" recognized by the DUT model as inputs to which it responds with a predictable output. This approach is suitable for low complexity training input design systems, since in this case there is no need to calculate the appropriate subtle modifications required in the other embodiments.

[0140] In each of the above embodiments, the test input design system designs test inputs (Te-I) in step S403 that match the form of the training inputs so that they provide the desired test functionality according to the modified training received by the DUT model.

[0141] In examples, the test inputs are matched to the shape of the predicted DUT model states after being trained on the training inputs, or the test inputs consist of instances of spurious "classes" that the DUT model is trained to recognize.

[0142] In a further embodiment, the training and testing inputs are designed by using known model inputs. The testing is then performed by checking that the model output provides the same output as during the original training within a given range. If the range deviates by more than a threshold, the model being updated based on the local training is determined to be invalid. In one example, the predicted output may refer to an estimated channel quality indicator (CQI) value or a classification for a given networking pattern.

[0143] In the final step S404 ((Tr-I / Te-I→DB), the acquired training inputs and the matched test inputs are stored in an external input database indexed to the DUT model type and the system is ready for use.

[0144] Below, examples of implementations of at least some of the above embodiments are described.

[0145] If the model classifies three types of RF environments (which could be, for example, indoor, outdoor line-of-sight (LOS), outdoor non-line-of-sight (NLOS)), then the known model inputs are RF data such as signal strength, fading, etc. This data is represented as many scalograms corresponding to true examples of the observed conditions, labeled as such, and used for model training (either provided directly from the network to the DUT, or used to train a network-side model, with the weights then transferred to the DUT).

[0146] This training can then be modified to insert vulnerabilities that are then tested with appropriate test inputs.

[0147] As a first option, examples of all the above classes are "radioactively" tagged at training time to form training inputs that affect the classification statistics of the DUT model in a subtle but observable way. Subsequent testing then simply involves presenting known untagged model inputs (which in this case serve as test inputs) and observing the output statistics. Multiple classes across the entire model can then be examined with a single test sequence, at the cost of either having to provide more test inputs or providing some form of observing normal outputs over time to build up statistics.

[0148] As a second option, an "indoor", an "outdoor LOS" and an "outdoor NLOS" backdoor are constructed for the above type of RF environment. This can be done by providing more heavily modified training inputs, while the test inputs can be designed to be clearly different and not used in normal use (e.g., reserved in communication standards such as the 5G specification as an example, and including unrealizable conditions such as a UE speed of 600 Km / h or a heart rate of 400 when the UE is carried by a person on foot). During testing, the backdoors cause the DUT model to generate highly anomalous outputs if they are still present. However, a correct response to an indoor backdoor test does not necessarily reveal information about the status of the outdoor LOS or outdoor NLOS classifiers. The advantage is that if only a few classes (e.g., three classes) are present, only a corresponding number of tests are needed.

[0149] As a third option, a brute force approach is applied by training an additional class that is not normally present in the usage environment (e.g., "satellite communication"). Then, during testing, it can be checked that the model still recognizes "satellite communication". What is learned from such testing is that too much external training was not done because, if it was done, the "satellite communication" class would likely be forgotten because it was never observed. The precise threshold for what constitutes too much external training is highly implementation specific.

[0150] FIG. 5 illustrates a flow diagram of the use phase of a model testing procedure according to one embodiment.

[0151] The use phase begins when the DUT indicates to the network, e.g., via the model communication system, in step S501 (Tr-REQ), that training (or a suitable trained model) is required. Alternatively, the requirement for training or testing is triggered by an internal timer of the model communication system 20 or the signal design system 10 (e.g., based on information that a certain maximum time has passed since the last training), or by a trigger received from a core network function (e.g., the Network Data Analysis Function (NWDAF)) or an application function indicating that (re)training is required. The DUT determines that training is required due to reaching at least one of several conditions, such as a new deployment of the DUT or DUT model, a new use of an existing DUT model, carrier or owner policy requirements, time elapsed since the last training, etc. In one example, the DUT signals the requirement for training (or a trained model) or testing to the network, e.g., via a standard communication protocol. The network also obtains such an indication based on policy or by examining the possible DUT behavior.

[0152] In another example, the DUT or model communication system detects that the location of the DUT has changed beyond a certain policy criterion (e.g., moved far away from a particular position or distance from a gNB or other detection), which triggers the DUT or model communication system to signal a request to be tested or trained again. Other policy criteria that trigger the DUT to request training include, for example, the number of connection failures, signal quality / signal strength criteria, battery level, amount of time out of coverage, establishment of a roaming connection to a visiting public land mobile network (PLMN) (rather than being connected to the device's home PLMN), establishment of a connection to a different gNB or UE. Other criteria also trigger a request for the DUT to be trained again, such as if the DUT has been subjected to a cybersecurity-related attack (e.g., denial of service attack, intrusion attack) or has begun to exhibit anomalous transmission activity (e.g., has begun to send messages to the network at a much higher frequency). If the DUT is a User Equipment (UE) connected to a 5G network, such behavior, and also deviations from policy, are detected by a Network Data Analysis Function (NWDAF), which sends a message to the model communication system to trigger one or more DUTs to be tested or trained again. Additional triggers to retest / retrain the DUTs come from the Network Exposure Function, through which an authorized third party may control / adapt the behavior of a set of UEs.

[0153] In response to the above instruction, the network provides training inputs (or trained models) to the DUT, for example, via a model communication system, in step S502 (Tr-I→DUT).

[0154] The training inputs modified by the signal design system may be provided simultaneously with additional known model inputs (e.g., to train the DUT model normally) or provided in advance, with the training being balanced by the DUT being performed independently.

[0155] In one example, the model communication system provides the training inputs by retrieving the required test inputs from an external input database and using an RF control algorithm that modifies the test transceiver to provide the required RF characteristics to the DUT, which then signals via the model communication system that it has received the training inputs.

[0156] In step S503, initial test inputs are sent to the DUT and the DUT responses are checked against expected values ​​to ensure successful training (Te-I→DUT; CHK RSP). Thus, the network tests the correct reception and application of training inputs by sending one or more test inputs immediately after training or model deployment. This testing phase can be performed multiple times, triggered by the network or the DUT.

[0157] In one embodiment, the network provides test inputs to the DUT, e.g., via a test transceiver, in the same manner as, e.g., training inputs. A DUT response is then received or otherwise detected via behavior of the DUT 30 (e.g., selection of a particular channel, etc.), which may be read by the model communication system. The received or otherwise detected DUT response may be compared to an expected response related to the transmitted test input by referencing an external input database. Additionally or alternatively, the DUT logs its wireless operation (e.g., by creating a log file) (e.g., whereby every or selected frame that the DUT transmits or receives via the wireless network interface results in a new log entry (possibly with a timestamp), whereby additional information regarding, e.g., frequency of transmission, transmit power, modulation and coding scheme, antenna configuration or usage, RF measurements, or other relevant RF communication parameters is added as part of the logged information). The selection of the frame may be based on a policy configured for the DUT by a management entity. The policy may determine when it should be logged and under what circumstances. For example, events identified as unlikely or suspicious may be logged, and information resulting from these logs (e.g., log files) may be sent (either on demand, policy-driven, or by push) to model communication system 20 for further analysis as to whether the DUT behavior in the provided logs corresponds to expected behavior or DUT responses based on the respective tests.

[0158] If the DUT responses match the expected responses, the DUT is considered ready for use and its status is logged in a DUT status database. If there is a mismatch, the testing and / or training is repeated.

[0159] In another embodiment, the network provides test inputs to the DUT along with the predicted outputs. The DUT responses are then evaluated locally and compared to the predicted outputs. If there is a mismatch or the outputs deviate beyond a threshold according to the deployed policy, the DUT must revert to a previous version of the model. This decision can be made by the DUT itself or after interaction with the system.

[0160] The DUT 30 is then used normally while the DUT model collects additional known model inputs during its use. At intervals, in step S504, for example when the DUT is within range of a network (e.g., a gNB cell) or according to policy, a test timing algorithm may be used to determine whether a test update is required (T-UD: Te-I → DUT; RX RSP).

[0161] Test timing is determined based on several factors in various embodiments.

[0162] In one example, the DUT is, for example, an IoT node that needs to minimize signaling and calculate power usage at the node, so that it is advantageous to have a test that only operates while using the "normal" RF capabilities of the device. In this case, the neural network model is not used for RF communication tasks. The neural network model is simply provided with networking related RF parameters to enable this remote low resource consumption test mode.

[0163] In another example, an IoT node is configured to repurpose its RF hardware (normally used for communication) to service sensing functions that may include, for example, room occupancy detection. The occupancy detection process is influenced by a neural network operating on RF parameters. The "occupancy" model is remotely tested by training the neural network to respond to test inputs within the range accessible to the communication network. The IoT model is updated externally even if on-device training is not used and still requires remote testing as a service. Thus, tests can be appropriately designed and timed such that IoT nodes are tested only after a long period of time.

[0164] In a further example, the DUT is a UE, such as a smartphone, that may communicate over the sidelink to relay UEs or other peer devices and may communicate over regular uplink / downlink communications. During sidelink communications, the UE is actively using its neural network model to perform channel selection, scheduling, etc., while during other times it is under network (e.g., gNB) control. In this case, each "session" on the sidelink is likely used to collect additional training data by the DUT model. Thus, the network tests the UE every time it comes into range and / or the relay UE is requested to perform model testing (in this case, the relay UE is also optionally used for training). This testing may determine whether additional training has been performed in the intervening sidelink session. This allows the network to ensure that standard compliance is maintained by the UE when communicating on the sidelink.

[0165] In a further example, the DUT is a UE with a neural network or other ML model for positioning. In this case, the known model input is received from signal strength (RSS) samples or other signal quality samples or synchronization / timing information (e.g., of a position reference signal) of some anchor device (e.g., a device whose location is known), and the predicted output is the UE's location. The model training is refined locally on the UE by user input of other positioning methods, such as Global Navigation Satellite System (GNSS). The test then consists of a set of RSS samples and the predicted model output (i.e., UE location), which may be provided, for example, by a gNB "spoofing" the anchor node. In this case, the timing may be opportunistically determined, for example, when the UE is in a suitable relationship with a gNB that can accurately provide the required RSS value.

[0166] Test timing is determined based on the state of multiple DUTs. In large networks, it may be more efficient to distribute test data or commands to trigger local tests in a multicast or broadcast channel. This may be a suitable option to minimize communication overhead or reduce the delay to proper operation of all devices.

[0167] If, in step S504, the test timing algorithm determines that a test is required, the network transmits the test input by repeating step S503.

[0168] If the test passes successfully, then in step S505 the DUT status is updated in the DUT status database (DUT-S→DB).

[0169] When the DUT performs a local test, the DUT only reports, for example, a message that the test failed or passed, depending on, for example, the previously deployed policy. If the test fails, the model communication system retrieves the corresponding policy from the DUT status database and enforces the policy via the policy enforcement module.

[0170] One example of such a policy is to force the DUT model to roll back to a previous version of the model, which can be enforced via conventional signaling procedures defined in public standards.

[0171] Optionally, according to a pre-deployment policy, the DUT may only need to upload the locally trained model to the system if it passes the test according to certain criteria, e.g., the number of false positives or false negatives when classifying a certain data pattern is not higher than a threshold, thereby reducing communication overhead, e.g., for federated learning, when the purpose of uploading the locally trained model includes, e.g., improving the global model.

[0172] Optionally, according to a pre-deployment policy, the DUT may need to upload a locally trained model to the system and, optionally, record the data that resulted in an updated model only if the test fails according to certain criteria, e.g., the number of false positives or false negatives when classifying a particular data pattern is higher than a threshold. Such an updated model may indicate, for example, that the DUT has been exposed to unknown data and events, such as an administrative entity in a system that relies on federated learning. This reduces, for example, the communication overhead of federated learning, when the purpose of uploading the locally trained model includes, for example, improving the global model.

[0173] FIG. 6 shows a schematic diagram illustrating the training phase according to one embodiment.

[0174] During the training phase, a generic neural network model (GMM-MD, e.g., version v1.0) is provisioned on the UE 30 and receives training data (NM-Tr-I) that is typically required during normal use from an internal or local model training system 610. Additionally, network-based training inputs (NW-Tr-I) are provided from the network 600 (e.g., gNB) to prepare the model for later testing.

[0175] FIG. 7 shows a schematic diagram illustrating the stages of use according to one embodiment.

[0176] In the architecture of FIG. 7, a first UE 32 and a second UE 34 with their respective individualized neural network models (IND-MD) are tested via a network-based test input (NW-Te-I) forwarded by the network 600.

[0177] For example, the first UE32's personalized neural network model of version v1.3.2 is a compliant model (CMPL) and therefore delivers a predicted output (OEXP) in response to a received test input.

[0178] In contrast, the individualized neural network model of the second UE34, for example of version v1.4.x, is a non-compliant bifurcation model (DIV) and therefore delivers an unpredicted output (OUEXP) in response to the received test input.

[0179] The network may therefore be made aware that the model of the second UE34 has diverged and requires new training or a reset, or may have diverged due to exposure to data as of yet unknown to the network.

[0180] In a further embodiment, input data is collected from an additional entity (a third party device). The main procedure of Fig. 3 is slightly modified to allow for the provision of known model inputs, training inputs and test inputs by a third party device (rather than directly from a tester (e.g., network) to the DUT). This embodiment relates to the use case of core network testing of a model of a gNB based on signal quality inputs (e.g., CQI, etc.) provided by a UE in the cell of the gNB. In this use case, the UE provides both known prediction inputs and training and test inputs to the gNB via modification of their reported values.

[0181] To accomplish this, the signal design system queries the third party device to provide known expected inputs that are stored in an external input database. The training and test input design system then designs training and test inputs that are based in part or in whole on modifications of the values ​​provided to the DUT by the third party device. In certain examples, the training and test inputs are even equal to the known inputs.

[0182] During the use phase, training and test inputs are delivered to the DUT by setting the design state in one or more third party devices, the backbone network communicating directly with the third party devices via predefined logical channels for this purpose.

[0183] The third party device then transmits the appropriate parameters as required by the network to provide training and test inputs.

[0184] The DUT responses (and therefore the results of the tests) are observed by the network, either directly by signaling from the DUT, or via reports provided by one or several third party devices that represent the configurations taken by the DUT during the test.

[0185] In particular, in certain embodiments, the system queries the DUT to perform a test. The DUT, having locally updated its model based on input received from a third party device, performs the test with test data that may have been received. The DUT verifies whether the model output falls within the bounds of the predicted output according to the corresponding policy.

[0186] In a further embodiment, the verified output of the first model is used as a test input of the second model. In one example, the UE has a neural network model for localization. For example, given RSS samples of several anchor devices, the model outputs the location of the UE. The model is further trained locally by using inputs of other localization methods.

[0187] Here, a test consists of a set of RSS samples and a predicted model output (i.e., UE location). Thus, the data storage can be modified to allow external devices to access the data in the DUT status database, so that the normal output of the DUT itself is used as input to another model, but only as long as the DUT model remains certified in the DUT status database.

[0188] This embodiment is expected to be relevant for location-based service use cases. For example, the output of the UE's location estimation model (which depends on network parameters and / or other inputs) is used to train a local model stored in the gNB that associates parameters such as RSS values ​​reported by the UE with the UE's location. The advantage of this case is that only certified data is allowed to be used for training by the second model. Optionally, the inputs are also checked by cross-comparison with data reported by other devices.

[0189] Thus, in addition to the pass / fail result, the model communications system also stores the model output values ​​(e.g., location) in the DUT status database at this point, and optionally also stores a flag indicating that these values ​​have been cross-checked against other device reports of the same data, which may be measured, for example, via GNSS or round trip time (e.g., RTT).

[0190] When the second model wishes to use the output of the first model, the second model sends a request to the network via the model communication system. The network responds by providing the DUT status and optionally additional data from the DUT status database, as mentioned above. As long as the DUT status indicates passing the test, the second model continues to be trained based on the output of the first model.

[0191] TR37.817 summarizes the principles of AI-enabled RAN intelligence, the functional framework (e.g., inputs / outputs of components for AI functions and AI-enabled optimization), and use cases and solutions for AI-enabled RAN. The functional framework for RAN intelligence is detailed in Section 4.2, which includes the definitions of data collection, model training, model inference, or actors. In particular, an "actor" (e.g., UE) provides "feedback" (e.g., measurements, device parameters, ...) that flows towards a "data collection" (e.g., database located in the backbone network or RAN node). The "collected data is used for "model training" and "model inference". TR37.817 includes three main use cases or key performance indicators, namely, network energy savings, load balancing, and mobility. For them, TR37.817 considers several types of deployment: (1) AI / ML model training is located in the backbone network (e.g., Operation and Maintenance Management (OAM), Network Data Analysis Function (NWDAF), or other network functions), and AI / ML model inference is located in the gNB. (2) In the case of a CU-DU split architecture where AI / ML model training and AI / ML model inference are both located in the gNB, the following solutions are possible: (3a) AI / ML model training is located in the OAM (e.g., gNB-DU) and AI / ML model inference is located in the gNB-CU; and (3b) AI / ML model training and AI / ML model inference are both located in the gNB-CU. In general, the gNB(-CU) is also enabled to continue model training based on the AI / ML models trained in the OAM, NWDAF, or other network functions. For example, Figure 5.1.2.2-1 or Figure 5.3-2 describes a system where model training and model inference are both located in the RAN node. For example, Figure 5.3-1 describes a system where training is performed in the core network (OAM) and inference is performed in the RAN node.As explained in the above embodiment, in a system such as that described in Figure 5.3-2 of TR37.817, the model may evolve in unexpected ways during model training (step 4) and therefore require testing prior to modal inference (step 6). For this purpose, reference is made to Figures 8 and 9. A management entity, e.g., a management entity in a backbone network, may deploy the training model for testing to a subset of RAN nodes, as shown in Figure 8, step 5. A management entity, e.g. a management entity in a core network or in a RAN node, selects one or more devices, e.g. at least UE or at least gNB, and instructs them to act as test devices for the trained model running in the device under test (e.g. a RAN node), as shown in step 6 of Figures 8 and 9. The management entity may also configure, in the selected test device, e.g. a test schedule determining a test schedule or a test policy determining conditions for performing tests of the device under test. The management entity may also configure, in the device under test, the identity of the test device. Each selected test device may send one or more test messages, e.g. measurement direction, together with an identifier indicating the fact that the measurement report will be processed by the device under test using the new trained (not validated) model (step 3 or 5 in Figure 5.3-2 and step 7 in Figures 8 and 9 of TR37.817). A management entity, e.g. a management entity in the core network or in the RAN node, may inform the RAN node regarding the identity of the selected device to designate as the test device, so that the RAN node processes the measurement reports being sent by the selected device using the new trained model before activation (e.g. step 6a of Figure 8). A device under test, e.g., a RAN node, may use the new trained model for inference based on the input of the selected device under test. The device under test may then (1) collect and evaluate the inferred decision (which may or may not need to be sent to the device under test) or (2) send a report including the inferred decision to a management entity for evaluation. This is shown in step 9 of FIG. 8. The evaluation is done only for the set of devices under test that are communicated with in step 6a of FIG. 8. For example, a device under test, such as a RAN node, may use the new trained model for inference based on the input of a selected testing device. The device under test then transmits the inferred decision to the testing device (message 9 in FIG. 9 ), which either (1) evaluates it and transmits the evaluation, or (2) transmits the received inferred decision to a management entity for evaluation (message 10 in FIG. 9 ). A management entity, e.g. a management entity in the core network or in a RAN node, may measure the performance of the trained tested model before it is activated for all UEs for inference (e.g. step 10 in FIG. 8 or step 12 in FIG. 9).

[0192] Tdoc S3-220410 describes the need to investigate some security issues arising from the use of artificial intelligence and machine learning for NR interfaces. A related issue refers to the fact that trained models rely on measurements and inputs from potentially untrusted devices such as UEs. Such untrusted devices, such as malicious UEs or malicious RAN nodes, may purposely generate erroneous input values ​​with the goal of affecting the trained models, affecting the performance of the system, for example, in terms of key performance indicators included in TR37.817, namely energy efficiency, load balancing, or mobility. For example, if a malicious UE (or malicious RAN node) provides erroneous inputs, the trained models in the RAN nodes (or backbone networks) may not operate as expected / required and the system may fail or experience reduced performance. To address this threat, the communication system monitors the correctness of RAN AI / ML models. The monitoring may be done by testing whether a newly trained model provides the desired performance. For example, some actions may be taken based on configuration policies. If the tests are successful, the new model can be deployed for inference, e.g., to all devices. If the test fails, the new model is not deployed for inference. If a test fails in a model that is already deployed, the previous model needs to be applied. This may require the managing entity to revert the model configuration in all entities currently using it. This may be done, for example, by asking each entity what model version they are running and asking them to revert to the previous version if they are using the affected model version. This may require the managing entity to track (e.g., in a database) the model versions run by each entity in the system and send a revert request to each entity using the affected model.

[0193] Furthermore, TR37.817 also describes system interactions where the UE and the two NG-RAN nodes exchange data as part of model training and / or inference in the NG-RAN node, e.g., step 4 and step 7 of FIG. 10, which correspond to FIG. 5.3.2.3-1. In TR37.817-h00, both model training and model inference are located in the RAN node. Input from the UE or the second NG-RAN node may be corrupted or may be based on a model locally developed in the second NG-RAN node, for example. This corrupted input may be delivered for either training or inference. For example, the input delivered by the second NG-RAN node may be the current or predicted resource status, the current or predicted energy efficiency, etc. Such predictions may be based on a model manipulated by an attacker or locally developed, for example. This use case may therefore have the advantages of the above embodiments, e.g., as follows: A first NG-RAN node tests a model in a second NG-RAN node. This test requires an interaction between both NG-RAN nodes over the Xn interface, where the first NG-RAN node sends a “test” request to the second NG-RAN node to verify the quality of the model in the second NG-RAN node. The second NG-RAN node may send a “reply” message containing the answer over the Xn interface. This test requires that at least one of the NG-RAN nodes, e.g. the first NG-RAN node, includes a UE portion intended for testing of the second NG-RAN node. For example, the first NG-RAN node instructs the UE portion to send a "test" request to the second NG-RAN node, which sends a "reply" message to the UE portion of the first NG-RAN node which evaluates it or forwards it to the first NG-RAN node for evaluation, or sends the reply directly to the first NG-RAN node (through the Xn interface). For example, this "test" signal, "reply" signal, and evaluation may occur prior to accepting input data for model training (step 4) or model inference (step 7), or may be triggered by one of those actions in FIG. 10. For example, a first NG-RAN node sends a (test) request or signal to a second NG-RAN node, which evaluates the output information. The first NG-RAN node or the OAM may include an apparatus including training / test input generation means or model evaluation means. The second NG-RAN node may include a test unit for providing training input information and test input information.

[0194] In summary, systems and methods are described for testing on-device neural network models or other machine learning models based on access to the device's normal input and output channels, where the model's structure is designed such that the model will generate distinctly different outputs for a given set of test inputs only so long as its internal structure remains in a desired state. The model is made available for such testing via training the model to the appropriate structure and / or model pre-training with training inputs that are designed to cause the model to generate distinctly different outputs when subsequently presented with a particular set of "test" inputs. If the model's structure remains within acceptable limits, the device will generate predictable outputs when tested. If not, the device is forced to revert to the pre-trained model.

[0195] While the present invention has been illustrated and described in detail in the drawings and above description, such illustrations and descriptions should be considered as illustrative or exemplary rather than limiting. The present invention is not limited to the disclosed embodiments. It may be applied to various types of UE or terminal devices, such as mobile phones, vital signs monitoring / telemetry devices, smart watches, detectors, vehicles (for vehicle-to-vehicle (V2V) communication or more general vehicle-to-exchange (V2X) communication), V2X devices, Internet of Things (IoT) hubs, IoT devices including low power medical sensors for health status monitoring, medical (emergency) diagnosis and treatment devices for hospital use or first responder use, virtual reality (VR) headsets, etc.

[0196] Furthermore, the above embodiments are implemented in a pseudo-distributed deployment in which the base station is a central unit (e.g., gNB-CU) and there are two distributed units (e.g., gNB-DU), one acting as a transmitting device and the other acting as a receiving device, while the central unit is an entity that synchronizes the distributed units.

[0197] Model functions include frequency selection, routing, resource allocation, power control, etc. for Licensed Assisted Access (LAA).

[0198] A network is represented by any network access device (base station, Node B (eNB, eNodeB, gNB, gNodeB, ng-eNB, etc.), access point, etc.) that provides a geographic service area.

[0199] Furthermore, at least some of the above embodiments are implemented to provide network equipment for 5G / 6G / xG cellular networks.

[0200] Those skilled in the art will understand and effect other variations to the disclosed embodiments in practicing the claimed invention, from studying the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be advantageously used. The above description details certain embodiments of the invention. However, no matter how detailed the above may appear in the text, it will be appreciated that the invention may be practiced in many ways and is therefore not limited to the disclosed embodiments. It should be noted that the use of certain terms to describe certain features or aspects of the invention should not be interpreted as implying that the terms are redefined herein to be constrained to include any particular characteristic of the feature or aspect of the invention with which the terms are associated.

[0201] Throughout the present specification and claims, the phrase "at least one of A, B and C" or "at least one of A, B or C" should be understood as meaning "A and / or B and / or C" in general. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be advantageously used. The above description details certain embodiments of the present invention.

[0202] The described operations, such as those shown in Figures 3-5, may be implemented as program code means of a computer program and / or as dedicated hardware in the respective associated network device or function. The computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium, provided together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

Claims

1. 1. An apparatus for remote testing of an on-device machine learning model operated by a device, comprising: training input generation means for designing training input information, said training input information configured to provide a predicted output of said device in response to test input information; a test unit for applying the training input information and the test input information to the device and for obtaining output information generated by the device in response to the test input information; model evaluation means for comparing the obtained output information with a predicted output to evaluate the on-device machine learning model; An apparatus comprising:

2. The test unit comprises: providing said training input information by transmitting it to said device and obtaining said output information by receiving it by a transceiver; or 2. The apparatus of claim 1, configured to provide the training input information by interfacing with a hardware detection unit on the device and to obtain the output information by analyzing an output of the hardware detection unit on the device.

3. 3. The apparatus of claim 1 or 2, wherein the apparatus is configured to deploy a policy triggered by at least one predefined condition to determine devices requiring testing.

4. The apparatus of claim 1 , further comprising a test input design system for designing a hardware-level input that triggers a state of the on-device machine learning model to cause the device to generate an output as the output information.

5. The apparatus of claim 1 or 2, wherein the test input information corresponds to at least a subset of the training input information, and the model evaluation means is based on an accuracy of a response of an on-board machine learning model.

6. The apparatus of claim 1 or 2, further comprising an external input database for storing the test input information, the training input information and the input information, the test input information being associated with a predicted response of a network device.

7. 10. The apparatus of claim 1, wherein the apparatus enables the on-device machine learning model to be tested by applying model pre-training using a mixed data vocabulary that includes network-accessible parameters mixed with true training data for an intended function of the on-device machine learning model.

8. 10. The apparatus of claim 1, further comprising: a radio frequency control algorithm for controlling the on-device machine learning model by using a test transceiver as networking hardware for the test unit, the radio frequency control algorithm capable of modifying at least one transmission characteristic of a transmission of the test transceiver.

9. 3. The apparatus of claim 1 or 2, further comprising a test timing algorithm for determining when the device requires testing, and a status database in which results of model tests and policies regarding actions to be taken for failed tests are stored.

10. The training input generating means Designing modifications to a portion of the known model inputs to cause one or more backdoor activations to be present in the on-device machine learning model; or providing data tagging by adding tagged data to the training set to enable statistical identification of specific training input information; or using said input information as said test input information to enable testing by checking whether, within given limits, the same output is obtained as during the original training, and / or The apparatus of claim 1 or 2, configured to apply federated learning to train the on-device machine learning model.

11. 3. The device according to claim 1 or 2, wherein the device is configured to distribute the test input information or commands to trigger local tests in a unicast or multicast or broadcast channel.

12. The apparatus of claim 1 or 2, wherein the training inputs are designed based on inputs derived from known usage of the on-device machine learning model and on the type or class of the on-device machine learning model.

13. 11. A first device comprising an on-device machine learning model operated by a first device, the first device being configured to determine that training of the on-device machine learning model is required in response to reaching at least one of a set of criteria consisting of a new deployment of the first device or the on-device machine learning model, a new use of the on-device machine learning model, a carrier or owner policy requirement, and a predetermined time elapsed since the last training, to signal a training requirement to a second network device comprising the apparatus of claim 1 via a standard communication protocol and to receive test and / or training information from the second device.

14. A system comprising at least one first device according to claim 13 and at least one second device comprising an apparatus according to claim 1.

15. 15. The system of claim 14, wherein the on-device machine learning model is distributed to the first device, and the test input information comprises information used to create the on-device machine learning model and the predicted output.

16. 1. A method for remote testing of an on-device machine learning model integrated within a device, the method comprising: designing training inputs based on input information derived from known usage of the on-device machine learning model and a type or class of the on-device machine learning model, the training inputs configured to provide a predicted output of the device in response to test inputs; providing said training input information and said test input information to said device; obtaining output information generated by the device in response to the test input information; comparing the obtained output information with the predicted output to evaluate the on-device machine learning model; The method comprising:

Images