Methods and systems for generating inference models for radio frequency (RF) signals

Abstract

Various methods and systems for generating an inference model for radio frequency signals are described herein. The systems described include processors operable to receive input training data including input training data samples, receive model targets including objectives for the inference model and a deployment target for the inference model, determine a generic inference model based on the model targets and determine optimization parameters for the determined generic inference model based at least on the training dataset to obtain the inference model. The input training data samples includes RF signals and / or data based on the RF signals. The generated inference model is deployable on the deployment target. In some embodiments, the generic inference model is determined based on a type of the input training data, a size of the input training data and / or the resource constraints of the deployment target.

Description

Title: METHODS AND SYSTEMS FOR GENERATING INFERENCE MODELS FOR RADIO FREQUENCY (RF) SIGNALSCross-Reference to Related Application

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 554,323, filed on February 16, 2024. The complete disclosure of U.S. Provisional Patent Application No. 63 / 554,323 is incorporated herein by reference for all purposes.Field

[0002] The described embodiments relate to systems and methods for generating inference models for radio frequency (RF) signals.Background

[0003] Radio frequency (RF) signals are used in a wide range of radio-enabled fields, for transmitting and receiving information and sensing and detecting objects. In many cases, analyzing RF signals can reveal valuable information. For example, in sensing and detecting applications, the analysis of RF signals can help detect objects and in communications applications, the analysis of RF signals can help gain insights on network utilization. Traditionally, these analyses were performed manually, by observing patterns and changes in the RF signals.

[0004] Inference models have emerged to at least partially automate the analysis process. Inference models can allow insights to be obtained about collected RF signals and can provide signal processing capabilities. However, as the requirements of these models and the complexity of insights and signal processing tasks have increased, the skill and knowledge required to develop these models has also increased. In many cases, it can be difficult and / or impractical to locate model generation specialists with the necessary expertise in RF-enabled fields to develop models specific to RF applications, and vice versa. Additionally, often, specific models need to be developed depending on the deployment target, further increasing the complexity of model development.

[0005] There is a need for improved methods and systems for generating inference models for RF signals.Summary

[0006] The various embodiments described herein generally relate to systems and methods for generating inference models for radio frequency (RF) signals.

[0007] In accordance with an example embodiment, there is provided a method for generating an inference model for radio frequency signals. The method involves operating one or more processors to: receive input training data comprising a plurality of input training data samples, the input training data samples comprising one of: RF signals and data based on the RF signals; receive one or more model targets, the one or more model targets comprising one or more objectives for the inference model and a deployment target for the inference model; determine a generic inference model based on the one or more model targets; and determine optimization parameters for the determined generic inference model based at least on the training dataset to obtain the inference model, and wherein the generated inference model is deployable on the deployment target.

[0008] In some embodiments, the method involves operating the one or more processors to determine the generic inference model based on a type of the input training data and / or a size of the input training data.

[0009] In some embodiments, the method involves operating the one or more processors to determine the resource constraints of the deployment target and determine the generic inference model based on the resource constraints of the deployment target.

[0010] In some embodiments, the method involves operating the one or more processors to receive the resource constraints of the deployment target from a computing device in communication with the one or more processors.

[0011] In some embodiments, the representation comprises one of: spectrogram data, data packets and bits.

[0012] In some embodiments, the input training data set comprises one or more of: curated observed recordings including captured ambiently-observed data, controlled recordings, testbed transmitted recording data, synthetically generated data.

[0013] In some embodiments .receiving input training data comprises receiving, from a signal receiver, RF signals.

[0014] In some embodiments, the method involves operating the one or more processor to process the input training data, wherein processing the training data comprises operating the one or more processors to perform one or more of: segmenting the training data, labeling the training data and assigning classifications to each sample of the training data.

[0015] In some embodiments, the method involves operating the one or more processors to generate synthetic data, the synthetic data comprising one of synthetic RF signals and synthetic data based on the synthetic RF signals, the method further involves operating the one or more processors to determine the optimization parameters based at least on the synthetic data.

[0016] In some embodiments, the method involves operating the one or more processors to select a subset of the input training data and determine the optimization parameters for the generic inference model based at least on the subset of the input training data.

[0017] In some embodiments, the subset of the input training data comprises selecting the subset based on a result of a qualification test.

[0018] In some embodiments, the method involves selecting the subset of the input training data comprises selecting the subset by one or more of: removing one or more classes of the input training data, subsampling the input training data, reducing a number of samples from one or more of the classes of the input training data and applying one or more impairment models to the input training data.

[0019] In some embodiments, the method involves operating the one or more processors to validate the generated inference model via one or more of: performance testing, profiling and benchmarking.

[0020] In some embodiments, validating the generated inference model comprises determining a performance of the inference model and the method further involves operating the one or more processors to generate a visual representation of the performance of the inference model.

[0021] In some embodiments, the method involves operating the one or more processors to generate an updated inference model based on a result of the validating of the generated inference model.

[0022] In some embodiments, the method involves operating the one or more processors to determine a performance of the inference model and / or of thedeployment target when the inference model is deployed on the deployment target, relative to one or more testing objectives and generate the updated inference model based on the determined performance and the model targets.

[0023] In some embodiments, the method involves operating the one or more processors to package the inference model according to one or more of: a type of the inference model, specifications of the deployment target, a task performed by the inference model, the one or more objectives for the inference model and an application for the inference model.

[0024] In some embodiments, the method involves operating the one or more processors to determine the specifications of the deployment target based on a selection of the deployment target.

[0025] In some embodiments, the method involves operating the one or more processors to receive a selection of the generic inference model from a computing device in communication with the one or more processors.

[0026] In some embodiments, the method involves operating the one or more processor to determine the optimization parameters using a modified differentiable architecture search (DARTS).

[0027] In some embodiments, the method involves operating the one or more processors to compress the generated inference model according to at least the resource constraints of the deployment target and wherein the compressed inference model is deployable on the deployment target.

[0028] In accordance with an example embodiment, there is provided a system for generating an inference model for radio frequency signals comprising one or more processors operable to: receive input training data comprising a plurality of input training data samples, the input training data samples comprising one of RF signals and data based on the RF signals; receive one or more model targets, the one or more model targets comprising one or more objectives for the inference model and a deployment target for the inference model; determine a generic inference model based on the one or more model targets; determine optimization parameters for the determined generic inference model based at least on the processed training dataset to obtain the inference model, and wherein the generated inference model is deployable on the deployment target.

[0029] In some embodiments, the one or more processors are operable to determine the generic inference model based on a type of the input training data and / or a size of the input training data.

[0030] In some embodiments, the one or more processors are operable to determine the resource constraints of the deployment target and determine the generic inference model based on the resource constraints of the deployment target.

[0031] In some embodiments, the representation comprises one of: spectrogram data, data packets and bits.

[0032] In some embodiments, the one or more processors are operable to receive the resource constraints of the deployment target from a computing device in communication with the one or more processors.

[0033] In some embodiments, the input training data set comprises one or more of: curated observed recordings including ambiently-observed data, testbed transmitted recording data, synthetically generated data.

[0034] In some embodiments, receiving input training data comprises receiving, from a signal receiver, RF signals.

[0035] In some embodiments, the one or more processors are operable to process the input training data, wherein processing the input training data comprises performing one or more of: segmenting the training data, labeling the training data and assigning classifications to each sample of the training data.

[0036] In some embodiments, the one or more processors are operable to receive synthetic data parameters from a computing device and wherein the synthetic data is generated according to the received synthetic data parameters.

[0037] In some embodiments, processing the input training data comprises selecting a subset of the input training data and the one or more processors are operable to determine the optimization parameters based at least on the subset of the input training data.

[0038] In some embodiments, selecting the subset of the input training data comprises selecting the subset based on a result of a qualification test.

[0039] In some embodiments, selecting the subset of the input training data comprises selecting the subset by one or more of: removing one or more classes of the input training data, subsampling the input training data, reducing a number ofsamples from one or more of the classes of the input training data and applying one or more impairment models to the input training data.

[0040] In some embodiments, the one or more processors are operable to validate the generated inference model via one or more of: performance testing, profiling and benchmarking.

[0041] In some embodiments, validating the generated inference model comprises determining a performance of the inference model and wherein the one or more processors are operable to generate a visual representation of the performance of the inference model.

[0042] In some embodiments, the one or more processors are operable to generate an updated inference model based on a result of the validating of the generated inference model.

[0043] In some embodiments, the one or more processors are operable to determine a performance of the inference model and / or of the deployment target when the inference model is deployed on the deployment target relative to one or more testing objectives and generate the updated inference model based on the determined performance and the model targets.

[0044] In some embodiments, the one or more processors are operable to package the inference model according to one or more of: a type of the model, specifications of the deployment target, a task performed by the inference model, the one or more objectives for the inference model and an application for the inference model.

[0045] In some embodiments, the one or more processors are configured to determine the specifications of the deployment target based on a selection of the deployment target.

[0046] In some embodiments, the one or more processors are operable to receive a selection of the generic inference model from a computing device in communication with the one or more processors.

[0047] In some embodiments, the one or more processors are operable to determine the optimization parameters using a modified differentiable architecture search (DARTS).

[0048] In some embodiments, the one or more processors are operable to compress the generated inference model according to at least the resourceconstraints of the deployment target and the compressed inference model is deployable on the deployment target.

[0049] In accordance with an example embodiment, there is provided a method for generating an inference model for radio frequency signals comprising operating one or more processors to: receive input training data comprising a plurality of input training data samples, the input training data samples comprising one of: RF signals and data based on the RF signals; receive one or more model targets, the one or more model targets comprising one or more objectives for the inference model and a deployment target for the inference model; determine a type of generic inference model based on the input training data and the one or more objectives for the inference model; select a specific generic inference model of the type determined based on the deployment target for the inference model; and determine optimization parameters for the selected generic inference model based at least on the training dataset to obtain the inference model, and wherein the generated inference model is deployable on the deployment target.

[0050] In some embodiments, the method involves operating the one or more processors to determine computational resources of the deployment target and select the specific generic inference model based on the determined computational resources of the deployment target.Brief Description of the Drawings

[0051] Several embodiments will be described in detail with reference to the drawings, in which:FIG. 1 is a block diagram of an example radio inference model generation system in communication with external components, in accordance with an embodiment;FIG. 2A is a block diagram showing example modules of the radio inference model generation system, in accordance with an embodiment;FIG. 2B is a block diagram showing example modules of the radio inference model generation system, in accordance with an embodiment;FIG. 3 is a flowchart of an example method for generating a radio inference mode, in accordance with an example embodiment;FIG. 4A is a diagram of an example Differentiable Architecture Search (DARTS) initial cell; andFIG. 4B is a diagram showing two example modified DARTS initial cell, in accordance with an example embodiment.

[0052] The drawings, described below, are provided for purposes of illustration, and not of limitation, of the aspects and features of various examples of embodiments described herein. For simplicity and clarity of illustration, elements shown in the drawings have not necessarily been drawn to scale. The dimensions of some of the elements may be exaggerated relative to other elements for clarity. It will be appreciated that for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the drawings to indicate corresponding or analogous elements or steps.Description of Example Embodiments

[0053] In many applications, analyzing and processing RF signals can provide valuable insights. For example, in sensing and detecting applications, the analysis of RF signals can help detect objects. The detection of objects through analysis of RF signals can be particularly useful in defense applications, or in automotive applications. As another example, in telecommunication applications, the analysis of RF signals can reveal that signals transmitted or received by a satellite are being impaired by weather conditions, or that there is interference due to the presence of another satellite.

[0054] Analyzing and processing RF signals is typically performed using models. These models can be highly complex and require significant amounts of data to train. However, experts specialized in model generation rarely have the knowledge of RF signals required to develop sophisticated models and often do not have access to large volumes of training data. Conversely, RF experts typically only have limited knowledge of model generation.

[0055] The various embodiments described herein allow inference models for RF signals to be generated without requiring expert knowledge of model generation techniques. At least some of the embodiments described herein can employ different model generation techniques, depending on the type of inference model being generated. In at least one embodiment, a neural architecture search (NAS) engine isused to generate the inference models. The NAS engine can be an improved Differentiable Architecture Search (DARTS) engine that uses fewer resources, when compared conventional DARTS engines.

[0056] The various embodiments described can be used by users possessing RF data to generate inference models and / or training data for inference models for a range of deployment targets having different processing capabilities and / or resource constraints and configured for various applications using the RF data and selected model targets. At least one of the embodiments described herein can involve reducing the size of a generated inference model in accordance with processing capability constraints and / or resource constraints.

[0057] At least some of the embodiments described herein can produce training datasets and generate inference models using an integrated, centralized system, thereby reducing the number of computing systems required for developing trained inference models.

[0058] The described embodiments can be used in a wide variety of fields in which it is advantageous to analyze radio signals to obtain insights, including but not limited to, telecommunications, including satellite communications, consumer electronics, urban transportation, automotive applications and defense applications. At least some of the embodiments can be implemented using cloud-based technologies.

[0059] The various embodiments can select suitable models, train the models, test the models and package the trained and tested models according to the specifications of the deployment target, including, for example, the processing capabilities of the deployment target and the resource constraints of the deployment target. The various embodiments described can accordingly facilitate the generation of models that can be ported to any deployment target. The various embodiments described can process input training data to produce a dataset that is suitable for training each model.

[0060] At least some of the embodiments described herein can generate additional training data (e.g., synthetic data) to improve the quality and / or quantity of data in the training dataset.

[0061] In at least one embodiment, the input training data is inspected to identify features of a content of the input training data and results of the inspection can beprovided. In some cases, the results can be used to obtain insights about the input training data and / or to refine the input training data.

[0062] The inference models generated by the various embodiments described herein can be any type of model that performs inferences on RF input signals. The inference models generated can be used as part of conventional RF systems, including wireless communication systems, including systems employing 5G, 4G LTE technologies, radar systems, navigation systems, satellite communication systems, etc.

[0063] Reference is first made to FIG. 1 , which illustrates an example block diagram 100 of an inference model generation system 108 in communication with an external data storage 102, a computing device 106 and a deployment target 120 via a network 104. Although only one computing device 106 is shown in FIG. 1 , the inference model generation system 108 may be in communication with a greater number of computing devices 106. Although only one deployment target 120 is shown in FIG. 1 , the inference model generation system 108 may be in communication with a greater number of deployment targets 120. The inference model generation system 108 can communicate with the computing device(s) 106 and deployment target(s) 120 over a wide geographic area via the network 104. While the system for generating inference models 108 and the computing device 106 are shown as separate components, in some cases, the inference model generation system 108 or one or more components of the inference model generation system 108 may be implemented within the computing device 106. Alternatively, the inference model generation system 108 can be cloud-based.

[0064] The inference model generation system 108 includes a storage component 110, a processor 112, and a communication component 114. The inference model generation system 108 can be implemented with more than one computer server distributed over a wide geographic area and connected via the network 104. The storage component 110, the processor 112 and the communication component 114 may be combined into a fewer number of components or may be separated into further components.

[0065] The processor 112 can be implemented with any suitable processor, controller, digital signal processor, and / or graphics processing unit that can provide sufficient processing power for the configuration, purposes and requirements of theinference model generation system 108. The processor 112 can include more than one processor with each processor being configured to perform different dedicated tasks.

[0066] The communication component 114 can include any interface that enables the inference model generation system 108 to communicate with various devices and other systems. For example, the communication component 114 can receive inputs (e.g., input training data, model targets) from the computing device 106 and store the inputs in the storage component 110 or external data storage 102. The processor 112 can then process the inputs according to the methods described herein.

[0067] The communication component 114 may also include an interface to component via one or more of an Internet, Local Area Network (LAN), Ethernet, Firewire, modem, fiber, or digital subscriber line connection. Various combinations of these elements may be incorporated within the communication component 114. For example, the communication component 114 may receive input from various input devices, such as a mouse, a keyboard, a touch screen, a thumbwheel, a trackpad, a track-ball, a card-reader, voice recognition software and the like depending on the requirements and implementation of the system for generating inference models 108.

[0068] The storage component 110 can include RAM, ROM, one or more hard drives, one or more flash drives or some other suitable data storage elements such as disk drives. The storage component 110 can include one or more databases for storing data such as, but not limited to related to the inference models including untrained models, the inference models generated by the inference model generation system 108, model tuning parameters and data for generating the inference models, data related to the validation of a trained inference model, including reports generated by the inference model generation system 108 and test results associated with the trained inference model, input data, including input training data, synthetic data generated and stored processed data and model targets, and packaged inference models.

[0069] The external data storage 102 can store data similar to that of the storage component 110. The external data storage 102 can, in some embodiments, be used to store data that is less frequently used and / or older data. In some embodiments, the external data storage 102 can be a third-party data storage storedwith input data for analysis by the inference model generation system 108. The data stored in the external data storage 102 can be retrieved by the computing device 106 and / or the inference model generation system 108 via the network 102.

[0070] The computing device 106 can include any device capable of communicating with other devices through a network such as the network 102. A network device can couple to the network 102 through a wired or wireless connection. The computing device 106 can include a processor and memory, and may be an electronic tablet device, a personal computer, workstation, server, portable computer, mobile device, personal digital assistant, laptop, smart phone, WAP phone, an interactive television, video display terminals, gaming consoles, and portable electronic devices or any combination of these. For example, the computing device 106 can be a user device used for obtaining a trained inference model generated by the inference model generation system 108. The computing device 106 can be configured to collect RF signal data and / or be in communication with a device for collecting RF signal data.

[0071] The deployment target 120 can be any a device, processor, controller, chip and / or an application on which the inference model is to be deployed which includes one or more processors configured to implement the inference model. For example, the deployment target can be a satellite, a ground station, a cloud service provider, a workstation, an embedded device, or any other computer hardware or software used for radio communications. In some embodiments, the deployment target 120 may not directly communicate with the inference model generation system 108 and instead may communicate with the inference model generation system 108 indirectly, via the computing device 106. For example, the computing device 106 can be configured to receive one or more inference models from the inference model generation system 108 and be configured to implement onto and / or transmit the one or more inference models to the deployment target 120. In some embodiments, the computing device 106 can be a deployment target.

[0072] The network 104 can include any network capable of carrying data, including the Internet, Ethernet, plain old telephone service (POTS) line, public switch telephone network (PSTN), integrated services digital network (ISDN), digital subscriber line (DSL), coaxial cable, fiber optics, satellite, mobile, wireless (e.g. WiFi, WiMAX), SS7 signaling network, fixed line, local area network, wide area network,and others, including any combination of these, capable of interfacing with, and enabling communication between, the inference model generation system 108, the external data storage 102, and the computing device 106.

[0073] Reference is next simultaneously made to FIGS. 2A and 3, which show modules of the inference model generation system 108 in accordance with an embodiment, and a flowchart of a method 300 of generating an inference model using one or more processors of the inference model generation system 108, in accordance with an embodiment, respectively. Though FIG. 2A shows four modules, the modules may be subdivided into sub-modules. Alternatively, some of the modules can be combined and the modules shown can be sub-modules.

[0074] At 310, the inference model generation system 108 receives input training data containing a plurality of data samples via the dataset managing module 210. The input training data can include data collected from radio devices. For example, the input training data can include signal data (e.g., IQ samples). In at least some embodiments, the input training data includes data derivative representations of the collected signal data (e.g., spectrogram data, data packets, bits). In some embodiments, the input training data includes labeled training data. The input training data can include collected data (e.g., curated observed recordings including ambiently-observed data, testbed transmitted recording data, synthetically generated data).

[0075] Ambiently-observed data can be data that aims to capture a target application or observable signal. Ambiently observed data can be captured by connecting a receiver (e.g., spectrum monitor, software-defined radio) to an antenna and capturing recordings. The data can be procedurally captured, i.e. , the signal can be captured using different capture parameters (e.g., capture frequencies, sample rates, gain settings, filter settings, antennae) and / or the data can be captured from two or more receiver-antenna pairs using different capture parameters. For example, for an LTE base station operating at 2630 MHz and operating a 10MHz cell, by performing captures at 20, 30, 40, 61.44 MS / s, at gains of 0, 5, 10, 15, 20, from two antennas, at center frequencies of 26202625 MHz, 2630, MHz 2635 MHz, 2640 MHz etc., 250 recordings can be obtained.

[0076] The ambiently-observed data can be processed before being used as input training data as described below with reference to step 320.

[0077] Synthetically generated data can be generated when large quantities of data (e.g., terabytes) are required. Synthetically generated data can be procedurally generated using a set of rules and a scenario definition (e.g., a scene) provided to a procedural engine. The procedural engine can generate transmission parameters for a scene based on the set of rules and the scenario definition. For example, the procedural engine can generate the following transmission parameters: Transmitter 1 at center frequency (CF) of 2630 MHz, 10MHz wide, QPSK, T ransmitter 2 with CF of Transmitter 1 + 5 MHz, 5 MHz wide, QAM64, half the transmission power of transmitter 1.

[0078] The synthetic data can be processed before being used as input training data as described below with reference to step 320.

[0079] Testbed data can be data collected from radios controlled by an orchestration application, where one or more transmitters generate a scene consisting of over-the-air transmissions, and one or more receivers collect the signals to produce recordings. The orchestration application can control the timing, execution, triggering of the radios and determine the transmission and / or capture parameters of the receivers (e.g., center frequencies, number of receivers). For example, the orchestration application can generate the following transmission and capture parameters: Transmitter 1 at CF 2630 MHz, 10MHz wide, QPSK, transmitter 2 has CF Transmitter 1 + 5 MHz, 5 MHz wide, QAM64, half the transmission power, receiver captures at 61 .44 MS / s with a CF of 2640 MHz.

[0080] The transmitters can also generate variations corresponding to different scenarios (e.g., different days, different times), allowing more diverse data to be captured.

[0081] The testbed data can be processed before being used as input training data, as described below with reference to step 320.

[0082] In some embodiments, the inference model generation system 108 generates data over-the-air. For example, the inference model generation system 108 can receive radio transmission data parameters, for example, from the computing device 106 defining a range of parameters (e.g., frequencies, phases, channels, impairments, etc.) for the data. The radio transmission data parameters can for example, be specified by a user. In such embodiments, the inference modelgeneration system 108 generates data according to the received radio transmission parameters.

[0083] The input training data can be received for example, from the computing device 106 via the network 104. Alternatively, the inference model generation system 108 can retrieve at least a portion of the input training data from the external data storage 102 or the storage component 110. For example, the input training data can include data previously used for generating an inference model. Alternatively, at least a portion of the input training data (e.g., synthetic data) can be generated by the inference model generation system.

[0084] Optionally, at 320, the inference model generation system 108 processes the input training data received at 310 via the dataset managing module 210 to obtain a processed training dataset. The processed training dataset can be a dataset that can be used as training data for a machine-learning model or as an input to a machine-learning model.

[0085] In some embodiments, processing the input training data involves discarding data that does not meet predefined quality threshold, examining each sample in the input training data, segmenting the sample, assigning a label to the sample and assigning a class to the sample. The inference model generation system 108 can process all of the samples in the input training data and group the samples according to the assigned classes. In some embodiments, two or more classes can be combined, for example, based on allowable predetermined variations. The predetermined variations can, for example, be received from the computing device 106.

[0086] In some embodiments, the labels can be assigned based on metadata associated with each sample. The labels can be assigned based on transmission parameters, capture parameters, classes, or any other category.

[0087] In some embodiments, processing the input training data involves slicing the input training data. The input training data can be shuffle sliced, that is, input training dataset can be cut into segments in randomly generated positions.

[0088] In some embodiments, processing the input training data involves generating synthetic data to augment the training data. In some embodiments, the synthetic data includes modulated versions of the input data and / or modified versions of the input training data modified according to models (e.g., impairment, time shifting,frequency shifting, random resampling, pulse shaping, multipath fading, impulsive noise and other noise models, other fading models, IQ imbalance).

[0089] In some embodiments, processing the input training data involves segmenting the input training data according to labels assigned to the samples contained in the input training data.

[0090] In some embodiments, processing the input training data involves combining the input training data with one or more existing training datasets. The existing training datasets can be retrieved from memory, for example, from the storage component 110 or the external data storage 102. The existing training datasets can include processed training datasets previously generated.

[0091] In some embodiments, processing the input training data involves selecting a subset of the input training data to be used for generating the inference model. The subset can be selected based on a result of one or more qualification tests. Selecting a subset of the input training data can allow the inference model generation system 108 to select data meeting quality requirements.

[0092] For example, the qualification test(s) can include measurements-based tests and / or label-based tests root mean squared (RMS) based qualification test(s), quantization test(s), energy detection test(s), model-based test(s) (including machine learning model-based test(s)) and any combination thereof. In some embodiments, only data meeting certain qualification test(s) may be retained for training the inference model.

[0093] Further processing can be performed on the input training data, for example, as described in PCT / CA2024 / 050912, herein incorporated by reference.

[0094] In some embodiments, the inference model generation system 108 receives a selection of the subset. For example, the inference model generation system 108 can receive user selections from the computing device 106.

[0095] In some embodiments, the subset of the input training data is obtained by performing modifications to the input training data, including but not limited to, reducing the size of the subset based on predetermined size parameters, removing classes from the subset, homogenizing classes (e.g., removing one or more samples from a class to homogenize the number of samples across classes), modifying the data in the subset according to various models.

[0096] The processed training dataset can be stored, for example, in the external data storage 102 or the storage component 110 at one or more stages of the processing. For example, the augmented training data can be stored and the subset of the input training data can be stored separately. In some embodiments, the processed training dataset is saved for future retrieval. For example, the processed training dataset can be used to augment other training data and / or can be used to generate other inference models.

[0097] At 330, the inference model generation system 108 receives one or more model targets for the inference model. The one or more model targets can be user- specified model targets, for example, targets specified by a user using the computing device 106 and transmitted to the inference model generation system 108 via the network 104. The one or more model targets can include one or more objectives for the inference model and a deployment target for the inference model. The one or more objectives can define one or more inference tasks that are desired to be obtained. For example, the one or more objectives can include anomaly detection, signal classification, event detection, consumption patterns, network utilization, network configurations, impairment classification, vacancy forecasting, resource selection, resource assignment, beam selection, user tracking, sensing, ranging, channel selection, signal fingerprinting, transmitter classification, signal filtering, etc.

[0098] As explained, the deployment target can correspond to a device, a controller, a processor, chip and / or an application on which the inference model is to be deployed and / or configured. For example, the deployment target can be a satellite, a ground station, a cloud service provider, a workstation, an embedded device, or any other computer hardware or software used for radio communications. As other examples, the deployment target can be an FPGA, a CPU, a neural processor, etc.

[0099] In some embodiments, the inference model generation system 108 receives constraints for the inference model to be generated. For example, the inference model generation system 108 can receive information about the processing capabilities and / or resource constraints of the deployment target. Alternatively, the inference model generation system 108 can determine constraints for the inference model to be generated, based on the deployment target. For example, the inference model generation system 108 can determine the processing capabilities of the one or more processors or chip(s) configured to implement the inference model based onthe deployment target specified. For example, processing capabilities of the processor(s) or chip(s) of deployment targets can be stored in memory, for example, in the storage component 110 or the external data storage 102.

[0100] At 340, the inference model generation system 108 determines a generic inference model and determines optimization parameters (e.g., hyperoptimization) for the generic inference model to obtain the inference model, based at least on the one or more model targets and the input training data or the processed training dataset and generates an inference model having the determined configuration, via the model building module 230.

[0101] In some embodiments, determining a generic inference model involves selecting a specific generic inference model or a type of generic inference model from a data storage such as the storage component 110 or the external data storage 102. The generic inference model model can be a machine-learning model, including, but not limited to a Resnet model, a time series transformer model, the Inception model, an MLP-Mixer model, other transformer models, models from the Efficientnet family, models from the Mobilenet family, other segmentation and object detection computer vision models (e.g., Deeplab V3, YOLO), and long short-term memory (LSTM) deep learning models.

[0102] As explained, the inference model generation system 108 can determine a generic inference model based at least on the model objectives and / or deployment targets and the input training data or processed input training data. For example, in determining a generic inference model, the inference model generation system 108 can analyze the nature of the input training data, the nature of the model objective(s), the complexity of different types of inference models and the resource and / or operational constraints of the deployment target.

[0103] For example, raw IQ signals typically have strong temporal dependencies and accordingly, sequential models may be more suited for performing inferences on raw IQ signals. As another example, spectrograms or images typically require spatial processing, and accordingly models with convolution or vision-focused architectures may be more suited for performing inferences on spectrograms and images.

[0104] Similarly, sequence-to-sequence or time forecasting tasks typically require models that are capable of capturing temporal correlations and accordingwhen the model objectives include such tasks, the inference model generation system 108 can select a generic inference model that is capable of capturing temporal correlations.

[0105] Similarly, if the deployment target is a type of target that requires realtime inferences to be performed, the inference model generation system 108 can select a generic inference model with a lightweight or efficient architecture to minimize latency and memory usage. As another example if the deployment target has substantial computational resources, the inference model generation system 108 can select a more complex generic inference model that can process more data.

[0106] For example, if the input training data includes raw samples from an RF receiver, the model objective is to classify or detect RF interference in real-time, requiring short-lived transient bursts and the deployment target is a target that has tight latency requirements, the inference model generation system 108 can select a lightweight transformer-based model configured for sequence modeling since transformer-based models are well-suited for effectively capturing long-range correlations and subtle intermittent patterns, in part due to their global attention mechanism, and a lightweight model can have lower latency.

[0107] As another example, if the input training data is an RF signal processed into 2D or 3D representations (e.g., range-Doppler maps), the model objective is to detect objects, and the deployment target is a target with limited computational resources, the inference model generation system 108 can select a model with a lightweight convolutional neural network-based (CNN-based) architecture, since CNN are well-suited for efficient, real-time processing of spatial data. As a further example, if the model objective is to detect objects overtime, the inference model generation system 108 can select a model with a CNN-transformer architecture, since CNN- transformer models are well suited for processing of temporal and spatial data.

[0108] In some embodiments, the inference model generation system 108 determines the optimization parameters for the generic inference model based on the model target(s) received at 330, resource constraints determined or received by the inference model generation system 108, the type of input training data received at 310 and / or the size of the input training dataset or the processed training dataset.

[0109] Alternatively, in other embodiments, the inference model generation system 108 first determines a type of generic inference model for the inference modelbased on the model target(s) received at 330. The inference model generation system 108 then determines the optimization parameters for the determined type of model based on the type of input training data, the size of the input training data and / or the resource constraints of the deployment target, as determined by the inference model generation system 108 or as received by the inference model generation system 108 from the computing device 106 (e.g., a user input specifying resource constraints).

[0110] Alternatively, in other embodiments, the inference model generation system 108 first determines a type of generic inference model for the inference model based on the type of input training data and the model objective(s) and determine a specific generic inference model of the type determined based on the deployment target. For example, if the input training data includes raw samples from an RF receiver and the objective is to classify or detect RF interference in real-time, the inference model generation system 108 can first determine that a transformer-based model ora lightweight transformer-based model may be suitable. The inference model generation system 108 can then determine that the deployment target has abundant computational resources and select, from a selection transformer-based models, a specific deep transformation model having many layers.

[0111] In some embodiments, the inference model generation system 108 receives a selection of inference model requirements from the computing device 106. For example, the inference model generation system 108 can receive a selection of the generic inference model.

[0112] In other embodiments, the inference model generation system 108 subsamples the input training data or the processed training dataset and performs parameter optimization on two or more generic models to determine the generic model most suited for the input training data and the model objective(s).

[0113] Determining optimization parameter involves determining tuning parameters for the model to select optimal parameters (e.g., hyperparameters) for the model, based on the input training dataset or the processed training dataset and the model. The optimization can be performed, for example, using hyperparameter optimization techniques. The hyperparameter optimization techniques can be unconstrained, involving a search of all possible options available and broad searching strategies, or constrained, involving a limited search space and ofsearching strategies. The optimization technique used can vary, depending on the type of generic model selected.

[0114] In some embodiments, for example, in embodiments where the type of generic model selected is a neural network, the inference model generation system 108 implements a Neural Architecture Search (NAS) engine to determine the optimal parameters for the inference model. The NAS engine can be a modified DARTS engine. As is generally known to those skilled in the art, DARTS is an optimization technique that can determine model parameters for a neural network using an input dataset, a selection of a type of model architecture (e.g., transformer model, recurrent neural networks for time-series, recurrent neural networks for sequential data, convolutional neural networks for images, convolutional neural networks for spatial data, and multilayer perceptrons, which are suited for general-purpose tasks) and an optimization objective (e.g., cross-entropy loss; mean-squared error; inference time, model size, memory efficiency, or a combination thereof). Using these inputs, the DARTS engine can initialize a generic neural network of the type selected and through an iterative process, jointly optimize the network’s weights and architecture by cycling through the input dataset until convergence is achieved.

[0115] DARTS reformulates the traditionally discrete and combinatorial optimization problem of designing and optimizing neural network architectures into a continuous bi-level optimization framework. DARTS involves an inner optimization that focuses on training the neural network parameters, and an outer optimization that refines the architecture itself. The DARTS can search for optimal neural network parameters within the space of possible parameters, which can vary, depending on the type of the neural network.

[0116] The bi-level optimization framework of DARTS allows gradient-based optimization to be used, which accelerates the NAS process when compared to conventional discrete optimization methods, reducing computational costs and enhancing resource efficiency, making DARTS particularly well suited when limited computational resources are available.

[0117] The modified DARTS engine is a search that uses fewer computational resources, when compared to a conventional DARTS. FIG. 4A shows an overview of an example cell of a conventional DARTS prior to optimization and FIG. 4B shows an overview of two example cells cell / and cell j of the modified DARTS prior tooptimization. As shown in FIG. 4A, in a conventional DARTS cell, each node 0, 1 , 2, 3 is connected to all other nodes in the cell via candidate operations. In contrast, as shown in FIG. 4B, the modified DARTS engine uses fewer connections since each node 0, 1 , 2, 3 is only connected to the immediately preceding and immediately subsequent node, which enables the neural network’s weights to converge more rapidly, in fewer computational steps. Although FIG. 4B shows cells / and j having four nodes, there can be fewer or more than four nodes in a cell.

[0118] The modified DARTS engine employs fewer cells when compared to a conventional DARTS engine. The modified DARTS engine can employ fewer cells by interconnecting cells having matching dimensionality, as shown in FIG. 4B, which shows interconnected cells / and j, unlike a conventional DARTS engine, which uses unconnected cells. The interconnections can be learnable skip connections. By interconnecting cells, the DARTS can use fewer cells and the neural network’s architecture can converge more rapidly, using fewer computation steps when compared to a conventional DARTS.

[0119] When compared to a conventional DARTS, the modified DARTS can be less susceptible to overfitting. In a conventional DARTS, batch normalization is applied throughout the architecture and at the node level to stabilize the training process. Batch normalization involves normalizing each input feature by subtracting the mean and dividing by the standard deviation calculated from the current minibatch during training. During inference, however, normalization relies on running mean and variance statistics accumulated as moving averages during training. When unseen data is used during inference, the running statistics can fail to generalize, which can lead to overfitting to the training data.

[0120] The modified DARTS instead employs group normalization, wherein normalization is performed after each cell is optimized. Group normalization operates independently of a batch size and instead normalizes each sample (e.g., cell) individually. Group normalization involves dividing the feature channels into groups and computing the mean and variance for each group within the sample, thereby eliminating the need for mini-batch statistics while retaining the stabilization benefits of normalization.

[0121] By performing normalization after each cell is optimized, the modified DARTS retains the optimization stability provided by normalization while minimizing the risk of overfitting.

[0122] In some embodiments, the modified DARTS includes modified reduction cells. In a conventional DARTS, reduction cells halve the spatial dimensions of the input feature map while keeping the number of channels constant throughout the architecture. In the modified DARTS, modified reduction cells double the number of channels, providing finer control over the feature map size and channel dimensions, when compared to conventional DARTS and enabling the modified DARTS to explore models with varying levels of representational capacity. By using modified reduction cells, the performance of the modified DARTS is therefore improved when compared to a conventional DARTS.

[0123] In some embodiments, the deployment target may be resource constrained (e.g., the deployment target may be a mobile device, an edge device and computing platforms, an Internet of Things system) and / or the deployment target requires inferences to be determined in real-time. In such cases, large inference models may be suboptimal, since they can require substantial computational resources and time to perform inferences, can have high memory and storage requirements, and can consume large amounts of energy.

[0124] The inference model generation system 108 can compress the inference model generated to obtain an inference model having a smaller size (e.g., a model that is 50%, 80%, etc. smaller) and / orwith reduced computational complexity. For example, the inference model generation system 108 can use model compression techniques. The compressed inference model can be an inference model that has a smaller size / reduced computational complexity but that performs similarly to the uncompressed inference model. By compressing the inference model, the inference model generation system 108 can achieve reductions in inference time, energy consumption, and computational complexity, when deployed, since smaller models require fewer computations, and can allow the inference model to be stored on deployment targets having limited storage capabilities. Reducing inference time and computational complexity can be particularly advantageous when the inference model is used for performing real-time inferences and / or when the deployment target is resource-constrained. Compressing models can also facilitate deployment, sincemore efficient models can be more easily deployed in distributed and federated learning applications.

[0125] The inference model generation system 108 can employ structured or unstructured pruning for compressing the inference model. Pruning techniques can reduce the size of a neural network model by removing redundant or unnecessary parameters, neurons or layers determined to be less important, while maintaining performance. Structured pruning involves removing entire components of a neural network’s architecture, such as neurons, filter channels, and / or layers, which can substantially reduce memory usage and computational requirements and allow for efficient execution on hardware accelerators using optimized libraries. Unstructured pruning involves removing individual weights from a neural network, without altering the overall structure of the neural network, instead resulting in sparser weight matrices. When compared to structured pruning, unstructured pruning allows for more fine-grained control over which parameters are pruned, which can allow for accuracy to be preserved.

[0126] In at least one embodiment, the inference model generation system 108 employs magnitude-based pruning or Taylor importance-based pruning methods for compressing the inference model.

[0127] Magnitude-based pruning is a computationally inexpensive pruning technique that involves pruning weights with the smallest absolute values and is based on the assumption that weights with smaller magnitudes contribute less to the output of a model and can accordingly be removed with minimal impact on the model’s performance. Magnitude-based pruning can be performed in stages, gradually increasing the pruning ratio while fine-tuning the model to maintain performance. The inventors have found that using magnitude-based pruning to prune a MobileNetV3 model, the inference model generation system 108 can achieve an up to 50% reduction in the number of parameters of the model, while losing only about 2% in accuracy, when compared to the uncompressed model. The reduced size of the model enables inference time to be reduced by approximately 30%. As explained, in applications where real-time inferences are required, reducing inference time can improve the performance of the inference model. Magnitude-based pruning can also reduce computational complexity, reducing computational and energy demands.

[0128] Taylor importance-based pruning is a pruning technique that uses first- order Taylor expansion to estimate the impact of removing each parameter, on the loss function and that considers both the weight values and their gradients. When compared to magnitude-based pruning, Taylor importance-based pruning tends to preserve model performance better. The inventors have found that using Taylor importance-based pruning to prune a MobileNetV3 model, the inference model generation system 108 can achieve an up to 50% reduction in the number of parameters of the model while losing only about 1 % in accuracy, when compared to the uncompressed model. The reduced size of the model enables inference time to be reduced by approximately 30%. As explained, in applications where real-time inferences are required, reducing inference time can improve the performance of the inference model. Taylor importance-based can also reduce computational complexity, reducing computational and energy demands.

[0129] In some embodiments, the inference model generation system 108 can determine the pruning technique according to the resource constraints of the deployment target, the model objective, the type of deployment target and / or the resource constraints of the inference model generation system 108. For example, when the inference model generation system 108 has limited computational resources to prune the inference model, the inference model generation system 108 can employ magnitude-based pruning to prune the inference model as magnitudebased pruning employs fewer computational resources than Taylor importance-based pruning. As another example, when inference model is to be deployed on a deployment target that requires high accuracy, the inference model generation system 108 can employ Taylor importance-based pruning.

[0130] In some embodiments, the inference model generation system 108 receives model compression parameters (e.g., reduction percentage) and / or compression objectives (e.g., minimize execution time, minimize parameter count, balanced execution time and parameter count, minimize training time), for example, from computing device 106. For example, a user input can specify that an 80% reduction in the inference model size is desired. In such embodiments, the inference model generation system 108 can prune the inference model until the desired inference model size is achieved.

[0131] In some embodiments, the inference model generation system 108 retrains the pruned inference model to improve the performance of the pruned inference model. For example, the inference model generation system 108 can employ knowledge distillation techniques to distill knowledge from the uncompressed inference model and retrain the pruned inference model.

[0132] In at least some embodiments, at 350, the inference model generation system 108 validates the generated inference model via the model validation module 230. Validating the generated inference module can involve testing the inference model. Testing the inference model can involve benchmarking the inference model to obtain data about the inference model’s performance (e.g., total inference time, performance of each layer of the inference model), performance testing the inference model and / or profiling the inference model.

[0133] In some embodiments, testing the inference model involves determining an overall accuracy of the model using a testing dataset. The testing dataset can be a pre-existing testing dataset stored in memory, for example in the storage component 110 or the external data storage 102. The testing dataset can include data similar to the input training dataset, data different from the input training dataset, data having a wider or narrower range, data falling outside the range of the input training dataset and / or data of the same nature but having different parameters (e.g., different modulation) In some embodiments, the testing dataset includes data received from a user via, for example, the computing device 106. For example, the inference model generation system 108 can receive a test file from the computing device 106.

[0134] In some embodiments, testing the inference model involves over-the-air testing. For example, the inference model can be deployed on a test target (e.g., test hardware and / or test software) and the performance and / or the behavior of the inference model can be assessed. The test target can be generally similar to the deployment target for the inference model. As another example, as will be described in further detail below, the inference model can be deployed onto the deployment target prior to being tested.

[0135] Benchmarking can involve deploying the inference model on the deployment target and observing the inference model’s impact on the deployment target (e.g., a processor, processing unit, a controller of the deployment target) by measuring metrics such as resource consumption, timing, behavior of the deploymenttarget or the inference model, inference time, latency, rate. Benchmarking can also involve running the inference for an extended period of time to determine the stability of the deployment target when the inference model is deployed on the deployment target. The results of the benchmarking can be used to determine whether changes to the inference model are required, for example, whether the inference model requires further tuning, whether the architecture of the inference model is adequate and whether the inference model requires further compression.

[0136] Performance testing involves measuring features of the inference model, for example, an average confidence level and accuracy of the inference model, including the confidence level and accuracy of the inference model for all inputs or for specific inputs (e.g., classes of data) to determine if the inference model can be generalized to conditions on which the inference model has not been trained. Performance testing can be performed by varying testing data, for example, by varying transmission parameters. Testing data can be synthetic data, testbed data or ambiently-observed data, as described above. Testing data can include data on which the inference model has not been trained. For example, performance testing can involve progressing the inference model through a test regime that exposes the inference model to “challenge” data (e.g., a harder variant of a task for which the inference model has been trained) and “unseen data” (e.g., data having different transmission parameters as data on which the inference the model has been trained). The results of performance testing can be used to determine whether changes to the inference model are required, for example, whether the inference model requires further tuning, whether the architecture of the inference model is adequate and whether the inference model requires further compression.

[0137] For example, if performance testing indicates that the inference model is highly accurate, the inference model generation system 108 may determine that the inference model can be compressed to save computational resources while still maintaining adequate accuracy.

[0138] During performance testing, testing data can be automatically provided to the inference model. Results obtained using testing data can be compared with ground truths to determine where model breakdowns may occur (e.g., the types of data that may lead to model breakdown) or where the inference model may be more limited and the overall performance of the inference model.

[0139] Profiling the inference model can involve determining the inference model’s characteristics, including, but not limited to, its execution requirements, activations and behaviors. The individual components (e.g., each step, process, operation) of the inference model can be monitored as data is passed through the inference model. Based on the results of the profiling test, the inference model generation system 108 can retrain the inference model to improve its performance. For example, if, during a profiling test, a sigmoid of the inference model is determined to be slow, the inference model generation system 108 may retrain the inference model by, for example, replacing the sigmoid with a ReLU activation function.

[0140] In some embodiments, the inference model generation system 108 determines the performance of the model based on the testing and generates report(s) of the performance. In some embodiments, determining the performance of the model involves generating a performance score. In some embodiments, the performance of the model is assessed by comparing an output of the model against the ground truth, against one or more independent input variables and / or against simple and harsh impairments to input variables.

[0141] In some embodiments, the report(s) include visual representations (e.g., graphs, tables). In some embodiments, the report(s) include an indication of areas where the performance of the model may be less accurate, for example, inputs associated with predictions having a confidence score that falls below a predetermined threshold. The visual representation can be displayed on a graphical user interface, for example, a graphical user interface of the computing device 106.

[0142] In some embodiments, the inference model is further trained based on the result of the validation to obtain an updated inference model. For example, based on tests performed, the inference model generation system 108 may determine that the performance of the inference model is inadequate and / or inadequate for the one or more model targets received at 330. As another example, the inference model generation system 108 may determine that the requirements of the inference model exceed the capabilities of the deployment target. In such cases, the method 300 can involve returning to step 340 and generating a new model or retraining the model to generate an updated inference model.

[0143] In some embodiments, the model and / or the revised model is assessed to determine the resource requirements of the inference model. In some cases, thedeployment target may have limited resources. In such cases, the inference model generation system 108 can determine if the generated model is suitable for the limited resources of the deployment target. To assess the requirements of the generated model, a testing dataset can be used. The testing dataset can be a pre-existing testing dataset stored in memory, for example in the storage component 110 or the external data storage 102. Alternatively, or in addition thereto, the testing dataset can include testing data generated in real-time. The testing dataset can include the input training data and / or the processed training dataset.

[0144] In some embodiments, a prioritization engine 250 is employed to update the the inference model based on the results of the model validation (see FIG. 2B). The prioritization engine can form part of the inference model generation system 108 or can be separate from the inference model generation system 108 and receive data from the inference model generation system. The prioritization engine 250 can be an engine controlled by an operator (e.g., a user) or a trained machine-learning model (e.g., a trained large language model). The prioritization engine 250 can determine trade-offs for the inference model based on the deployment target and / or the model objective.

[0145] In embodiments employing a prioritization engine, one or more testing objectives can be determined for the inference model and the inference model’s performance and / or the performance of the deployment target during testing can be assessed relative to the testing objective. The testing objective(s) can be selected by a user and / or predetermined. Testing objectives can include but are not limited to, total inference time, performance of each layer of the inference model, accuracy of the inference model, performance of the inference model as a whole, impact of the inference model on the deployment target, etc.

[0146] Based on the inference model’s performance and / or the performance of the deployment target relative to the testing objective, the prioritization engine 250 can determine an updating process for generating an updated inference model. The updating process can involve determining updated optimization parameters for the inference model, compressing or further compressing the inference model, using different factors (e.g., model targets, type of input training data, size of the input training data, resource constraints of the deployment target) to determine the genericinference model or weighting the different factors in a different manner than originally weighted and / or changing components of the generic inference model.

[0147] The prioritization engine 250 can determine the updating process based on the model objective(s) and the deployment target. For example, the prioritization engine 250 can determine acceptable tradeoffs (e.g., model size vs accuracy, inference time vs accuracy) based on the model objective(s) and the deployment target.

[0148] In at least some embodiments, the inference model generation system 108 packages the generated inference model via the model packaging module 240 according to specifications (e.g., hardware, software specifications) of the deployment target, the type of model generated, the task performed by the inference model, an objective of the inference model, the application for which the inference model is intended and / or any combination of these. For example, the inference model generation system 108 can receive a selection of an application for which the inference model is intended and receive information related to the processing capabilities of one or more processors of the deployment target and / or a configuration of the deployment target from the computing device 106. As another example, the inference model generation system 108 can determine the processing capabilities of the one or more processors of the deployment target and / or the configuration of the deployment target based on the type of the deployment target and known information about different types of deployment targets. The selection of an application may be received prior to generating the inference model, for example, when the input training data is received, or after the inference model is generated, prior to packaging the inference model.

[0149] In some embodiments, the inference model generation system 108 can select a packaging configuration for the inference model by referencing a database of packaging configurations. Selecting a packaging configuration from a database of packaging configurations can reduce the processing requirements associated with packaging the inference model generated.

[0150] In some embodiments, the inference model generation system 108 transmits the packaged inference model to an external device such as the computing device 106. The external device can correspond to the deployment target. Alternatively, the external device may be in communication with the deployment targetand be configured to implement the received inference model onto the deployment target. For example, the external device can be configured to integrate the inference model into an application and implement the application on the deployment target. Alternatively, in some embodiments, the inference model generation system 108 can integrate the inference model into an application, which may be transmitted to the external device. The inference model deployed can be used to implement an inference engine at the deployment target.

[0151] In some embodiments, the inference model is further tested, once deployed. Based on a result of the testing, the inference model can be retrained or further updated and the updated inference model can be redeployed onto the deployment target.

[0152] In some embodiments, the inference model generation system 108 stores the packaged inference model, for example, for future use or retrieval. The packaged inference model can be stored, for example, in the storage component 110 or the external data storage 102.

[0153] It will be appreciated that numerous specific details are set forth in order to provide a thorough understanding of the example embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Furthermore, this description and the drawings are not to be considered as limiting the scope of the embodiments described herein in any way, but rather as merely describing the implementation of the various embodiments described herein.

[0154] The embodiments of the systems and methods described herein may be implemented in hardware or software, or a combination of both. These embodiments may be implemented in computer programs executing on programmable computers, each computer including at least one processor, a data storage system (including volatile memory or non-volatile memory or other data storage elements or a combination thereof), and at least one communication interface. For example and without limitation, the programmable computers (referred to below as computing devices) may be a server, network appliance, embedded device, computer expansion module, a personal computer, laptop, personal data assistant, cellular telephone,smart-phone device, tablet computer, a wireless device or any other computing device capable of being configured to carry out the methods described herein.

[0155] In some embodiments, the communication interface may be a network communication interface. In embodiments in which elements are combined, the communication interface may be a software communication interface, such as those for inter-process communication (IPC). In still other embodiments, there may be a combination of communication interfaces implemented as hardware, software, and combination thereof.

[0156] Program code may be applied to input data to perform the functions described herein and to generate output information. The output information is applied to one or more output devices, in known fashion.

[0157] Each program may be implemented in a high-level procedural or object- oriented programming and / or scripting language, or both, to communicate with a computer system. However, the programs may be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program may be stored on a storage media or a device (e.g. ROM, magnetic disk, optical disc, USB) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein. Embodiments of the system may also be considered to be implemented as a non-transitory computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.

[0158] Furthermore, the system, processes and methods of the described embodiments are capable of being distributed in a computer program product comprising a computer readable medium that bears computer usable instructions for one or more processors. The medium may be provided in various forms, including one or more diskettes, compact disks, tapes, chips, wireline transmissions, satellite transmissions, internet transmission or downloadings, magnetic and electronic storage media, digital and analog signals, and the like. The computer useable instructions may also be in various forms, including compiled and non-compiled code.

[0159] Various embodiments have been described herein by way of example only. Various modification and variations may be made to these example embodiments without departing from the spirit and scope of the invention, which is limited only by the appended claims.

Claims

WE CLAIM:

1. A method for generating an inference model for radio frequency signals comprising operating one or more processors to: receive input training data comprising a plurality of input training data samples, the input training data samples comprising one of: RF signals and data based on the RF signals; receive one or more model targets, the one or more model targets comprising one or more objectives for the inference model and a deployment target for the inference model; determine a generic inference model based on the one or more model targets; and determine optimization parameters for the determined generic inference model based at least on the training dataset to obtain the inference model, and wherein the generated inference model is deployable on the deployment target.

2. The method of claim 1 , further comprising operating the one or more processors to determine the generic inference model based on a type of the input training data and / or a size of the input training data.

3. The method of claim 1 or 2, further comprising operating the one or more processors to determine the resource constraints of the deployment target and determine the generic inference model based on the resource constraints of the deployment target.

4. The method of claim 3, further comprising operating the one or more processors to receive the resource constraints of the deployment target from a computing device in communication with the one or more processors.

5. The method of claim 1 , wherein the representation comprises one of: spectrogram data, data packets and bits.

6. The method of any one of claims 1 to 5, wherein the input training data set comprises one or more of: curated observed recordings including captured ambiently- observed data, controlled recordings, testbed transmitted recording data, synthetically generated data.

7. The method of any one of claims 1 to 6, wherein receiving input training data comprises receiving, from a signal receiver, RF signals.

8. The method of any one of claims 1 to 7, further comprising operating the one or more processor to process the input training data, wherein processing the training data comprises operating the one or more processors to perform one or more of: segmenting the training data, labeling the training data and assigning classifications to each sample of the training data.

9. The method of any one of claims 1 to 8, further comprising operating the one or more processors to generate synthetic data, the synthetic data comprising one of synthetic RF signals and synthetic data based on the synthetic RF signals, the method further comprising operating the one or more processors to determine the optimization parameters based at least on the synthetic data.

10. The method of any one of claims 1 to 9, further comprising operating the one or more processors to select a subset of the input training data and determine the optimization parameters for the generic inference model based at least on the subset of the input training data.11 . The method of claim 10, wherein selecting the subset of the input training data comprises selecting the subset based on a result of a qualification test.

12. The method of claim 10 or 11 , wherein selecting the subset of the input training data comprises selecting the subset by one or more of: removing one or more classes of the input training data, subsampling the input training data, reducing a number ofsamples from one or more of the classes of the input training data and applying one or more impairment models to the input training data.

13. The method of any one of claims 1 to 12, further comprising operating the one or more processors to validate the generated inference model via one or more of: performance testing, profiling and benchmarking.

14. The method of claim 13, wherein validating the generated inference model comprises determining a performance of the inference model and wherein the method further comprises operating the one or more processors to generate a visual representation of the performance of the inference model.

15. The method of any one of claims 13 to 14, further comprising operating the one or more processors to generate an updated inference model based on a result of the validating of the generated inference model.

16. The method of claim 15, further comprising operating the one or more processors to determine a performance of the inference model and / or of the deployment target when the inference model is deployed on the deployment target, relative to one or more testing objectives and generate the updated inference model based on the determined performance and the model targets.

17. The method of any one of claims 1 to 16, further comprising operating the one or more processors to package the inference model according to one or more of: a type of the inference model, specifications of the deployment target, a task performed by the inference model, the one or more objectives for the inference model and an application for the inference model.

18. The method of claim 17, further comprising operating the one or more processors to determine the specifications of the deployment target based on a selection of the deployment target.

19. The method of any one of claims 1 to 18, further comprising operating the one or more processors to receive a selection of the generic inference model from a computing device in communication with the one or more processors.

20. The method of any one of claims 1 to 19, further comprising operating the one or more processor to determine the optimization parameters using a modified differentiable architecture search (DARTS).21 . The method of any one of claims 1 to 20, further comprising operating the one or more processors to compress the generated inference model according to at least the resource constraints of the deployment target and wherein the compressed inference model is deployable on the deployment target.

22. A system for generating an inference model for radio frequency signals comprising one or more processors operable to: receive input training data comprising a plurality of input training data samples, the input training data samples comprising one of RF signals and data based on the RF signals; receive one or more model targets, the one or more model targets comprising one or more objectives for the inference model and a deployment target for the inference model; determine a generic inference model based on the one or more model targets; determine optimization parameters for the determined generic inference model based at least on the processed training dataset to obtain the inference model, and wherein the generated inference model is deployable on the deployment target.

23. The system of claim 22, wherein the one or more processors are operable to determine the generic inference model based on a type of the input training data and / or a size of the input training data.

24. The system of claim 22 or 23, wherein the one or more processors are operable to determine the resource constraints of the deployment target and determine the generic inference model based on the resource constraints of the deployment target.

25. The system of claim 22, wherein the representation comprises one of: spectrogram data, data packets and bits.

26. The system of claim 25, wherein the one or more processors are operable to receive the resource constraints of the deployment target from a computing device in communication with the one or more processors27. The system of any one of claims 22 to 26, wherein the input training data set comprises one or more of: curated observed recordings including ambiently-observed data, testbed transmitted recording data, synthetically generated data.

28. The system of any one of claims 22 to 27, wherein receiving input training data comprises receiving, from a signal receiver, RF signals.

29. The system of any one of claims 22 to 28, wherein the one or more processors are operable to process the input training data, wherein processing the input training data comprises performing one or more of: segmenting the training data, labeling the training data and assigning classifications to each sample of the training data.

30. The system of claim 29, wherein the one or more processors are operable to receive synthetic data parameters from a computing device and wherein the synthetic data is generated according to the received synthetic data parameters.31 . The system of any one of claims 22 to 30, wherein processing the input training data comprises selecting a subset of the input training data and wherein the one or more processors are operable to determine the optimization parameters based at least on the subset of the input training data.

32. The system of claim 31 , wherein selecting the subset of the input training data comprises selecting the subset based on a result of a qualification test.

33. The system of claim 31 or 32, selecting the subset of the input training data comprises selecting the subset by one or more of: removing one or more classes of the input training data, subsampling the input training data, reducing a number of samples from one or more of the classes of the input training data and applying one or more impairment models to the input training data.

34. The system of any one of claims 22 to 33, wherein the one or more processors are operable to validate the generated inference model via one or more of: performance testing, profiling and benchmarking.

35. The system of claim 34, wherein validating the generated inference model comprises determining a performance of the inference model and wherein the one or more processors are operable to generate a visual representation of the performance of the inference model.

36. The system of any one of claims 34 to 35, wherein the one or more processors are operable to generate an updated inference model based on a result of the validating of the generated inference model.

37. The system of claim 36, wherein the one or more processors are operable to determine a performance of the inference model and / or of the deployment target when the inference model is deployed on the deployment target relative to one or more testing objectives and generate the updated inference model based on the determined performance and the model targets.

38. The system of any one of claims 22 to 37, wherein the one or more processors are operable to package the inference model according to one or more of: a type of the model, specifications of the deployment target, a task performed by the inference model, the one or more objectives for the inference model and an application for the inference model.

39. The system of claim 38, wherein the one or more processors are configured to determine the specifications of the deployment target based on a selection of the deployment target.

40. The system of any one of claims 22 to 39, wherein the one or more processors are operable to receive a selection of the generic inference model from a computing device in communication with the one or more processors.41 . The system of any one of claims 22 to 40, wherein the one or more processors are operable to determine the optimization parameters using a modified differentiable architecture search (DARTS).

42. The system of any one of claims 22 to 41 , wherein the one or more processors are operable to compress the generated inference model according to at least the resource constraints of the deployment target and wherein the compressed inference model is deployable on the deployment target.

43. A method for generating an inference model for radio frequency signals comprising operating one or more processors to: receive input training data comprising a plurality of input training data samples, the input training data samples comprising one of: RF signals and data based on the RF signals; receive one or more model targets, the one or more model targets comprising one or more objectives for the inference model and a deployment target for the inference model; determine a type of generic inference model based on the input training data and the one or more objectives for the inference model; select a specific generic inference model of the type determined based on the deployment target for the inference model; and determine optimization parameters for the selected generic inference model based at least on the training dataset to obtain the inference model, and wherein the generated inference model is deployable on the deployment target.

44. The method of claim 43, further comprising operating the one or more processors to determine computational resources of the deployment target and select the specific generic inference model based on the determined computational resources of the deployment target.

Images