Test and measurement device, and testing method of adaptive system under test
The test system addresses the limitations of existing methods by evaluating both transmitting and receiving capabilities of adaptive systems, ensuring accurate signal recognition and effective response selection in dynamic electromagnetic contexts.
Patent Information
- Application Number
- JP2024073799
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-04-23
- Filing Date
- 2024-04-30
- Publication Date
- 2024-11-08
AI Technical Summary
Existing methods for testing machine learning-based adaptive systems, such as RF systems, only evaluate the response performance of receiving devices without considering the effectiveness in the electromagnetic context of the received signal, neglecting the bidirectional nature of these systems.
A comprehensive test system that evaluates both transmitting and receiving capabilities of adaptive systems, including communication and radar systems, by generating test signals, analyzing responses, and verifying the effectiveness of adaptive functions through machine learning algorithms, using specialized test and measurement devices like arbitrary waveform generators and spectrum analyzers.
Ensures accurate recognition of input signal characteristics and effective response selection, enhancing the adaptive system's performance in dynamic electromagnetic environments by evaluating both signal detection and response effectiveness.
Smart Images

Figure 2024159737000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a test system, and more particularly to a test and measurement instrument for verifying the operation of a machine learning based adaptive transmit / receive system, and a method for testing the adaptive system under test. [Background technology]
[0002] Many transmit and receive systems, such as communications systems and radio frequency (RF) systems including radar, require the ability to dynamically adapt or respond to the current conditions of the signal environment in order to maximize application specific functional requirements and performance metrics, such as communications data throughput, radar jamming effectiveness, dynamic spectrum allocation, and interference avoidance.
[0003] These transmit and receive systems are often referred to as "adaptive" or "cognitive" and are very often enabled by machine learning (ML) algorithms (or more broadly, "artificial intelligence (AI) / ML") that provide statistically robust models for making decisions under both known and unknown operating conditions.
[0004] One example of these transmitting and receiving systems may be an adaptive transmitting / receiving system that senses interfering traffic or jamming signals within the frequency range in which it operates and responds in a manner that maintains a sufficient data rate. The adaptive system may respond by adjusting the operating frequency to an open channel, adjusting power, using a different modulation scheme to maximize the signal-to-noise ratio (SNR), allocating a higher proportion of packet data to error correction, etc., at a more protocol level. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] U.S. Patent No. 11,047,957
Non-Patent Literature
[0006]
Non-Patent Literature 1
Non-Patent Literature 2
Non-Patent Literature 3
Non-Patent Literature 4
Summary of the Invention
Problems to be Solved by the Invention
[0007] U.S. Patent No. 11,047,957 describes one approach to testing and validating machine learning decision logic in RF systems. This approach is applied only to evaluating the response performance of the receiving device. However, this approach does not evaluate the effectiveness of applying the response in the electromagnetic context of the received signal that gave rise to the response. [Means for solving the problem]
[0008] The validity of a two-way system (both transmit and receive) with adaptive capabilities needs to be verified at two main levels. This is especially true for systems where the adaptive capabilities arise from machine learning, such as deep learning neural networks (DNNs). In this description, the transmit and receive systems are referred to as the "adaptive system" and the test system evaluates the ability of the adaptive system to adapt to the input signal and the environment in which it operates. Adaptive systems can take many forms, such as communication systems like radio frequency (RF) systems, radar systems, LIDAR (light detection and ranging), and communication systems using embedded physical (PHY) layer models in an Ethernet environment.
[0009] The first level of validation verifies the ability of the receiving system and the machine learning system to properly detect signals and identify (discriminate) the type and characteristics of the incoming signal. Removing signal impairments from the receiving environment improves the ability of the adaptive system to correctly recognize the characteristics of the incoming signal. The ability to recognize the characteristics of the incoming signal depends on the signal characteristics and the ability of the algorithm to overcome a combination of impairments from the receiving environment and impairments from the receiving chain of the adaptive system. These three sub-steps within this first level can be thought of as "detection," "parameter identification," and "removal or avoidance of impairments." This first step may consist of any or all of these three sub-steps.
[0010] The second level of validation verifies the validity of the response selection part of the adaptive system's operational logic. This operational logic includes the selection of the most effective parameters for the response, as measured by the ability of the adaptive system to maximize the figure of merit (data rate, jamming effectiveness, etc.) of its operation. In other words, this validation level is a way to validate the validity "intelligence" of the application of an artificial intelligence (AI) / machine learning (ML) based adaptive system (also called the system under test (SUT)). The validation results can be in the form of matrices, real or complex valued vectors, scalars or simple "pass" / "fail" results.
[0011] Due to the bidirectional nature of these adaptive transmit / receive systems, training the adaptive system's decision logic and validating the performance quality of the SUT requires both the ability to stimulate the SUT and the ability to receive the SUT response to determine the effectiveness of both steps. This capability is different from U.S. Pat. No. 11,047,957, which focuses on radar object detection and uses classification at the output of the receiving device's machine learning algorithm as the final judge of performance. However, in operation, the performance quality of the adaptation functions in these adaptive systems includes the quality of the SUT's response, which requires the ability to receive the SUT's response and evaluate the performance characteristics specific to the application of the SUT's response. This evaluation requires a training and validation system that includes both stimulus and response capabilities.
[0012] The response of a radio, communication, radar, electronic countermeasure (ECM) or other type of SUT depends on the particular characteristics of the environment in which the SUT operates, referred to herein as the "electromagnetic (EM) environment." The objective effectiveness of the SUT involves the transmission of a response signal, which is measured at the physical (or simulated) point where the signal is intentionally generated. This measurement includes the ability of the response to overcome the characteristics of the current electromagnetic environment. The SUT's response is verified at the point where the SUT's response is delivered and where the responsive transmission signal is received, and by verifying the operation of an algorithm that determines the appropriate response. As part of the first level "obstacle removal or avoidance" step, the obstacles themselves may also be extracted as parameters (e.g., in matrix, vector, scalar real- or complex-valued form) and provided to the second level effectiveness assessment. [Brief description of the drawings]
[0013] [Figure 1] FIG. 1 shows an embodiment of a test system for an adaptive transmit and receive system. [Diagram 2] FIG. 2 shows an embodiment of a test setup used to test and evaluate the performance of the transmitting and receiving systems. [Diagram 3] FIG. 3 shows a flow chart of an embodiment for testing a transmitting and receiving system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] FIG. 1 shows an example of a configuration of a test system 100 according to an embodiment of the disclosed technique. In this example, the transmission / reception system 100 is an RF system. The test system 100 includes a test and measurement apparatus 10 and an adaptive RF system 20 as a system under test (SUT). The test and measurement apparatus 10 includes an RF signal generating section 12, an RF signal analyzing section 16, and a signal processing and system control unit 22. The test and measurement apparatus 10 may be embodied as one or more separate test and measurement apparatuses, or may be comprised of a single test and measurement apparatus including the above elements in a single housing. For ease of explanation, any of these embodiments will be referred to as the test and measurement apparatus 10 in this application.
[0015] When the test and measurement apparatus 10 is embodied using one or more separate test and measurement apparatuses, the RF signal generator 12 may be, for example, an arbitrary waveform generator manufactured by Tektronix (see Non-Patent Document 2). The RF signal generator 12 may generate an analog signal using a built-in digital-to-analog converter (DAC) based on digital waveform data stored in advance in the RF signal generator, digital waveform data received from the signal processing and system control unit 22, and setting parameters of the waveform data. The RF signal analyzer 16 may be, for example, a spectrum analyzer or signal analyzer manufactured by Tektronix (see Non-Patent Document 3), or an oscilloscope capable of analyzing signals from the time domain and the frequency domain, such as the 6 Series B MSO mixed signal oscilloscope manufactured by Tektronix (see Non-Patent Document 4). The RF signal analyzer 16 may generate digital waveform data from the response signal received from the SUT using a built-in analog-to-digital converter (ADC), and use the digital waveform data to analyze the response signal from various viewpoints, such as the time domain and the frequency domain. For example, time domain waveform data of the response signal, frequency domain waveform data (ie, spectrum) of the response signal, and associated analysis data may then be provided to the signal processing and system control unit 22 .
[0016] The test and measurement instrument 10 may include one or more communication ports, such as communication ports 14 and 18. The RF signal generating section 12 and the signal analyzing section 16 may use a common communication port or may each have their own port, as illustrated in Figure 1. In the case of an RF system, the communication ports 14 and 18 may each have separate antennas 15 and 19, respectively, for wireless communication or may have a wired connector for connecting to the SUT via a wired connection (e.g., cable).
[0017] The SUT 20 can take many forms, and the test and measurement instrument 10 may communicate with many different types of SUTs. The embodiments are not intended to be limiting or suggestive of an adaptive system. In the following description, for simplicity, the system under test 20 may be referred to as an adaptive RF system, but the embodiments apply to all types of adaptive transmit and receive systems. In FIG. 1, an exemplary SUT 20 has a transmitter 26 and a receiver 28, and is connected to a communication port 24 via a signal conditioning unit 32. The communication port 24 may wirelessly transmit and receive RF signals to and from the communication ports 14 and 18 via an antenna 23 and further via antennas 15 and 19. Although FIG. 1 shows an example in which the communication ports 14 and 18 are wirelessly connected to the communication port 24, the communication between the communication ports 14 and 18 and the communication port 24 may be wired (e.g., cabled). In addition, the wired communication may use electrical signals, optical signals, etc. The transmitter 26 and receiver 28 are connected to a machine learning system 30, such as a deep learning network.
[0018] FIG. 2 illustrates an embodiment of the signal processing system and control unit 22 of the test and measurement instrument 10. The signal processing system and control unit 22 includes a memory 42, a processor 40, and one or more communication ports 44. The communication ports 44 may be connected to the signal analyzer 16 and the signal generator 12, or may be directly connected to the SUT 20 shown in FIG. 1. The test and measurement instrument 10 may include a user workspace 46. The user workspace 46 may have a connection to the adaptive system under test 20, separate from the wireless communication between the antennas. The user workspace 46 may take many forms, including a partition or other segment of the memory 42, a dedicated memory, or a combination of the memory 42 and some processing resources. In an example of such a user workspace 46, a development environment such as Visual Studio®, LabVIEW®, MATLAB®, etc., running on the test and measurement instrument may be run within the user workspace 46. These allow a user or a provider of the test and measurement instrument to develop logic and parameters for determining pass / fail results of a test run.
[0019] FIG. 3 illustrates one embodiment of a method for performing a training or validation process using the test system. The test and measurement instrument 10 of FIG. 1 generates a "test signal" at step 50. The test signal may be, for example, a radar, communication signal (e.g., signals with competing frequencies), or a jamming signal that disrupts or stresses the signal currently being transmitted by the SUT 20. In response, the SUT 20 detects and identifies the parameters and characteristics of the test signal and selects an appropriate response signal at step 52. In the case of a radar signal, the response signal from the SUT 20 may consist of a countermeasure or electronic countermeasure (ECM) signal. The SUT 20 responds to the test signal by transmitting a response signal, such as a jamming signal to counter the test signal or a signal based on other avoidance mechanisms to avoid interference. Thus, the test signal may be described as a signal that disrupts or stresses the signal that the SUT 20 was transmitting before transmitting the response signal. It should be noted that actions within methods shown in dashed lines indicate actions performed by SUT 20 or actions involving SUT 20 and / or test and measurement instrument 10.
[0020] The test and measurement instrument 10 then captures the response signal 54 with the signal analysis unit 16, generates waveform data thereof, and generates various aspects of data based thereon as described above. The test and measurement instrument 10 then verifies, at step 56, the selection of the response signal selected by one or more processors 40 within the test and measurement instrument 10 or within the signal analysis unit 16 as a countermeasure against the test signal adversely affecting or stressing the transmission signal of the SUT 20. This verification at step 56 may include comparing the response signal to the test signal to determine the appropriateness of the response signal.
[0021] This comparison process may compare the response signal and the test signal, for example, in terms of frequency (channel being used), power, encoding method being used, etc. This makes it possible to verify whether, for example, a response was made in response to a test signal, such as changing the frequency (channel) of the signal that the SUT 20 had been transmitting so far so that it would not conflict with the frequency (channel) of the test signal, changing the power of the signal that had been transmitted so far to one higher than the power of the test signal, or changing the encoding method compared to the test signal to one that can transmit data more reliably even if the surrounding electromagnetic conditions are poor.
[0022] The test and measurement instrument 10 then evaluates the performance of the SUT 20 based on the response signals at step 58. Measured performance parameters include power, bandwidth, frequency, pulse shape, etc., and are evaluated against desired performance parameters to determine whether the SUT 20 is responding properly to adverse effects of the test signals. If the SUT 20 meets the performance measurements at step 60, the system passes and the test proceeds to step 62, where the test ends.
[0023] The test and measurement instrument 10 may report the evaluation results to the SUT 20 or to the user workspace 46 on the test and measurement instrument 10 (step 58). The test and measurement instrument 10 or the SUT 20 may then analyze the response signal and determine whether the response signal was satisfactory from the standpoint of the test signal. This determination may include finding out whether the machine learning system 30 of FIG. 2 needs further training or additional testing on different test signals is required at step 64. This determination may further include the test and measurement instrument 10 making recommendations to the user or working directly with the user workspace 46 to manage the training and validation sequence. The test and measurement instrument 10 or the SUT 20 may specify settings for the new test signal. The test and measurement instrument 10 may then adjust settings as necessary so that the signal generator 12 generates the new test signal at step 66.
[0024] In this way, the performance of the adaptive system under test is evaluated in terms of the transmitted test signal and not just its reception performance, so that the machine learning system providing the response signals to the test signals has enough information to enable the most accurate response.
[0025] Aspects of the disclosed technology can operate on specially created hardware, firmware, digital signal processors, or specially programmed general-purpose computers, including processors that operate according to programmed instructions. The term "controller" or "processor" in this application contemplates microprocessors, microcomputers, ASICs, and dedicated hardware controllers, among others. Aspects of the disclosed technology can be implemented in computer-available data and computer-executable instructions, such as one or more program modules, executed by one or more computers (including a monitoring module) or other devices. Generally, program modules include routines, programs, objects, components, data structures, and the like, which, when executed by a processor in a computer or other device, perform particular tasks or implement particular abstract data formats. The computer-executable instructions may be stored in computer-readable storage media, such as hard disks, optical disks, removable storage media, solid-state memory, RAM, and the like. As will be appreciated by those skilled in the art, the functionality of the program modules may be combined or distributed as desired in various embodiments. Furthermore, such functionality may be embodied in whole or in part in firmware or hardware equivalents, such as integrated circuits, field programmable gate arrays (FPGAs), etc. Certain data structures may be used to more effectively implement one or more aspects of the disclosed techniques, and such data structures are considered to be within the scope of the computer-executable instructions and computer-usable data described herein.
[0026] The disclosed aspects may, in some cases, be implemented in hardware, firmware, software, or any combination thereof. The disclosed aspects may also be implemented as instructions carried by or stored on one or more computer-readable media that may be read and executed by one or more processors. Such instructions may be referred to as computer program products. A computer-readable medium as described herein means any medium that can be accessed by a computing device. By way of example, and not limitation, computer-readable media may include computer storage media and communication media.
[0027] Computer storage media means any medium that can be used to store computer-readable information. By way of example and not limitation, computer storage media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, and any other volatile or non-volatile removable or non-removable media implemented in any technology. Computer storage media excludes signals themselves and transitory forms of signal transmission.
[0028] A communication medium refers to any medium capable of communicating computer readable information. By way of example, and not limitation, communication media may include coaxial cables, fiber optic cables, air, or any other medium suitable for communicating electrical, optical, radio frequency (RF), infrared, acoustic, or other types of signals. Working Example
[0029] In the following, examples useful for understanding the technology disclosed in the present application are presented. The embodiment of the technology may include one or more of the examples described below and any combination thereof.
[0030] A first embodiment of the present invention is a test and measurement device, comprising: a signal generating unit for generating a test signal to be transmitted to an adaptive system under test (SUT), a signal analyzing unit for receiving a response signal from the SUT, one or more communication ports for transmitting the test signal to the SUT and receiving the response signal, and one or more processors, the one or more processors being configured to execute a program that causes the one or more processors to perform the following processes: transmitting configuration data to the signal generating unit for causing the signal generating unit to generate the test signal, transmitting the test signal to the SUT, receiving response signal data of the response signal from the signal analyzing unit, measuring performance of the response signal in response to the test signal, and reporting the performance to at least one of the user workspaces of the SUT and the test and measurement device. Note that the test signal may be a signal that adversely affects a signal transmitted by the SUT before transmitting the response signal.
[0031] A second embodiment is the test and measurement device of the first embodiment, wherein the program that causes the one or more processors to perform a process of measuring the performance of the response signal to the test signal causes the one or more processors to perform a process of comparing characteristics of the test signal and the response signal.
[0032] Example 3 is the test and measurement instrument of example 1, in which the configuration data includes one or more of a power level, a pulse width, a pulse rate, a frequency, and a coding scheme.
[0033] Example 4 is a test and measurement device of any of Examples 1 to 3, wherein the program that causes the one or more processors to perform a process for measuring the performance causes the one or more processors to perform a process for measuring one or more of power, bandwidth, frequency, and pulse shape.
[0034] Example 5 is a test and measurement device of any of Examples 1 to 4, wherein the one or more processors are configured to execute a program that causes the one or more processors to perform a process of requesting a second test signal and receiving a message from the SUT indicating that the SUT is ready to receive the second test signal.
[0035] Example 6 is the test and measurement instrument of example 4, wherein the one or more processors are further configured to adjust a setting of the signal generator before transmitting the second test signal.
[0036] Example 7 is the test and measurement device of any of Examples 1 to 6, wherein the one or more processors are further configured to execute a program that causes the one or more processors to perform a process of analyzing the response signal and a process of determining a setting for a second test signal.
[0037] Example 8 is the test and measurement instrument of any of examples 1-7, wherein the one or more communications ports include a control interface to the SUT.
[0038] Example 9 is the test and measurement instrument of Example 8, wherein the user workspace is accessible from the control interface.
[0039] Example 10 is the test and measurement instrument of any of Examples 1 to 9, further comprising one or more antennas coupled to the one or more communication ports.
[0040] Example 11 is a method for testing an adaptive system under test (SUT), comprising the steps of generating a test signal in a signal generation unit of a test and measurement equipment and transmitting the test signal to the SUT, receiving a response signal from the SUT in a signal analysis unit of the test and measurement equipment, measuring performance of the response signal in response to the test signal, and reporting the performance to at least one of the SUT and a user workspace on the test and measurement equipment.
[0041] Example 12 is the method of example 11, in which the generating and transmitting the test signal by the signal generating unit includes determining a setting for the signal generating unit.
[0042] Example 13 is the method of example 12, wherein the settings include one or more of a power level, a pulse width, a pulse rate, a frequency, and a coding scheme.
[0043] Example 14 is the method of any of examples 11 to 13, wherein the measuring the performance includes measuring one or more of a power level, a pulse width, a frequency, and a coding scheme.
[0044] Example 15 is the method of any of examples 11-15, further comprising receiving a message from the SUT requesting a second test signal.
[0045] Example 16 is the method of example 15, wherein the message includes a new setting for the second test signal.
[0046] Example 17 is any of the methods of examples 11 to 16, further comprising: analyzing the response signal; and determining a setting for a second test signal within the user workspace of the test and measurement instrument.
[0047] Example 18 is any of the methods of Examples 11 to 17, wherein the process of receiving the response signal includes a process of receiving the response signal wirelessly or a process of receiving the response signal through a wire connected to the SUT.
[0048] Example 19 is the method of example 11, in which measuring the performance of a response signal to the test signal includes comparing characteristics of the test signal and the response signal.
[0049] In addition, the present description refers to specific features. It should be understood that the disclosure herein includes all possible combinations of those specific features. When a specific feature is disclosed in connection with a particular aspect or embodiment, that feature can also be used in connection with other aspects and embodiments, to the extent possible.
[0050] Furthermore, when the application refers to a method having two or more defined steps or processes, those defined steps or processes may be performed in any order or simultaneously, unless the context precludes such a possibility.
[0051] All features disclosed in the specification, claims, abstract and drawings, and all steps in any disclosed method or process, may be combined in any combination, except where at least some of such features or steps are mutually exclusive combinations. Each feature disclosed in the specification, abstract, claims and drawings may be replaced by an alternative feature serving the same, equivalent or similar purpose, unless expressly stated otherwise.
[0052] For convenience of illustration, specific embodiments of the invention have been illustrated and described, but it will be understood that various modifications can be made therein without departing from the spirit and scope of the invention. Accordingly, the invention should not be limited, except as by the appended claims. [Explanation of symbols]
[0053] 10 Test and measurement equipment 12 RF signal generation section 14 Communication Ports 15 Antenna 18 Communication Ports 19 Antenna 20 Adaptive RF System (System Under Test: SUT) 22 Signal Processing and System Control Unit 23 Antenna 24 communication ports 26 Transmitter 28 Receiver 30 Machine Learning Systems 32 Signal Conditioning Unit 40 processors 42 Memory 44 Ports 46 User Workspaces 100 Test Systems
Claims
1. 1. A test and measurement device comprising: a signal generator for generating a test signal to be transmitted to an adaptive system under test (SUT); A signal analyzer for receiving a response signal from the SUT; one or more communication ports for transmitting said test signals to said SUT and receiving said response signals; One or more processors Equipped with the one or more processors a process of transmitting setting data to the signal generating unit for causing the signal generating unit to generate the test signal; transmitting the test signal to the SUT; receiving response signal data of the response signal from the signal analysis unit; measuring the performance of said response signals to said test signals; reporting the performance to at least one of a user workspace of the SUT and the test and measurement instrument; a test and measurement instrument configured to execute a program that causes the one or more processors to:
2. 2. The test and measurement instrument of claim 1, wherein the test signal is a signal that adversely affects a signal that was being transmitted by the SUT prior to transmitting the response signal.
3. 2. The test and measurement instrument of claim 1, wherein the program that causes the one or more processors to perform a process for measuring the performance of the response signal to the test signal causes the one or more processors to perform a process for comparing characteristics of the test signal and the response signal.
4. 2. The test and measurement instrument of claim 1, wherein the program that causes the one or more processors to measure performance of the response signal to the test signal causes the one or more processors to measure one or more of the power, bandwidth, frequency, pulse shape and coding scheme of the response signal.
5. 2. The test and measurement instrument of claim 1, wherein the one or more processors are further configured to execute a program that causes the one or more processors to analyze the response signal and determine a setting for a second test signal.
6. 1. A method for testing an adaptive system under test (SUT), comprising: A process of generating a test signal in a signal generating unit of a test and measurement device and transmitting the test signal to the SUT; receiving a response signal from the SUT in a signal analysis portion of the test and measurement instrument; measuring the performance of said response signals to said test signals; reporting the performance to at least one of the SUT and a user workspace on the test and measurement instrument; A method for testing an adaptive system under test comprising:
7. 7. The method of claim 6, wherein said test signal is a signal that adversely affects a signal that was being transmitted by said SUT prior to transmitting said response signal.
8. 7. The method of claim 6, wherein measuring the performance of said response signal to said test signal comprises measuring one or more of the power, bandwidth, pulse width, frequency and coding scheme of said response signal.
9. 7. The method of testing an adaptive system under test of claim 6, further comprising receiving a message from said SUT requesting a second test signal.
10. analyzing the response signal; determining a configuration for a second test signal within the user workspace of the test and measurement instrument; 7. The method of testing an adaptive system under test of claim 6, further comprising:
Citation Information
Patent Citations
US11,047,957
