Tensor construction based domain adaptation multi-modal large language model for calibration and measurement

Abstract

A test and measurement instrument comprising: a port to allow the test and measurement instrument to be connected to a device under test (DUT) to receive a signal from the DUT; one or more analog-to-digital converters (ADCs) to receive a signal from the DUT and convert the signal to one or more digital waveforms; a user interface to allow a user to enter a query; and one or more processors configured to execute code that causes the one or more processors to: construct one or more images of the one or more digital waveforms from the one or more ADCs; send the one or more images to a domain adapted multi-modal large language model (MLLM); receive parameters from the domain adapted MLLM; provide the parameters for the DUT to the user in response to the query; and apply the parameters to the DUT.

Description

[0001] Cross-reference to related applications This disclosure is a non-provisional application filed on August 28, 2024, entitled “DOMAIN-ADAPTED MULTIMODAL LARGELANGUAGE MODELS BASED ON TENSOR BUILD FOR CALIBRATION AND MEASUREMENT”, and claims the benefit thereof, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to test and measurement systems, and more particularly to test and measurement systems that employ machine learning (ML) to calibrate and / or measure a device under test (DUT). Background Technology

[0003] Communication systems typically include transmitters and receivers. As signal speeds increase, more complex equalizers are used in both the transmitter and receiver to improve system performance. Transmitter tuning and receiver equalizer adaptation involve more optimization, and system performance measurement involves more computation. U.S. Patent Application Publication No. 20220373598 (hereinafter referred to as "598 Publication"), entitled "SHORT PATTERNWAVEFORM DATABASE BASED MACHINE LEARNING FOR MEASUREMENT," published November 24, 2022, the contents of which are hereby incorporated herein by reference in their entirety, discloses a machine learning (ML) system based on a short-mode waveform tensor builder, which, among other uses, can be used for tuning and measuring devices under test (DUTs), such as transmitters. The short-mode waveform tensor builder-based ML system is designed to improve measurement speed by using a short-mode waveform tensor as a feature representation of a long waveform obtained from the DUT, and then performing transfer learning to obtain a mapping from the 2-D tensor image to the measurement result.

[0004] ML systems based on short-mode waveform tensors have been extended to cover other wired and wireless communication applications. For example, U.S. Patent Application No. 18 / 754,871 (hereinafter referred to as "871 application"), filed June 26, 2024, entitled "MULTIPLE PULSE EXTRACTION FOR TRANSMITTERCALIBRATION," the contents of which are hereby incorporated herein by reference in their entirety, discloses techniques for extracting multiple linearly fitted impulse responses from waveforms. For example, as... Figure 1 and Figure 2As shown, multiple linearly fitted impulse responses (MLFPRs) are extracted from the PAM4 waveform.

[0005] U.S. Patent Application Publication No. 20240169210 (hereinafter referred to as "210 Publication"), entitled "METHODS FOR 3D TENSOR BUILDER FOR INPUT TOMACHINE LEARNING" and published on May 23, 2024, the contents of which are hereby incorporated herein by reference in their entirety. As disclosed in "210 Publication," the extracted pulse and vertical noise histograms can be constructed into a 3-D tensor. In the 3-D tensor, the intensity of a pixel represents the amplitude of a sample in the pulse or the number of hits in the vertical histogram. The 3-D tensor is represented in an image, an example of which is shown in... Figure 3 As shown in the diagram. In this particular example, in the gray lines at the bottom (such as 14), the darker colors represent the high amplitude of the linearly fitted pulses. Line 12 at the top represents a histogram of the number of hits of the waveform represented by this tensor image. The current architecture uses convolutional neural networks (CNNs), such as... Figure 4 The convolutional neural network shown here, where the convolutional neural network is derived from Figure 3 The image is used as input to the model. The CNN has an input layer 20, hidden layers (such as 22), and an output layer 24.

[0006] This method utilizes transfer learning because pre-trained CNN models (such as ResNet) have the ability to extract features from images. The transfer learning process adapts fully connected layers and output layers to form an accurate mapping from the input data image to the corresponding label, thereby producing the measurement result. This example of an ML system has been adapted to handle calibration and representation applications.

[0007] However, this ML system has limitations. For the same waveform, if different measurement parameters, calibration parameters, or characterization parameters are required, the ML model needs to be retrained, or a separate ML model needs to be built. Attached Figure Description

[0008] Figure 1 An example of a Pulse Amplitude Modulation-4 (PAM4) waveform is shown.

[0009] Figure 2 Examples of four linearly fitted impulse responses from a PAM4 waveform are shown.

[0010] Figure 3 An example of an image representing a 3-D tensor is shown.

[0011] Figure 4 An example of a convolutional neural network architecture is shown.

[0012] Figure 5 An example of a test and measurement instrument is shown.

[0013] Figure 6 An example of a current machine learning system is shown.

[0014] Figure 7 An example of a multimodal large language model (MLLM) undergoing training for calibration, characterization, and measurement is shown.

[0015] Figure 8 An example of an MLLM model in use is shown. Detailed Implementation

[0016] The embodiments described here relate to novel systems and methods using short-mode waveform tensors and their derivatives (all of which generate images). The embodiments use text to represent measurement parameters, calibration parameters, and characterization parameters.

[0017] Text-based Large Language Models (LLMs) possess a wealth of knowledge; for example, they can summarize a long novel. An LLM understands that a brief summary of a novel is highly correlated with the full-length version. Beyond summaries, a pre-trained LLM has other understandings of a novel without additional training. Further developments from LLMs have led to Multimodal Large Language Models. As used herein, the term "Multimodal Large Language Model" (MLLM) refers to a model that combines the capabilities of large language models (such as Chat-GPT) with modalities other than text, such as audio, video, and images. Examples of these models include CLIP (Contrastive Language-Image Pre-trained), a visual-language model.

[0018] The example performs domain adaptation on a multimodal large language model (MLLM) that has been pre-trained with publicly available information prior to adaptation. Utilizing additional domain-specific training, the example creates a foundational model for domain adaptation, used for measurement, calibration, and characterization.

[0019] Currently available MLLMs contain a wealth of knowledge across several modalities. In addition to pre-training MLLMs using generally available public data, domain adaptation allows MLLMs to be adapted to specific domains. Domain-adapted MLLMs possess greater intelligence and can generate more accurate results. These domain-adapted MLLMs can be deployed in test and measurement environments as local MLLMs on test and measurement instruments, or located where they can be accessed by the instruments.

[0020] Figure 5An embodiment of test and measurement instrument 30 is illustrated. A device under test (DUT) 32 generates a waveform, which the test and measurement instrument captures. The test setup may include test and measurement instruments such as an oscilloscope 30. The test and measurement instrument 30 receives signals from the DUT 32 directly or via an instrument probe 34. In the case of an optical transmitter DUT, the probe will typically include a test fiber coupled to a photoelectric converter (not shown) that provides signals to the test and measurement instrument through one or more ports 36. Two ports may be used for differential signal transmission, while one port may be used for single-ended signal transmission. These signals are sampled and digitized by one or more analog-to-digital converters (ADCs) 38 to become digital waveforms.

[0021] The test and measurement instrument has one or more processors, represented by processor 42, memory 44, and user interface 40. The memory may store executable instructions in the form of code, which, when executed by the processor, cause the processor to perform tasks. The user interface 40 of the test and measurement instrument allows the user to interact with the instrument 30, such as inputting settings, configuring tests, and making queries (which will be discussed in more detail later).

[0022] The embodiments described herein employ machine learning in the form of a machine learning system 48. The machine learning network may include a processor programmed with an MLLM, which is either part of a test and measurement instrument or accessible to the test and measurement instrument. As test equipment capabilities and processors evolve, one or more processors (such as 42) may include both. The machine learning system may take the form of a programmed model operating on one or more processors.

[0023] In current ML systems (such as Figure 6 In the ML system shown, the ML network 54 is trained on a tensor image 52, for which calibration parameters, representation parameters, and measurement parameters are labeled as simple vectors 56. This results in an efficient ML system for a specific set of cases. For example, an ML model may have been trained to output the tap values ​​of an optimal feedforward equalizer (FFE) with 5 taps. If the user wants the tap values ​​of an FFE with 9 taps, the current model needs to be retrained, or the user must build a separate ML system because 5 taps and 9 taps are completely different for the current ML model. The determination of the FFE taps may include: measurements defined by textual specifications (such as the IEEE 800G / 800G Ethernet standard), or representations of the taps.

[0024] The embodiments in this paper now provide the machine learning system 48 as a domain-adapted MLLM. In the example above, if the domain-adapted MLLM has been trained to produce a 5-tap FFE, the domain-adapted MLLM can understand that the 5-tap FFE and the 9-tap FFE have encoding vectors that are close to each other in terms of cosine distance. The domain-adapted MLLM can then "infer" and produce the value of the 9-tap FFE.

[0025] Figure 7 An embodiment of a domain-adapted MLLM is illustrated during training. The training process involves providing the domain-adapted MLLM with domain-specific training. According to some embodiments of this disclosure, the training process begins with feature extraction. Features of the waveform are extracted and presented in a manner similar to, for example... Figure 6 The current implementation of machine learning in this way is represented by image 76. For example, for Figure 1 The PAM4 signal shown is pulsed and, as shown in the figure, the pulse is extracted ... Figure 2 The image shown is represented here.

[0026] These images are then used to prepare a domain-specific training dataset. The images represent features of the dataset, and calibration parameters, characterization parameters, and measurement parameters are represented by text 72. Each image has a corresponding text entry.

[0027] The pre-trained MLLM (such as CLIP) is refined using a domain-specific dataset by adapting the text encoder 74 to align with the image encoder 78. The alignment between the text modality model and the image modality model is measured. In one embodiment, the alignment of two vectors can be measured using cosine distance. In cosine distance, -1 indicates that the vectors point in opposite directions, so aligned vectors will have a metric very close to or equal to 1. In matrix 80, the center values ​​of the image and text alignment are patterned. Vector alignment results in these intersections having high values.

[0028] After model refinement is complete, a domain-adapted MLLM can be used at runtime to generate results based on one or more input images 76 via an image encoder 78. This is typically accompanied by... Figure 8 This occurs as shown in the query. Domain-adapted MLLMs can receive an image representing the features as input and generate calibration, characterization, and measurement parameters based on user queries about the DUT received through the user interface of the test and measurement instruments. In this case, the text includes the output rather than the input, hence the arrow points in the opposite direction.

[0029] The text decoder outputs text results based on a text vector, which is the same vector encoded based on the image input and query text. The query text indicates the types of calibration parameters, characterization parameters, and measurement parameters. The text encoder 74 converts or decodes the text vector from the domain-adapted MLLM to produce parameters for the DUT.

[0030] Depending on the nature of the parameter, applying a parameter to the DUT can take different forms. For example, a parameter may include characterization parameters that indicate the properties of the DUT. The characterization of the DUT can be compared to the design specifications to see if the design meets the requirements and whether further design changes are needed during the design phase. Characterization may also include information provided during the tuning process as part of a tensor image. When the parameter includes calibration parameters, the instrument can adjust settings or other operating parameters on the DUT to match the values ​​of the calibration parameters. When the parameter includes measurements of the DUT, those measurements can be compared to test specifications rather than design specifications (although these test specifications and design specifications may be the same or similar) to determine whether the DUT passes or fails the test (meaning whether the DUT conforms to or does not conform to the specification). These contents are merely examples and are not intended to limit the scope of the embodiments or claims.

[0031] The embodiments in this paper extend machine learning (ML) applications for calibration, characterization, and measurement to domain-adaptive MLLMs. Text modal models within MLLMs possess a wealth of knowledge and intelligence, with a better ability to understand the meaning of calibration, characterization, and measurement parameters. This allows domain-adaptive MLLMs to handle a wide range of applications without having to train multiple different models using ML methods. Domain-adaptive MLLMs possess more intelligence and can generate more accurate results.

[0032] Various aspects of this disclosure can operate on specially created hardware, on firmware, on a digital signal processor, or on a specially programmed general-purpose computer including a processor that operates according to programmed instructions. As used herein, the terms "controller" or "processor" are intended to include microprocessors, microcomputers, application-specific integrated circuits (ASICs), and special-purpose hardware controllers. One or more aspects of this disclosure can be embodied in computer-usable data and computer-executable instructions executed by one or more computers (including monitoring modules) or other devices, such as in one or more program modules. Typically, program modules include routines, programs, objects, components, data structures, etc., for performing specific tasks or implementing specific abstract data types when executed by a processor or other device in a computer. Computer-executable instructions can be stored on a non-transitory computer-readable medium such as a hard disk, optical disk, removable storage medium, solid-state memory, random access memory (RAM), etc. As those skilled in the art will appreciate, the functionality of program modules can be combined or distributed in various ways as needed. Furthermore, functionality can be embodied, wholly or partially, in firmware or hardware equivalents such as integrated circuits, FPGAs, etc. A particular data structure may be used to more efficiently implement one or more aspects of this disclosure, and such a data structure is intended to be within the scope of computer-executable instructions and computer-usable data described herein.

[0033] In some cases, the disclosed aspects may be implemented in hardware, firmware, software, or any combination thereof. The disclosed aspects may also be implemented as instructions carried on or stored on one or more non-transitory computer-readable media, which may be read and executed by one or more processors. Such instructions may be referred to as a computer program product. As discussed herein, a computer-readable medium means any medium that can be accessed by a computing device. By way of example, and not limitation, a computer-readable medium may include computer storage media and communication media.

[0034] Computer storage media means any medium that can be used to store computer-readable information. By way of example, and not limitation, computer storage media may include RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital video disc (DVD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, and any other volatile or non-volatile, removable or non-removable media implemented in any technology. Computer storage media excludes signals themselves and the transient form of signal transmission.

[0035] Communication medium means any medium that can be used for communication of computer-readable information. By way of example, and not limitation, communication medium may include coaxial cable, fiber optic cable, air, or any other medium suitable for communication of electrical, optical, radio frequency (RF), infrared, acoustic, or other types of signals.

[0036] Example Illustrative examples of the disclosed techniques are provided below. Embodiments of the techniques may include one or more of the examples described below, as well as any combination thereof.

[0037] Example 1 is a test and measurement instrument, comprising: a port for allowing the test and measurement instrument to connect to a device under test (DUT) to receive signals from the DUT; one or more analog-to-digital converters (ADCs) for receiving signals from the DUT and converting the signals into one or more digital waveforms; a user interface for allowing a user to enter queries; and one or more processors configured to execute code that causes the one or more processors to: construct one or more images of one or more digital waveforms from one or more ADCs; send one or more images to a domain-adapted multimodal large language model (MLLM); receive parameters from the domain-adapted MLLM; provide the user with parameters for the DUT in response to a query; and apply the parameters to the DUT.

[0038] Example 2 is the test and measurement instrument of Example 1, wherein the MLLM resides in the test and measurement instrument.

[0039] Example 3 is a test and measurement instrument of Example 1 or 2, wherein the MLLM resides remotely from the test and measurement instrument.

[0040] Example 4 is a test and measurement instrument of any of Examples 1 to 3, wherein the image includes one of a three-dimensional tensor or a two-dimensional tensor.

[0041] Example 5 is a test and measurement instrument for any of Examples 1 through 4, wherein one or more processors are also configured to execute code to train a multimodal large language model (MLLM) to create a domain-adapted MLLM.

[0042] Example 6 is a test and measurement instrument of Example 5, wherein the code that enables one or more processors to train an MLLM includes code that enables one or more processors to upload one or more training datasets, the datasets including: images of features extracted from waveforms, and corresponding text representing at least one of calibration parameters, characterization parameters and measurement parameters of the waveforms.

[0043] Example 7 is a test and measurement instrument of any of Examples 1 to 6, wherein the received parameters are characterization parameters of the DUT, and the code that causes one or more processors to apply the parameters to the DUT includes: code that causes one or more processors to compare the characterization parameters of the DUT with the design specifications to determine whether the design meets the design requirements.

[0044] Example 8 is a test and measurement instrumentation of Example 7, wherein the code that enables one or more processors to compare the characterization parameters of the DUT includes code that enables one or more processors to determine whether additional design changes are required during the design phase.

[0045] Example 9 is a test and measurement instrument of any of Examples 1 through 8, wherein the received parameters are measurement parameters of the DUT, and the code that causes one or more processors to apply the parameters to the DUT includes: code for comparing the measurement parameters with the specification to determine whether the DUT passes or fails.

[0046] Example 10 is a test and measurement instrument of any of Examples 1 through 9, wherein the parameters include calibration parameters, and the code that causes one or more processors to apply the parameters to the DUT includes: code for setting the parameters on the DUT to the values ​​of the calibration parameters.

[0047] Example 11 is a method comprising: receiving a query from a user via a user interface of a test and measurement instrument connected to a device under test (DUT); receiving one or more digital waveforms from one or more analog-to-digital converters (ADCs) in the test and measurement instrument; constructing one or more images of the one or more digital waveforms; sending the one or more images to a domain-adapted multimodal large language model (MLLM); receiving parameters from the domain-adapted MLLM; providing the user with parameters for the DUT in response to the query; and applying the parameters to the DUT.

[0048] Example 12 is a method of Example 11, wherein sending one or more images to a domain-adapted MLLM includes: sending one or more images to a domain-adapted MLLM on a test and measurement instrument.

[0049] Example 13 is a method of Example 11 or 12, wherein sending one or more images to a domain-adapted MLLM includes: sending one or more images to a domain-adapted MLLM located remotely from the test and measurement instruments.

[0050] Example 14 is a method of any of Examples 11 through 13, wherein the image comprises either a three-dimensional tensor or a two-dimensional tensor.

[0051] Example 15 is a method of any of Examples 11 through 14, and also includes: training a multimodal large language model (MLLM) to create a domain-adaptive MLLM.

[0052] Example 16 is the method of Example 15, wherein training the MLLM includes: performing feature extraction on a set of waveforms to produce a set of features; constructing a set of images representing the features; creating one or more training datasets, the one or more training datasets including a set of images and corresponding text for each image, the text including at least one of calibration parameters, characterization parameters and measurement parameters; and uploading the one or more training datasets to the MLLM to create a domain-adapted MLLM.

[0053] Example 17 is a method of any of Examples 11 to 16, wherein the parameters include characterization parameters, and applying the parameters to the DUT includes: comparing the characterization parameters with the design specifications to determine whether the design meets the design requirements.

[0054] Example 18 is the approach of Example 17, and also includes: determining whether additional design changes are needed.

[0055] Example 19 is a method of any of Examples 11 to 18, wherein the parameters include measurement parameters, and applying the parameters to the DUT includes: comparing the measurement parameters with the test specification and determining whether the DUT passes or fails.

[0056] Example 20 is a method of any of Examples 11 to 19, wherein the parameters include calibration parameters, and applying the parameters to the DUT includes setting the parameters on the DUT to the values ​​of the calibration parameters.

[0057] All features disclosed in the specification (including the claims, abstract, and drawings) and all steps in any disclosed method or process may be combined in any combination, except where at least some of such features and / or steps are mutually exclusive combinations. Unless otherwise expressly stated, each feature disclosed in this specification (including the claims, abstract, and drawings) may be replaced by an alternative feature for the same, equivalent, or similar purpose.

[0058] Additionally, specific features are mentioned in this written description. It is to be understood that the disclosure of this specification includes all possible combinations of those specific features. Where a specific feature is disclosed in the context of a particular aspect or example, that feature may also be used in the context of other aspects and examples, to the extent possible.

[0059] Furthermore, when a method having two or more defined steps or operations is mentioned in this application, the defined steps or operations may be performed in any order or simultaneously unless the context precludes those possibilities.

[0060] While specific examples of the invention have been illustrated and described for illustrative purposes, it will be understood that various modifications can be made without departing from the spirit and scope of the invention. Therefore, the invention should not be limited except by the appended claims.

Claims

1. A testing and measuring instrument, comprising: A port for allowing the test and measurement instruments to connect to the device under test (DUT) to receive signals from the DUT; One or more analog-to-digital converters (ADCs) are used to receive signals from the DUT and convert the signals into one or more digital waveforms; The user interface is used to allow users to enter queries. as well as One or more processors are configured to execute code that causes the one or more processors to: Construct one or more images of the one or more digital waveforms from the one or more ADCs; The one or more images are sent to a domain-adaptive multimodal large language model (MLLM). Parameters are received from the MLLM adapted to the domain; In response to the query, provide the user with parameters for the DUT; as well as Apply the parameters to the DUT.

2. The test and measurement instrument of claim 1, wherein the MLLM resides within the test and measurement instrument.

3. The test and measurement instrument of claim 1, wherein the MLLM resides remotely from the test and measurement instrument.

4. The test and measurement instrument according to claim 1, wherein the image comprises one of a three-dimensional tensor or a two-dimensional tensor.

5. The test and measurement instrument of claim 1, wherein the one or more processors are further configured to execute code to train a multimodal large language model (MLLM) to create a domain-adapted MLLM.

6. The test and measurement instrument of claim 5, wherein the code that causes the one or more processors to train the MLLM includes code that causes the one or more processors to upload one or more training datasets, the datasets comprising: An image of features extracted from the waveform, and corresponding text representing at least one of the calibration parameters, characterization parameters, and measurement parameters of the waveform.

7. The test and measurement instrument of claim 1, wherein the received parameters are characterization parameters of the DUT, and the code that causes the one or more processors to apply the parameters to the DUT comprises: The code that enables one or more processors to compare the characterization parameters of the DUT with the design specifications to determine whether the design meets the design requirements.

8. The test and measurement instrument of claim 7, wherein the code that causes the one or more processors to compare the characterization parameters of the DUT comprises: Code that enables the one or more processors to determine whether additional design changes are needed during the design phase.

9. The test and measurement instrument of claim 1, wherein the received parameter is a measurement parameter of the DUT, and the code that causes the one or more processors to apply the parameter to the DUT comprises: Code used to compare the measurement parameters with the specifications to determine whether the DUT passes or fails.

10. The test and measurement instrument of claim 1, wherein the parameters include calibration parameters, and the code that causes the one or more processors to apply the parameters to the DUT includes: Code used to set the parameters on the DUT to the values ​​of the calibration parameters.

11. A method comprising: Queries are received from the user through the user interface of the test and measurement instruments, which are connected to the device under test (DUT). Receive one or more digital waveforms from one or more analog-to-digital converters (ADCs) in the test and measurement instruments; Construct one or more images of the one or more digital waveforms; The one or more images are sent to a domain-adaptive multimodal large language model (MLLM). Parameters are received from the MLLM adapted to the domain; In response to the query, provide the user with parameters for the DUT; as well as Apply the parameters to the DUT.

12. The method of claim 11, wherein sending the one or more images to the domain-adapted MLLM comprises: Send the one or more images to the field-adapted MLLM on the test and measurement instrument.

13. The method of claim 11, wherein sending the one or more images to the domain-adapted MLLM comprises: The one or more images are sent to a field-adapted MLLM located away from the test and measurement instruments.

14. The method of claim 11, wherein the image comprises one of a three-dimensional tensor or a two-dimensional tensor.

15. The method of claim 11, further comprising: Train a multimodal large language model (MLLM) to create a domain-adapted MLLM.

16. The method of claim 15, wherein training the MLLM comprises: Perform feature extraction on the waveform set to generate a feature set; Construct a set of images representing the features; Create one or more training datasets, the one or more training datasets including the image set and corresponding text for each image, the text including at least one of calibration parameters, characterization parameters and measurement parameters; as well as Upload the one or more training datasets to the MLLM to create the domain-adapted MLLM.

17. The method of claim 11, wherein the parameters include characterization parameters, and applying the parameters to the DUT comprises: The characterization parameters are compared with the design specifications to determine whether the design meets the design requirements.

18. The method of claim 17, further comprising: Determine if any further design changes are needed.

19. The method of claim 11, wherein the parameters include measurement parameters, and applying the parameters to the DUT comprises: The measurement parameters are compared with the test specifications to determine whether the DUT passes or fails.

20. The method of claim 11, wherein the parameters include calibration parameters, and applying the parameters to the DUT comprises: Set the parameters on the DUT to the values ​​of the calibration parameters.

Images