System and method for cross-modal signal reasoning using audio signals

Abstract

Systems and methods for converting a primary one-dimensional signal to a secondary one-dimensional signal of another modality. The primary signal is spliced into a plurality of consecutive frames. The first linear transformation transforms the frame into a corresponding vector. Location encoding is provided on the vector to encode relative location information associated with each sample within each frame. The multi-head self-attention machine learning model compares the relative importance of each sample within each vector to each other in that vector to produce a high-level representation vector. A second linear transformation transforms the high level representation vector into a corresponding secondary signal frame. The secondary signal frames are joined into a reconstructed one-dimensional secondary signal having a different modality than the primary signal.

Description

Technical Field

[0001] The present disclosure relates to systems and methods for cross-modal signal reasoning using audio signals. In an embodiment, the present disclosure relates to converting a one-dimensional signal (eg, audio) into other one-dimensional signals by utilizing deep learning techniques. Background Art

[0002] Signal-to-signal conversion is a fundamental concept in engineering that involves the manipulation, analysis, and transformation of signals from one form to another. Signals can take many forms, such as voltage, electromagnetic waves, or even digital data streams. Signal-to-signal conversion is a key aspect of modern technology, enabling a wide range of applications across a variety of engineering disciplines. It plays a central role in fields such as telecommunications, audio processing, control systems, and more. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 A system for training a neural network according to one embodiment is generally illustrated.

[0004] Figure 2 A computer-implemented method for training and utilizing a neural network is generally illustrated according to one embodiment.

[0005] Figure 3A A diagram generally illustrates a signal converter configured to convert a primary signal into a secondary signal using machine learning according to one embodiment.

[0006] Figure 3B Generally, a method for Figure 3A The signal converter performs signal-to-signal conversion in the architecture.

[0007] Figure 3C An example of frame-based signal-to-signal conversion according to the principles of the present disclosure is generally illustrated.

[0008] Figure 3D An example of an original secondary signal and its reconstructed signal converted from a primary signal according to one embodiment is generally illustrated.

[0009] Figure 4 is a flow chart generally illustrating a method of signal conversion according to the principles of the present disclosure.

[0010] Figure 5 A schematic diagram depicting the interaction between a computer-controlled machine and a control system according to one embodiment.

[0011] Figure 6 Describes a method according to an embodiment Figure 5Schematic diagram of a control system configured to control a vehicle, which can be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot, or a fully autonomous robot.

[0012] Figure 7 Depiction Figure 5 Schematic diagram of a control system configured to control a manufacturing machine of a manufacturing system such as part of a production line, such as a punch cutter, a cutting machine, or a gun drill.

[0013] Figure 8 Depiction Figure 5 Schematic diagram of a control system configured to control a power tool, such as a power drill or driver, having an at least partially autonomous mode.

[0014] Fig. 9 Depiction Figure 5 Schematic diagram of a control system configured to control an automated personal assistant.

[0015] Fig.10 Depiction Figure 5 Schematic diagram of a control system configured to control a monitoring system, such as a controlled access system or a surveillance system.

[0016] Fig.11 Depiction Figure 5 Schematic diagram of a control system configured to control an imaging system, such as an MRI device, an x-ray imaging device, or an ultrasound device. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure are described herein. However, it is to be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily drawn to scale; some features may be exaggerated or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as restrictive, but merely as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any one of the figures may be combined with the features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. The combination of the illustrated features provides representative embodiments for typical applications. However, for specific applications or embodiments, various combinations and modifications of features consistent with the teachings of the present disclosure may be desired.

[0018] As used herein, "a", "an", and "the" refer to both singular and plural references unless the context clearly dictates otherwise. For example, a "processor" programmed to perform various functions refers to one processor programmed to perform each function, or more than one processor that are collectively programmed to perform each of the various functions.

[0019] Signal-to-signal conversion is a fundamental concept in engineering that involves the manipulation, analysis, and transformation of signals from one form to another. Signals can take many forms, such as voltage, electromagnetic waves, or even digital data streams. Signal-to-signal conversion is a key aspect of modern technology, enabling a wide range of applications across a variety of engineering disciplines. It plays a central role in fields such as telecommunications, audio processing, control systems, and more.

[0020] Audio to signal conversion involves converting an audio signal (usually an analog waveform representing sound) into another type of signal or modality. This can be a vibration signal, tactile response, tactile feedback, seismic activity, or other signal indicating the operating characteristics of a computer-controlled machine. The benefits of replacing high-cost sensory signals with audio signals have been demonstrated in many applications. This is mainly because audio signals can be obtained much more economically and the installation of acoustic sensors is flexible.

[0021] Traditional signal processing and linear transformation methods are often used as the default approach to establish a transformation estimate between a reference signal and its description. While these conventional methods are effective in certain applications, they present several challenges. For example, adaptability issues may arise for real-time adaptation to new, unseen signals. In addition, accurate modeling of nonlinear signals can be problematic. Furthermore, these methods often require domain knowledge or manual engineering to extract meaningful features from the raw data.

[0022] Thus, the present disclosure provides a data-centric framework for audio-to-signal conversion, leveraging advances in transformer-based architecture design.

[0023] The present disclosure utilizes a transformer-based architecture to perform signal conversion or translation. The present disclosure introduces systems and methods for efficiently converting a one-dimensional signal (e.g., an audio signal) into a variety of other one-dimensional secondary signals. By utilizing advanced deep learning techniques, the disclosed system employs a transformer-based architecture that is capable of capturing positional and global relationships between input and output signals. The proposed method exhibits improved accuracy, flexibility, scalability, and adaptability in converting audio signals into a variety of secondary signals.

[0024] Transformer-based architectures were first introduced in Vaswani, Ashish et al., “Attention is all you need”, Advances in Neural Information Processing Systems 30 (2017). This publication is referred to as Vaswani in the following. The Transformer architecture (a neural network model) in Vaswani revolutionized the field of deep learning, specifically in natural language processing (NLP). A key innovation of the Transformer architecture is the self-attention mechanism, which computes the dependencies between each pair of positions in the input sequence in parallel, capturing global dependencies in the data. The self-attention mechanism allows the model to capture dependencies between words regardless of their position in the sentence. This eliminates the need for recurrent or convolutional layers, making training faster and more parallelizable. Since then, Transformers have become the basis of many state-of-the-art models in NLP, including BERT, GPT, and many others. Moreover, although the Transformer architecture was originally used in the field of NLP, it has also shown strong performance in other fields such as computer vision and signal processing.

[0025] Figure 1 A system 100 is shown for training a neural network (e.g., a deep neural network as part of the backbone of a deep learning model or algorithm). The neural network being trained may be a deep learning network, such as a signal converter model or signal translator model disclosed herein. The system 100 may include an input interface for accessing training data 102 for the neural network. For example, Figure 1 As shown in the figure, the input interface can be constituted by a data storage interface 104, which can access the training data 102 from a data storage device 106. For example, the data storage interface 104 can be a memory interface or a permanent storage interface, such as a hard disk or SSD interface, but can also be a personal network, local area network or wide area network interface, such as a Bluetooth, Zigbee or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage device 106 can be an internal data storage device of the system 100, such as a hard disk drive or SSD, or an external data storage device, such as a network accessible data storage device.

[0026] In some embodiments, data storage 106 may further include a data representation 108 of an untrained version of the neural network, which data representation may be accessed by system 100 from data storage 106. However, it will be appreciated that each of training data 102 and data representation 108 of the untrained neural network may also be accessed from a different data storage, for example, via a different subsystem of data storage interface 104. Each subsystem may be of a type as described above with respect to data storage interface 104.

[0027] In some embodiments, the data representation 108 of the untrained neural network may be generated internally by the system 100 based on the design parameters of the neural network, and thus may not be explicitly stored in the data storage device 106. The system 100 may further include a processor subsystem 110 that may be configured to provide an iterated function as a replacement of a stack of layers of the neural network to be trained during operation of the system 100. Here, the layers of the replaced stack may have mutually shared weights and may receive as input the output of a previous layer, or, for a first layer of the stack, receive an initial activation and a portion of the input of the stack.

[0028] The processor subsystem 110 may further be configured to iteratively train the neural network using the training data 102. Here, iterations of training by the processor subsystem 110 may include a forward propagation portion and a back propagation portion. The processor subsystem 110 may be configured to perform the forward propagation portion by determining, among other operations that define the forward propagation portion that can be performed, an equilibrium point of the iterative function at which the iterative function converges to a fixed point and by providing the equilibrium point as a replacement for the output of the stack in the neural network, wherein determining the equilibrium point includes using a numerical root finding algorithm to find a radical solution of the iterative function minus its input.

[0029] The system 100 may further include an output interface for outputting a data representation 112 of the trained neural network; this data may also be referred to as trained model data 112. For example, as also described in Figure 1 , the output interface may be constituted by the data storage interface 104, which in these embodiments is an input / output ('IO') interface, via which the trained model data 112 may be stored in the data storage device 106. For example, the data representation 108 defining an "untrained" neural network may be at least partially replaced by the data representation 112 of the trained neural network during or after training, as the parameters of the neural network (such as the weights, hyperparameters, and other types of parameters of the neural network) may be adapted to reflect the training based on the training data 102. This is also Figure 1 106 by reference numerals 108, 112 referring to the same data records on the data storage device 106. In other embodiments, the data representation 112 may be stored separately from the data representation 108 defining the "untrained" neural network. In some embodiments, the output interface may be separate from the data storage interface 104, but may generally be of the type described above for the data storage interface 104.

[0030] The structure of system 100 is one example of a system that can be utilized to train a signal converter model or signal translator model described herein. Figure 2Additional structures for operating and training machine learning models are shown in .

[0031] Figure 2 A signal conversion system 200 is depicted for implementing a system for converting or translating a data signal according to one embodiment. The signal conversion system 200 may utilize a signal converter model or a signal translator model disclosed herein. The signal conversion system 200 may include at least one computing system 202. The computing system 202 may include at least one processor 204, which may be operably connected to a memory unit 208. The processor 204 may include one or more integrated circuits that implement the functionality of a central processing unit (CPU) 206. The CPU 206 may be a commercially available processing unit that implements an instruction set such as one of the x86, ARM, Power, or MIPS instruction set families.

[0032] During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. The stored program instructions may include software that controls the operation of CPU 206 to perform the operations described herein. In some examples, processor 204 may be a system on a chip (SoC) that integrates the functionality of CPU 206, memory unit 208, network interfaces, and input / output interfaces into a single integrated device. Computing system 202 may implement an operating system for managing various aspects of operation. Although in Figure 2 One processor 204, one CPU 206, and one memory 208 are shown in FIG. 1 , but of course more than one processor 204, CPU 206, and memory 208 may be utilized in the overall system.

[0033] The memory unit 208 may include volatile memory and non-volatile memory for storing instructions and data. The non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is disabled or loses power. The volatile memory may include static and dynamic random access memory (RAM) that stores program instructions and data. For example, the memory unit 208 may store a machine learning model 210 or algorithm, a training data set 212 for the machine learning model 210, and an original source data set 216. The machine learning model may be or include a signal converter model or a signal translator model disclosed herein.

[0034] The computing system 202 may include a network interface device 222 configured to provide communications with external systems and devices. For example, the network interface device 222 may include a wired and / or wireless Ethernet interface as defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface device 222 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). The network interface device 222 may further be configured to provide a communication interface to an external network 224 or a cloud.

[0035] The external network 224 may be referred to as the World Wide Web or the Internet. The external network 224 may establish a standard communication protocol between computing devices. The external network 224 may allow information and data to be easily exchanged between computing devices and the network. One or more servers 230 may communicate with the external network 224.

[0036] The computing system 202 may include an input / output (I / O) interface 220, which may be configured to provide digital and / or analog input and output. The I / O interface 220 is used to transmit information between an internal storage device and an external input and / or output device (e.g., an HMI device). The I / O 220 interface may include an associated circuit system or bus (BUS) network to transmit information to (one or more) processors and storage devices or to transmit information between (one or more) processors and storage devices. For example, the I / O interface 220 may include a digital I / O logic line that can be read or set by (one or more) processors, a handshake line for supervising data transmission via the I / O line, a timing and counting facility, and other structures known to provide such functions. Examples of input devices include keyboards, mice, sensors, etc. Examples of output devices include monitors, printers, speakers, etc. The I / O interface 220 may include an additional serial interface (e.g., a universal serial bus (USB) interface) for communicating with external devices. The I / O interface 220 may be referred to as an input interface (because it transmits data from an external input such as a sensor), or an output interface (because it transmits data to an external output such as a display).

[0037] The computing system 202 may include a human-machine interface (HMI) device 218, which may include any input device that enables the system 200 to receive control inputs. The computing system 202 may include a display device 232. The computing system 202 may include hardware and software for outputting graphical and textual information to the display device 232. The display device 232 may include an electronic display screen, a projector, a printer, or other suitable device for displaying information to a user or operator. The computing system 202 may be further configured to allow interaction with a remote HMI and a remote display device via a network interface device 222.

[0038] System 200 may be implemented using one or more computing systems 202. While this example depicts a single computing system 202 that implements all of the described features, it is intended that the various features and functions may be separated and implemented by multiple computing units that communicate with each other. The specific system architecture selected may depend on a variety of factors.

[0039] The system 200 can implement a machine learning model 210 or algorithm configured to analyze a raw source data set 216. The raw source data set 216 can include raw or unprocessed sensor data, which can represent an input data set for a machine learning system. The raw source data set 216 can include audio, audio clips, video, video clips, images, text-based information, audio or human speech, time series data (e.g., a pressure sensor signal that varies over time), and raw or partially processed sensor data (e.g., a radar map of an object). Figure 5-11 Several different examples of inputs are shown and described. In some examples, the machine learning model 210 can be or include a neural network algorithm (e.g., a deep neural network) designed to perform a predetermined function. For example, the neural network algorithm can be configured in an automotive application to predict the torque of a motor based on the audio emitted by the motor. The machine learning model 210 may include an algorithm configured to operate a signal converter model or a signal translator model described herein.

[0040] The computing system 202 can store a training data set 212 for the machine learning algorithm 210. The training data set 212 can represent a set of previously constructed data for training the machine learning model 210. The machine learning algorithm 210 can use the training data set 212 to learn weighting factors associated with the neural network algorithm. The training data set 212 may include a set of source data having corresponding achievements or results that the machine learning algorithm 210 attempts to replicate through the learning process. In this example, the training data set 212 may include an input image containing an object (e.g., a street sign). The input image may include various scenes of the recognized object.

[0041] The machine learning algorithm 210 may be operated in a learning mode using the training data set 212 as input. The machine learning algorithm 210 may be subjected to multiple iterations using data from the training data set 212. With each iteration, the machine learning algorithm 210 may update internal weighting factors based on the results obtained. For example, the machine learning algorithm 210 may compare the output results (e.g., a reconstructed or supplemented image in the case where image data is input) with those included in the training data set 212. Since the training data set 212 includes expected results, the machine learning algorithm 210 may determine when the performance is acceptable. After the machine learning algorithm 210 reaches a predetermined performance level (e.g., 100% consistent with the results associated with the training data set 212) or converges, the machine learning algorithm 210 may be executed using data that is not in the training data set 212. It should be understood that in the present disclosure, "convergence" may mean that a set (e.g., predetermined) number of iterations have occurred, or that the residual is sufficiently small (e.g., the change in the approximation probability in the iteration changes by an amount less than a threshold), or other convergence conditions. The trained machine learning algorithm 210 may be applied to a new data set to generate annotated data.

[0042] The machine learning algorithm 210 may be configured to identify specific features in the original source data 216. The original source data 216 may include multiple instances or input data sets that are expected to supplement the results. For example, the machine learning algorithm 210 may be configured to identify a pattern or signature in an audio signal, and convert the audio signal into a secondary signal based on the pattern or signature. The machine learning algorithm 210 may be programmed to process the original source data 216 to identify the presence of specific features. The machine learning algorithm 210 may be configured to identify the features in the original source data 216 as predetermined features. The original source data 216 may be obtained from various sources. For example, the original source data 216 may be actual input data collected by the machine learning system. The original source data 216 may be machine generated for testing the system. As an example, the original source data 216 may include the original audio signal from a microphone or microphone array.

[0043] Figure 3A The use of the signal conversion system 200 is generally illustrated in the form of a diagram. Herein, according to one embodiment, the signal conversion system 200 is configured to convert a primary signal into a secondary signal using machine learning. A primary signal (e.g., an audio signal) is received as an input. A signal converter model 300 (also referred to as a signal translator model, an audio translation model, a transformer model, such as the ML model 210) converts the primary signal into a secondary signal. To do this, the signal converter model relies on a translator-based model that is based on a Figure 3BThe primary signal is further illustrated in and further described herein. In one embodiment, the primary signal is an audio signal such as generated from a microphone. The secondary signal can be one of a plurality of desired modalities. For example, the secondary signal can be a torque signal such that a received sound signal is translated into a torque signal, thereby allowing the system to determine the torque of the device based on the sound emitted by the device.

[0044] Figure 3B An example architecture of a signal converter model 300 is illustrated. The signal converter model 300 is configured to transform an input signal in a first mode or modality into an output signal in a second mode or modality. In general, the signal converter model 300 may be or include an attention-based neural network that performs a machine translation task. The attention-based neural network includes an encoder neural network that receives an input sequence and generates an encoded representation of the input sequence, and outputs a high-level representation that is directly processed by a linear transformation layer to generate a secondary signal.

[0045] At 302, given a primary (eg, audio) signal received (or pre-processed) from a microphone over time T The primary signal is divided or concatenated into consecutive frames, each frame having a size τ. Figure 3C An embodiment of frames of an audio input shown in bounding boxes is shown. The size of each frame may be predetermined and / or may be based on the size (length) of the primary signal. In one embodiment, each frame has the same size τ. Each of these frames may be represented as It can be fed into the transformer model as a separate instance.

[0046] Although the original audio input Figure 3B ' is shown as "fuel pump sound", but this is just one example of a sound generated by a vehicle fuel pump for translation. It should be understood that the primary signal need not be audio, and the secondary signal need not be a torque signal. In fact, it should be recognized that the model described herein can be implemented for translation between any one-dimensional signals, including but not limited to sound, vibration, torque, temperature, heart rate variability, seismic waveforms, bio-signals (e.g., EEG, EMG, EOG), time series data, radar echoes, and others.

[0047] At 304, each input audio frame First, it is transformed into where d is the model dimension. This vector represents the input data for each corresponding frame of audio data and has dimension Each audio frame is independent and processed separately, each audio frame has a single feature vector. Each audio frame may have a common size, for example, τ=512 samples in each frame.

[0048] Position encoding or position embedding is then added at 306. Here, in order to maintain the temporal order of the audio in the input sequence, position encoding is added to the embedding. This allows the model to understand the position of each sound in the audio signal, specifically each position (sample) within each audio frame. In one embodiment, information about the relative position of each data (e.g., audio) sample within the vector or frame is embedded into each vector associated with each frame. Position encoding allows the signal converter model to perform tasks that require sequence understanding well, such as the translation of the one-dimensional signal shown here, where the order of the frames is very important. By combining position encoding, the converter can effectively capture both the content and order of samples in the input data. In one embodiment, sine and cosine functions of different frequencies can be used for position encoding, where each dimension of the position encoding corresponds to a sine curve.

[0049] With position encoding added, at 308, the vector is then passed through a transformer encoder comprising N identical structures. Within each structure, a multi-headed attention mechanism (also known as a multi-headed attention machine learning model) is applied to derive self-attention of the input vector. The attention mechanism can be configured to map a query and a set of key-value pairs to an output, where the query, key, value, and output are all vectors; the output is calculated as a weighted sum of the values, where the weight assigned to each value is calculated by a compatibility function of the query and the corresponding key. The attention mechanism used in this article is a self-attention mechanism because there is only one type of input (e.g., audio). The use of a multi-headed self-attention mechanism allows the neural network to compare each individual position (sample) with other positions (samples) within a given frame of the input data. For example, referring to Figure 3C , for each frame shown in the bounding box, each sample (e.g., 512 samples per frame) is compared to all other samples in that frame. This allows the translation model to weigh the importance or relevance of samples in each frame. When comparing samples, each head learns a different attention pattern.

[0050] The multi-head self-attention mechanism allows the transformer model to capture the complex relational dependencies between positions (samples) within each audio frame. One idea behind multi-head self-attention is to have multiple "heads" or different perspectives on how frames in the input data are related. These heads work together to better understand the entire audio input. Each head focuses on different aspects of the relationship between samples. To make this successful, frames in the input data are used to create three vectors: query, key, and value. Each head gets these query, key, and value vectors and calculates how much attention each sample should give to each other sample in the frame. The score of each sample can be derived from this. For example, one part of the audio data may be more important than another part of the audio data for translation purposes. The attention score determines how much each sample should contribute to the output. In other words, some samples are weighted more than other samples. And, for multiple heads, each head learns different attention patterns associated with different samples of the input data. After each head has completed its own calculations, their results are combined. This mixture of different attention patterns helps the model capture a variety of relationships between samples within each frame.

[0051] like Figure 3B As shown in , the transformer encoder at 308 has one or more addition and normalization features, or layer normalization. When applied, these improve the training stability and performance of the model. For addition, after each layer in the transformer (e.g., a self-attention layer or a feedforward layer), the output of that layer is added to the input of that layer. This is called a residual connection or a skip connection. This retains the original information from the input, making it easier for the model to learn. After the addition step, layer normalization is applied, which involves normalizing the values ​​in the output of the layer to have a mean and standard deviation of 1. Layer normalization helps stabilize the learning process by ensuring that the inputs of each layer are within a similar range of values.

[0052] In short, the self-attention mechanism splices the input into subspaces and then computes the relationships between these splices. The self-attention aspect of the model allows the model to compute the relationship between each sample (position) and all other samples (positions). The model works to learn the relationship between different samples (positions) in the data sequence.

[0053] At 310, the high-level representation vector output from the self-attention mechanism is passed through a linear transformation layer to make the vector have dimensions equal to the secondary signal dimensions (e.g., torque signal). Since the primary and secondary signals have different sampling rates, the dimensions can be calculated by the sampling rates of the two domain signals. For example, the linear transformation layer output shape is Here, f aand f t Represent the sampling rates of the primary signal and the secondary signal respectively.

[0054] At 312, the secondary signal frames are then concatenated to obtain a reconstructed signal Or a secondary signal. This reconstructed signal represents an estimated signal or a secondary signal based on the primary signal. In one example, the reconstructed signal is a torque (Tor) signal indicating a torque associated with a device (eg, a vehicle) emitting the sound used as input.

[0055] Figure 3C An example of frame-based signal-to-signal conversion using the above system is shown. In this embodiment, the input signal is an audio signal and the output signal is a torque signal. Other input and output signals can be used and derived by using the signal converter model disclosed herein. Within each bounding box is a signal that is passed through Figure 3B Sound frame of the transformer model.

[0056] Figure 3D An example of a predicted secondary signal and an actual secondary signal is shown. The ground truth signal (shown here as the "true torque" signal) is the input to the model, and the reconstructed signal or secondary signal is the output of the model. The predicted secondary signal is the result of a transformed primary signal. For example, a transformation model performed on an audio signal produces a predicted torque signal. The true torque signal is the result of an actual torque sensor measurement, such as from a torque sensor. Figure 3D As shown in , the predicted secondary signal and the actual secondary signal are very close, indicating the high performance of the transformation model.

[0057] In some embodiments, the original audio signal (i.e., in the time domain) may undergo a transformation into the frequency domain, such as a spectrogram, before being input into the signal converter model. In this case, a direct process for generating a spectrogram may be applied, followed by a subsequent step of reshaping or flattening the resulting two-dimensional spectrogram into a one-dimensional sequence. For example, this sequence may be formed by concatenating each time slice of the spectrogram.

[0058] In some embodiments, variants of the Transformer architecture may be adopted for modeling, such as Conformer, which is a hybrid CNN-Transformer model, etc.

[0059] Thus, the methods and systems disclosed herein provide a model that is data-centric and based on machine learning / deep learning methods to convert a one-dimensional signal (such as audio) to a different one-dimensional signal modality and vice versa. This model allows tracking and potential modification of signal representations to achieve seamless transitions between modalities. It should be understood that the same model described herein can be used to convert secondary signals (e.g., torque) to primary signals (e.g., audio).

[0060] The methods and systems disclosed herein can employ a transformer-based architecture as the core conversion model, which includes flexible frame sizes for optimal adaptability. The framework simultaneously handles the relationship between the two modalities at the frame level. This process ensures high accuracy and high speed of reconstruction of the secondary signal. This provides a direct extension to other one-dimensional signal-to-signal translations.

[0061] The models disclosed herein can be trained with cross-modal data. For example, the input signal during training can be audio, and the resulting secondary signal can be a representative signal of torque, where the actual torque value is used to compare the secondary signal. The model can be trained until convergence, i.e., when the secondary signal calculated by the model is within a threshold of the actual signal generated by the sensor.

[0062] Figure 4 A method 400 of transforming a primary signal into a secondary signal of another modality according to one embodiment is illustrated. The method may be executed by one or more processors described herein in dependence on instructions stored in a memory described herein.

[0063] At 402, a processor receives a primary signal. The primary signal may be an audio signal or other one-dimensional signal. For example, the audio signal may be received from a microphone or pre-processed by an associated processor.

[0064] At 404, the processor concatenates or segments the primary signal into a plurality of frames. Each frame may have the same size τ, which may depend on the size of the signal. Thus, the signal of each frame may be represented as

[0065] At 406, the processor performs a first linear transformation. In an embodiment, the linear transformation layer transforms the signal of each frame into a signal belonging to where d is the model dimension.

[0066] At 408, the processor performs position encoding to maintain the time order of the primary signal. In one embodiment, sine and cosine functions of different frequencies may be used for position encoding, where each dimension of the position encoding corresponds to a sinusoid.

[0067] At 410, the processor executes a transformer encoder comprising N structures, each having a multi-head self-attention mechanism. In an embodiment, the multi-head self-attention mechanism is configured to map a query and a set of key-value pairs to an output, where the query, key, value, and output are all vectors; the output is calculated as a weighted sum of the values, where the weight assigned to each value is calculated by a compatibility function of the query and the corresponding key. This results in a high-level representation vector

[0068] At 412, the processor performs a secondary linear transformation on the high-level representation vector so that the vector has dimensions equal to the secondary signal dimensions.

[0069] At 414, the processor concatenates the secondary signal frames to obtain a single reconstructed secondary signal. The secondary signal may have a different modality than the primary signal.

[0070] Figure 5 Schematic diagram depicting the interaction between a computer-controlled machine 500 and a control system 502. The computer-controlled machine 500 includes an actuator 504 and a sensor 506. The actuator 504 may include one or more actuators, and the sensor 506 may include one or more sensors. The sensor 506 is configured to sense a condition of the computer-controlled machine 500. The sensor 506 may be configured to encode the sensed condition into a sensor signal 508 and transmit the sensor signal 508 to the control system 502. The sensor 506 may be one or more of the sensors described above; non-limiting examples of the sensor 506 include a microphone, a camera, a video, a radar, a LiDAR, an ultrasonic sensor, and a motion sensor. In one embodiment, the sensor 506 is a microphone configured to generate an audio signal of an environment within or near the computer-controlled machine 500.

[0071] The control system 502 is configured to receive sensor signals 508 from the computer controlled machine 500. As explained below, the control system 502 may be further configured to calculate actuator control commands 510 based on the sensor signals and transmit the actuator control commands 510 to the actuators 504 of the computer controlled machine 500.

[0072] like Figure 5As shown in , the control system 502 includes a receiving unit 512. The receiving unit 512 can be configured to receive the sensor signal 508 from the sensor 506 and transform the sensor signal 508 into an input signal x. In an alternative embodiment, the sensor signal 508 is directly received as the input signal x without the receiving unit 512. Each input signal x can be a part of each sensor signal 508. The receiving unit 512 can be configured to process each sensor signal 508 to generate each input signal x. The input signal x can include data corresponding to the image recorded by the sensor 506.

[0073] The control system 502 includes a classifier 514. The classifier 514 can be configured to classify the input signal x into one or more labels using a machine learning algorithm (such as the neural network described above). The classifier 514 is configured to be parameterized by parameters, such as those described above (e.g., parameter θ). The parameter θ can be stored in and provided by the non-volatile storage device 516. The classifier 514 is configured to determine the output signal y from the input signal x. Each output signal y includes information that assigns one or more labels to each input signal x. The classifier 514 can transmit the output signal y to a conversion unit 518. The conversion unit 518 is configured to convert the output signal y into an actuator control command 510. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504, which is configured to actuate the computer-controlled machine 500 in response to the actuator control command 510. In another embodiment, the actuator 504 is configured to actuate the computer-controlled machine 500 directly based on the output signal y.

[0074] After the actuator 504 receives the actuator control command 510, the actuator 504 is configured to perform an action corresponding to the associated actuator control command 510. The actuator 504 may include control logic configured to transform the actuator control command 510 into a secondary actuator control command that is utilized to control the actuator 504. In one or more embodiments, the actuator control command 510 may be utilized to control a display instead of or in addition to the actuator.

[0075] In another embodiment, the control system 502 includes the sensor 506 instead of or in addition to the computer-controlled machine 500 including the sensor 506. The control system 502 may also include the actuator 504 instead of or in addition to the computer-controlled machine 500 including the actuator 504.

[0076] like Figure 5 As shown in , the control system 502 also includes a processor 520 and a memory 522. The processor 520 may include one or more processors. The memory 522 may include one or more memory devices. The classifier 514 of one or more embodiments (e.g., a machine learning algorithm, such as those described above with respect to the pre-trained classifier 306) can be implemented by the control system 502, which includes a non-volatile storage device 516, a processor 520, and a memory 522.

[0077] The non-volatile storage 516 may include one or more permanent data storage devices, such as a hard drive, an optical drive, a tape drive, a non-volatile solid-state device, a cloud storage device, or any other device capable of permanently storing information. The processor 520 may include one or more devices selected from a high-performance computing (HPC) system, which includes a high-performance core, a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field programmable gate array, a programmable logic device, a state machine, a logic circuit, an analog circuit, a digital circuit, or any other device that manipulates signals (analog or digital) based on computer executable instructions residing in the memory 522. The memory 522 may include a single memory device or multiple memory devices, including but not limited to random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0078] The processor 520 may be configured to read into the memory 522 and execute computer executable instructions residing in the non-volatile storage 516 and embodying one or more machine learning algorithms and / or methods of one or more embodiments. The non-volatile storage 516 may include one or more operating systems and applications. The non-volatile storage 516 may store programs compiled and / or interpreted from computer programs created using various programming languages ​​and / or technologies, including but not limited to and alone or in combination Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL.

[0079] When executed by the processor 520, the computer executable instructions of the non-volatile storage device 516 can cause the control system 502 to implement one or more of the machine learning algorithms and / or methods disclosed herein. The non-volatile storage device 516 can also include machine learning data (including data parameters) that support the functions, features, and processes of one or more embodiments described herein.

[0080] The program code embodying the algorithm and / or method described herein can be distributed separately or jointly as a program product in various forms. The program code can be distributed using a computer-readable storage medium having a computer-readable program instruction thereon to prompt the processor to perform aspects of one or more embodiments. The essentially non-temporary computer-readable storage medium may include volatile and non-volatile and removable and non-removable tangible media implemented by any method or technology for the storage of information (such as computer-readable instructions, data structures, program modules or other data). The computer-readable storage medium may further include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technology, portable compact disk read-only memory (CD-ROM) or other optical storage device, cassette, tape, magnetic disk storage device or other magnetic storage device, or any other medium that can be used to store the desired information and can be read by a computer. The computer-readable program instruction can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device or another device, or downloaded to an external computer or external storage device via a network.

[0081] Computer-readable program instructions stored in a computer-readable medium can be used to instruct a computer, other types of programmable data processing devices, or other devices to operate in a particular manner so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions, actions, and / or operations specified in the flowchart or chart. In certain alternative embodiments, the functions, actions, and / or operations specified in the flowchart and chart can be reordered, processed serially, and / or processed simultaneously in accordance with one or more embodiments. In addition, any flowchart and / or chart may include more or fewer nodes or blocks than those described in accordance with one or more embodiments.

[0082] A process, method or algorithm may be implemented in whole or in part using suitable hardware components, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a state machine, a controller or other hardware components or devices, or a combination of hardware, software and firmware components.

[0083] The models and teachings provided herein can be deployed and utilized in a variety of settings. Figure 6A schematic diagram depicting a control system 502 configured to control a vehicle 600, which may be an at least partially autonomous vehicle or an at least partially autonomous robot. The vehicle 600 includes an actuator 504 and a sensor 506. The sensor 506 may include one or more microphones, video sensors, cameras, radar sensors, ultrasonic sensors, LiDAR sensors, and / or location sensors (e.g., GPS). One or more of the one or more specific sensors may be integrated into the vehicle 600. In the context of sign recognition and processing as described herein, the sensor 506 is a camera installed into or integrated into the vehicle 600. Alternatively or in addition to the one or more specific sensors identified above, the sensor 506 may include a software module that is configured to determine the state of the actuator 504 when executed. A non-limiting example of a software module includes a weather information software module that is configured to determine the current or future state of the weather near the vehicle 600 or other location.

[0084] The classifier 514 of the control system 502 of the vehicle 600 may be configured to detect an object near the vehicle 600 based on the input signal x. In such an embodiment, the output signal y may include information characterizing that the object is near the vehicle 600. The actuator control command 510 may be determined based on this information. The actuator control command 510 may be used to avoid a collision with the detected object.

[0085] In embodiments where the vehicle 600 is at least partially autonomous, the actuator 504 may be embodied in the brakes, propulsion system, engine, drive train, or steering system of the vehicle 600. Actuator control commands 510 may be determined such that the actuator 504 is controlled such that the vehicle 600 avoids colliding with a detected object. The detected objects may also be classified according to what the classifier 514 considers them to be most likely to be, such as a pedestrian or a tree. The actuator control commands 510 may be determined based on the classification. In the event of a possible hostile attack, the above system may be further trained to better detect objects or recognize changes in lighting conditions or angles for sensors or cameras on the vehicle 600.

[0086] In other embodiments where the vehicle 600 is an at least partially autonomous robot, the vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot may be an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot. In such embodiments, actuator control commands 510 may be determined such that a propulsion unit, a steering unit, and / or a braking unit of the mobile robot may be controlled such that the mobile robot may avoid colliding with an identified object.

[0087] In another embodiment, the vehicle 600 is an at least partially autonomous robot in the form of a gardening robot. In such an embodiment, the vehicle 600 may use an optical sensor as a sensor 506 to determine the state of plants in the environment near the vehicle 600. The actuator 504 may be a nozzle configured to spray a chemical. Depending on the identified type and / or the identified state of the plant, the actuator control command 510 may be determined to cause the actuator 504 to spray the plant with an appropriate amount of an appropriate chemical.

[0088] The vehicle 600 may be an at least partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include a washing machine, a stove, an oven, a microwave, or a dishwasher. In such a vehicle 600, the sensor 506 may be an optical sensor configured to detect the state of an object to be processed by the household appliance. For example, in the case where the household appliance is a washing machine, the sensor 506 may detect the state of laundry to be washed inside the washing machine. The actuator control command 510 may be determined based on the detected state of the laundry to be washed.

[0089] Figure 7 A schematic diagram depicting a control system 502 of a manufacturing system 702 (such as part of a production line) configured to control the system 700 (e.g., a manufacturing machine), such as a punch cutter, a cutting machine, or a gun drill. The control system 502 may be configured to control an actuator 504 configured to control the system 700 (e.g., a manufacturing machine).

[0090] The sensor 506 of the system 700 (e.g., a manufacturing machine) may be an optical sensor configured to capture one or more properties of the manufactured product 704. The classifier 514 may be configured to determine the state of the manufactured product 704 based on the one or more captured properties. The actuator 504 may be configured to control the system 700 (e.g., a manufacturing machine) based on the determined state of the manufactured product 704 for subsequent manufacturing steps of the manufactured product 704. The actuator 504 may be configured to control the function of the system 700 (e.g., a manufacturing machine) on the subsequent manufactured product 706 of the system 700 (e.g., a manufacturing machine) based on the determined state of the manufactured product 704.

[0091] Figure 8 Depicted is a schematic diagram of a control system 502 configured to control a power tool 800 , such as a power drill or driver, having an at least partially autonomous mode. The control system 502 may be configured to control an actuator 504 configured to control the power tool 800 .

[0092] The sensor 506 of the power tool 800 can be an optical sensor configured to capture one or more properties of the working surface 802 and / or the fastener 804 driven into the working surface 802. The classifier 514 can be configured to determine the state of the working surface 802 and / or the fastener 804 relative to the working surface 802 based on the one or more captured properties. The state can be that the fastener 804 is flush with the working surface 802. Alternatively, the state can be the hardness of the working surface 802. The actuator 504 can be configured to control the power tool 800 so that the driving function of the power tool 800 is adjusted based on the determined state of the fastener 804 relative to the working surface 802 or one or more captured properties of the working surface 802. For example, if the state of the fastener 804 is flush with the working surface 802, the actuator 504 can interrupt the driving function. As another non-limiting example, the actuator 504 can apply additional or less torque based on the hardness of the working surface 802.

[0093] In another embodiment, the sensor 506 is a microphone configured to generate one-dimensional sound data associated with the power tool 800 when the power tool 800 drives the fastener 804 into the work surface 802. Based on the disclosure provided above, the classifier 514 can be configured to determine the torque of the motor of the tool 800 by converting the sound emitted by the tool into a torque signal.

[0094] Fig. 9 A schematic diagram depicting a control system 502 configured to control an automated personal assistant 900. The control system 502 may be configured to control an actuator 504 configured to control the automated personal assistant 900. The automated personal assistant 900 may be configured to control a household appliance, such as a washing machine, stove, oven, microwave, or dishwasher.

[0095] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive a video image of user 902's gesture 904. The audio sensor may be configured to receive a voice command of user 902.

[0096] The control system 502 of the automated personal assistant 900 can be configured to determine an actuator control command 510 configured to control the system 502. The control system 502 can be configured to determine the actuator control command 510 based on the sensor signal 508 of the sensor 506. The automated personal assistant 900 is configured to transmit the sensor signal 508 to the control system 502. The classifier 514 of the control system 502 can be configured to execute a gesture recognition algorithm to recognize a gesture 904 made by the user 902, determine the actuator control command 510, and transmit the actuator control command 510 to the actuator 504. The classifier 514 can be configured to retrieve information from the non-volatile storage device in response to the gesture 904, and output the retrieved information in a form suitable for receipt by the user 902.

[0097] Fig.10 A schematic diagram depicting a control system 502 configured to control a monitoring system 1000. The monitoring system 1000 can be configured to physically control access through a door 1002. A sensor 506 can be configured to detect scenes relevant to deciding whether to grant access. The sensor 506 can be an optical sensor configured to generate and transmit image and / or video data. The control system 502 can use such data to detect a person's face.

[0098] The classifier 514 of the control system 502 of the monitoring system 1000 can be configured to interpret the image and / or video data by matching the identities of known persons stored in the non-volatile storage device 516 to determine the identity of the person. The classifier 514 can be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504. In this embodiment, the actuator 504 can be configured to lock or unlock the door 1002 in response to the actuator control command 510. In other embodiments, non-physical logical access control is also possible.

[0099] Monitoring system 1000 may also be a surveillance system. In such an embodiment, sensor 506 may be an optical sensor configured to detect a scene under surveillance, and control system 502 is configured to control display 1004. Classifier 514 is configured to determine a classification of a scene, such as whether a scene detected by sensor 506 is suspicious. Control system 502 is configured to transmit actuator control commands 510 to display 1004 in response to the classification. Display 1004 may be configured to adjust displayed content in response to actuator control commands 510. For example, display 1004 may highlight an object that is considered suspicious by classifier 514. Using embodiments of the disclosed system, a surveillance system may predict the appearance of an object at a certain time in the future.

[0100] Fig.11 A schematic diagram depicting a control system 502 configured to control an imaging system 1100, such as an MRI device, an x-ray imaging device, or an ultrasound device. The sensor 506 may be, for example, an imaging sensor. The classifier 514 may be configured to determine a classification of all or part of a sensed image. The classifier 514 may be configured to determine or select an actuator control command 510 in response to a classification obtained by a trained neural network. For example, the classifier 514 may interpret an area of ​​the sensed image as a potential anomaly. In this case, the actuator control command 510 may be determined or selected to cause the display 1102 to display the imaging and highlight the potential abnormal area.

[0101] This specification uses the term "configured to" in conjunction with system and computer program components. For a system of one or more computers configured to perform specific operations or actions, it means that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform those operations or actions. For one or more computer programs configured to perform specific operations or actions, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0102] In this specification, the term "mechanism" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, a mechanism (e.g., a self-attention mechanism) will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific mechanism; in other cases, multiple mechanisms may be installed and run on the same one or more computers. References to multi-head self-attention mechanisms may also be referred to as multi-head self-attention machine learning models.

[0103] Although exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms covered by the claims. The words used in the specification are descriptive words, not limiting, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As described previously, the features of various embodiments can be combined to form further embodiments of the present invention that may not be explicitly described or illustrated. Although various embodiments may have been described as providing advantages or being superior to other embodiments or prior art implementations in terms of one or more desired characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, serviceability, weight, manufacturability, ease of assembly, etc. Therefore, to the extent that any embodiment is described as not as desirable as other embodiments or prior art implementations in terms of one or more characteristics, these embodiments are not outside the scope of the present disclosure and may be desirable for specific applications.

Claims

1. A method for converting an audio signal into a secondary signal of another mode, the method comprising: receiving an audio signal generated from a microphone; splicing the audio signal into a plurality of frames, each frame having a plurality of audio data samples; performing a first linear transformation to transform the frame into a corresponding vector; performing position encoding on the vector to encode relative position information associated with each sample in the vector; performing a transformer encoder on the vectors having encoded position information, wherein the transformer encoder has a multi-head self-attention mechanism configured to compare the relative importance of the vectors to each other to produce a high-level representation vector; performing a second linear transformation to transform the high-level representation vector into a corresponding secondary signal frame; as well as The corresponding secondary signal frames are concatenated into a reconstructed one-dimensional secondary signal having a different modality than the audio signal.

2. The method according to claim 1, wherein: Each frame has the same size τ.

3. The method according to claim 1, wherein: The transformer encoder has an addition and normalization feature, which is configured to add the output of a layer of the transformer encoder to the input of the layer and to normalize the values ​​in the output of the layer.

4. The method according to claim 1, wherein: The multi-head self-attention mechanism includes an attention function that maps a query and a set of key and value pairs to an output, wherein the query, the key, the value, and the output are all vectors.

5. The method according to claim 4, wherein: The multi-head self-attention mechanism is configured to compute the output vector as a weighted sum of the values.

6. The method according to claim 5, wherein: The weight assigned to each value is calculated by a compatibility function of the query with a corresponding key in the keys.

7. The method according to claim 1, wherein: The multi-head self-attention mechanism is configured to calculate a score of each sample, wherein the score represents the relative importance of the sample relative to other samples in the frame.

8. The method according to claim 7, wherein: The fraction is associated with how much each sample should contribute to the output of the transformer encoder.

9. The method according to claim 1, wherein: The secondary signal is a torque signal or a vibration signal.

10. A system for converting a primary one-dimensional signal into a secondary one-dimensional signal of another modality, the system comprising: a processor programmed to execute instructions stored in the memory to: splicing the primary signal into a plurality of consecutive frames; performing a first linear transformation to transform the frame into a corresponding vector; performing position encoding on the vector to encode relative position information associated with each sample, wherein the relative position information is associated with a sequential position of each sample within its corresponding frame; performing a multi-head self-attention mechanism configured to compare the relative importance of samples to each other within their corresponding frames to produce a high-level representation vector; performing a second linear transformation to transform the high-level representation vector into a corresponding secondary signal frame; as well as The secondary signal frames are concatenated into a reconstructed one-dimensional secondary signal having a different modality than the primary signal.

11. The system according to claim 10, wherein: Each frame has the same size.

12. The system according to claim 10, wherein: The multi-head self-attention mechanism has an addition and normalization feature, which is configured to add the output of a layer of the transformer encoder to the input of the layer and normalize the values ​​in the output of the layer.

13. The system according to claim 10, wherein: The multi-head self-attention mechanism includes an attention function that maps a query and a set of key and value pairs to an output, wherein the query, the key, the value, and the output are all vectors.

14. The system according to claim 13, wherein: The multi-head self-attention mechanism is configured to compute the output vector as a weighted sum of the values.

15. The system of claim 14, wherein: The weight assigned to each value is calculated by a compatibility function of the query with a corresponding key in the keys.

16. The system of claim 10, wherein: The multi-head self-attention mechanism is configured to calculate a score of each sample in a corresponding frame, wherein the score represents the relative importance of the sample relative to other samples in the frame.

17. The system of claim 16, wherein: The fraction is associated with how much each sample should contribute to the output of the transformer encoder.

18. The system of claim 10, wherein: The secondary signal is a sound signal.

19. A computer controlled machine comprising the system of claim 10, wherein: The computer controlled machine also includes an actuator configured to control operation of the computer controlled machine based on an output of the system.

20. A computer controlled machine comprising: at least one microphone configured to generate an audio signal; a control system configured to predict the operating characteristic of the computer-controlled machine by translating the audio signal into a secondary one-dimensional signal representative of the operating characteristic of the computer-controlled machine, the control system configured to: splicing the audio signal into a plurality of frames; performing a first linear transformation to transform the frame into a corresponding vector; performing position encoding on the vector to encode relative position information associated with each sample; performing a transformer encoder on the vectors having encoded position information, wherein the transformer encoder has a multi-head self-attention mechanism configured to compare the relative importance of samples to each other within the corresponding vectors to produce a high-level representation vector; performing a second linear transformation to transform the high-level representation vector into a corresponding secondary signal frame; as well as The corresponding secondary signal frames are concatenated into a reconstructed one-dimensional secondary signal having a different modality than the audio signal.

Images