Systems and methods for cross-modal signal inference using audio signals

Abstract

Systems and procedures for converting a one-dimensional primary signal into a one-dimensional secondary signal of a different modality. The primary signal is spliced ​​into a plurality of consecutive frames. A first linear transformation transforms the frames into corresponding vectors. Positional encodings are provided on the vectors to encode relative positional information associated with each sample within each frame. A multi-head self-attentional machine learning model compares the relative importance of the samples within each vector to obtain vectors with high-level representation. A second linear transformation transforms the vectors with high-level representation into corresponding secondary signal frames. The secondary signal frames are concatenated to form a reconstructed one-dimensional secondary signal with a different modality than the primary signal.

Description

Technical field

[0001] This disclosure relates to systems and methods for cross-modal signal inference using audio signals. In embodiments, this disclosure relates to the conversion of one-dimensional signals (e.g., audio) into other one-dimensional signals by exploiting a deep learning technology. background

[0002] Signal-to-signal conversion is a fundamental concept in engineering that encompasses the manipulation, analysis, and transformation of signals from one form to another. Signals can take many forms, such as electrical voltages, electromagnetic waves, or even digital data streams. Signal-to-signal conversion is an essential aspect of modern technology, enabling a wide range of applications across many engineering disciplines. It plays a central role in fields such as telecommunications, audio processing, control systems, and more. Brief description of the drawings Fig. Figure 1 illustrates a general system for training a neural network according to one embodiment. Fig. Figure 2 illustrates a general computer-implemented method for training and using a neural network according to one embodiment. Fig. Figure 3A illustrates a general diagram of a signal converter designed to convert a primary signal into a secondary signal using machine learning, according to one embodiment. Fig. Figure 3B illustrates a general architecture for signal-to-signal conversion, which is implemented by the signal converter consisting of Fig. 3A is executed according to one embodiment. Fig. 3C generally illustrates an example of frame-based signal-to-signal conversion according to the principles of the present disclosure. Fig. Figure 3D generally illustrates an example of an original secondary signal and its reconstructed signal, which is converted from the primary signal, according to one embodiment. Fig. Figure 4 is a flowchart that generally illustrates a signal conversion process according to the principles of the present disclosure. Fig. Figure 5 presents a schematic diagram of an interaction between a computer-controlled machine and a control system according to one embodiment. Fig. Figure 6 presents a schematic diagram of the tax system. Fig. 5, which is designed to control a vehicle that may be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot or a fully autonomous robot, according to one embodiment. Fig. Figure 7 presents a schematic diagram of the tax system. Fig. 5, which is designed to control a manufacturing machine, such as a punching tool, a cutting tool or a gun drill, or a manufacturing system, such as part of a production line. Fig. Figure 8 shows a schematic diagram of the tax system. Fig. 5, which is designed to control a power tool, such as an electric drill or screwdriver, which has at least a partially autonomous mode. Fig. Figure 9 presents a schematic diagram of the tax system. Fig. 5, which is trained to control an automated personal assistant. Fig. Figure 10 shows a schematic diagram of the tax system. Fig. 5, which is trained to control a monitoring system, such as an access control system or an observation system. Fig. Figure 11 presents a schematic diagram of the tax system Fig. 5, which is designed to control an imaging system, for example an MRI device, an X-ray imaging device or an ultrasound device. Detailed description

[0003] Embodiments of the present disclosure are described herein. It is understood, however, that the disclosed embodiments are merely examples and that other embodiments may take different and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or minimized to show details of specific components. Therefore, specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching a person skilled in the art how to use the embodiments in various ways.As an average person skilled in the art understands, various features illustrated and described with reference to any one of the figures can be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described. The combinations of illustrated features provide representative embodiments for typical applications. However, different combinations and modifications of the features in accordance with the teachings of this disclosure may be desired for specific applications or implementations.

[0004] “A”, “an”, and “the”, as used here, refer to both singular and plural forms, unless the context clearly indicates otherwise. For example, “a processor” programmed to perform various functions refers to a processor programmed to perform each individual function, or to more than one processor programmed together to perform each of the different functions.

[0005] Signal-to-signal conversion is a fundamental concept in engineering that encompasses the manipulation, analysis, and transformation of signals from one form to another. Signals can take many forms, such as electrical voltages, electromagnetic waves, or even digital data streams. Signal-to-signal conversion is an essential aspect of modern technology, enabling a wide range of applications across many engineering disciplines. It plays a central role in fields such as telecommunications, audio processing, control systems, and more.

[0006] Audio-to-signal conversion involves converting an audio signal—usually an analog waveform representing sound—into another type of signal or modality. This could be a vibration signal, a tactile response, haptic feedback, seismic activity, or other signals that indicate an operational characteristic of a computer-controlled machine. The advantages of substituting an expensive sensor signal with an audio signal have been demonstrated across numerous applications. This is primarily because audio signals can be obtained much more cost-effectively, and the installation of an acoustic sensor is flexible.

[0007] Conventional signal processing and linear transformation techniques often serve as the standard approach for generating transformation estimates between a reference signal and its representation. Despite their effectiveness in certain applications, these conventional methods present several challenges. For example, adaptability issues can arise when adapting to new, unseen signals in real time. Furthermore, accurately modeling nonlinear signals can be problematic. Additionally, these methods often require specialized knowledge or manual engineering to extract meaningful features from raw data.

[0008] Therefore, the present disclosure provides a data-centric framework for audio-to-signal conversion, leveraging advances in transformer-based architectural designs.

[0009] This disclosure utilizes a transformer-based architecture to perform signal conversion or translation. It introduces systems and methods for efficiently converting a one-dimensional signal (e.g., audio signals) into various other one-dimensional secondary signals. By leveraging advanced deep learning technology, the disclosed systems employ a transformer-based architecture capable of capturing positional and global relationships between input and output signals. The proposed method demonstrates improved accuracy, flexibility, scalability, and adaptability in converting audio signals into a diverse range of secondary signals.

[0010] A transformer-based architecture was first introduced in Vaswani, Ashish et al., “Attention is all you need,” Advances in neural information processing systems 30 (2017). This publication will be referred to as Vaswani. The transformer architecture in Vaswani (a neural network model) revolutionized the field of deep learning, particularly in natural language processing (NLP). A key innovation of the transformer architecture is the self-attention mechanism, which computes the dependencies between each pair of positions in an input sequence in parallel, capturing the global dependencies in the data. The self-attention mechanism enabled the model to capture dependencies between words regardless of their positions in a sentence. This eliminated the need for recurrent or folded layers, making training faster and more parallelizable.The transformer has since become the basis of many state-of-the-art models in NLP, including BERT, GPT, and others. And despite their initial use in NLP, transformer structures have also demonstrated strong performance in other fields, such as computer vision, signal processing, and more.

[0011] Fig. Figure 1 shows a System 100 for training a neural network, e.g., a deep neural network, which is part of the backbone of a deep learning model or algorithm. The neural network being trained can be a deep learning network, such as the signal converter or signal translator model disclosed here. The System 100 can include an input interface for accessing training data 102 for the neural network. For example, as shown in Fig. Figure 1 illustrates that the input interface is provided by a data storage interface 104, which can access the training data 102 from a data storage device 106. For example, the data storage interface 104 can be a storage interface or a persistent storage interface, such as a hard disk or SSD interface, but also a personal, local, or wide-area network interface, such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage device 106 can be internal data storage of the system 100, such as a hard disk or SSD, but also external data storage, such as network-accessible data storage.

[0012] In some embodiments, the data storage 106 may further comprise a data representation 108 of an untrained version of the neural network, which can be accessed by the system 100 from the data storage 106. It is understood, however, that the training data 102 and the data representation 108 of the untrained neural network can also each be accessed from another data storage, e.g., via another subsystem of the data storage interface 104. Each subsystem can be of a type as previously described for the data storage interface 104.

[0013] In some embodiments, the data representation 108 of the untrained neural network can be generated internally by the system 100 based on design parameters for the neural network and is therefore not explicitly stored in the data storage 106. The system 100 can further comprise a processor subsystem 110 configured to provide, during an operation of the system 100, an iterative function as a replacement for a stack of layers of the neural network to be trained. Here, the respective layers of the stack of layers being replaced can each have shared weights and can receive as an input an output from a previous layer, or, for a first layer of the stack of layers, an initial activation and part of the input of the stack of layers.

[0014] The processor subsystem 110 can further be configured to iteratively train the neural network using the training data 102. Here, an iteration of the training by the processor subsystem 110 can include a forward propagation part and a backward propagation part. The processor subsystem 110 can be configured to perform the forward propagation part by, among other operations that define the forward propagation part to be performed, determining an equilibrium point of the iterative function at which the iterative function converges to a fixed point. Determining the equilibrium point includes using a root-finding algorithm to find a root solution for the iterative function minus its input, and by providing the equilibrium point as a substitute for an output of the stack of layers in the neural network.

[0015] The system 100 can further include an output interface for outputting a data representation 112 of the trained neural network; this data can also be referred to as trained model data 112. For example, as also in Fig. Figure 1 illustrates that the output interface is given by a data storage interface 104, where in these embodiments the interface is an input / output (IA) interface through which trained model data 112 can be stored in the data storage 106. For example, the data representation 108 that defines the 'untrained' neural network can be replaced, at least partially, during or after training by the data representation 112 of the trained neural network, in which the parameters of the neural network, such as weights, hyperparameters, and other types of neural network parameters, can be adapted to reflect the training on the training data 102. This is also shown in Figure 1. Fig. 1 is illustrated by the reference numerals 108 and 112, which refer to the same data record in the data storage 106. In other embodiments, the data representation 112 can be stored separately from the data representation 108, which defines the 'untrained' neural network. In some embodiments, the output interface can be separate from the data storage interface 104, but can generally be of the type previously described for the data storage interface 104.

[0016] The structure of System 100 is an example of a system that can be used to train the signal converter model or signal translator model described here. An additional structure for operating and training the machine learning models is described in Fig. 2 shown.

[0017] Fig. Section 2 represents a signal conversion system 200 for implementing a system for converting or translating data signals according to one embodiment. The signal conversion system 200 can utilize the signal converter model or signal translator model described herein. The signal conversion system 200 can include at least one computing system 202. The computing system 202 can include at least one processor 204, which is operationally connected to a memory unit 208. The processor 204 can include one or more integrated circuits that implement the functionality of a central processing unit (CPU) 206. The CPU 206 can be a commercially available processing unit that implements an instruction set, such as one of the x86, ARM, Power, or MIPS instruction set families.

[0018] During the operation, the CPU 206 can execute stored program instructions retrieved from the memory unit 208. The stored program instructions can include software that controls an operation of the CPU 206 to perform the operation described here. In some examples, the processor 204 can be a system-on-a-chip (SoC) that integrates the functionality of the CPU 206, the memory unit 208, a network interface, and input / output interfaces into a single integrated device. The computing system 202 can implement an operating system to manage various aspects of the operation. Although a processor 204, a CPU 206, and a memory unit 208 in Fig. As shown in Figure 2, of course more than one of each can be used in a complete system.

[0019] The memory unit 208 can include volatile and non-volatile memory for storing instructions and data. The non-volatile memory can include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is turned off or loses electrical power. The volatile memory can include static and dynamic random-access memory (RAM) that stores program instructions and data. For example, the memory unit 208 can store a machine learning model 210 or algorithm, a training dataset 212 for the machine learning model 210, and a raw source dataset 216. The machine learning model can be or include the signal converter model or signal translator model disclosed herein.

[0020] The computing system 202 may include a network interface device 222 configured to provide communication with external systems and devices. For example, the network interface device 222 may include a wired and / or wireless Ethernet interface, as defined by the 802.11 family of standards of the Institute of Electrical and Electronics Engineers (IEEE). The network interface device 222 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). The network interface device 222 may also be configured to provide a communication interface to an external network 224 or a cloud.

[0021] The external network 224 can be called the World Wide Web or the Internet. The external network 224 can establish a standard communication protocol between computing devices. The external network 224 enables the easy exchange of information and data between computing devices and networks. One or more servers 230 can be in communication with the external network 224.

[0022] The computing system 202 may include an input / output (I / O) interface 220, which may be configured to provide digital and / or analog inputs and outputs. The I / O interface 220 is used to transfer information between internal storage and external input and / or output devices (e.g., HMI devices). The I / O interface 220 may include an associated circuit arrangement or associated bus networks to transfer information to or between the processor(s) and the storage. For example, the I / O interface 220 may include digital I / O logic lines that can be read or set by the processor(s), handshake lines for monitoring data transfer over the I / O lines, timing and counter devices, and other structures known for providing such functions.Examples of input devices include a keyboard, mouse, sensors, etc. Examples of output devices include monitors, printers, speakers, etc. The I / O interface 220 can include additional serial interfaces for communicating with external devices (e.g., a Universal Serial Bus (USB) interface). The I / O interface 220 can be described as an input interface (in that it transfers data from an external input, such as a sensor) or an output interface (in that it transfers data to an external output, such as a display).

[0023] The computing system 202 may include a human-machine interface (HMI) device 218, which may include any input device that enables the system 200 to receive control input. The computing system 202 may include a display device 232. The computing system 202 may include hardware and software for outputting graphic and text information to the display device 232. The display device 232 may include an electronic screen, projector, printer, or other suitable device for displaying information to a user or operator. The computing system 202 may further be configured to enable interaction with a remote HMI and remote display devices via the network interface device 222.

[0024] System 200 can be implemented using one or more computing systems 202. Although the example represents a single computing system 202 that implements all of the described features, it is intended that different features and functions can be separated and implemented by multiple computing units communicating with each other. The specific system architecture chosen may depend on a variety of factors.

[0025] System 200 can implement a machine learning model 210 or algorithm trained to analyze the raw source dataset 216. The raw source dataset 216 can contain raw or unprocessed sensor data that may be representative of an input dataset for a machine learning system. The raw source dataset 216 can include audio, audio segments, video, video segments, images, text-based information, audio or human speech, time-series data (e.g., a pressure sensor signal over time), and raw or partially processed sensor data (e.g., a radar map of objects). Some different examples of inputs are given with reference to Fig. Figures 5-11 show and describe. In some examples, the machine learning model 210 may be or include a neural network algorithm (e.g., a deep neural network) trained to perform a predetermined function. For example, in automotive applications, the neural network algorithm may be trained to predict the torque of an engine based on the audio emitted by the engine. The machine learning models 210 may include algorithms trained to operate the signal converter model or signal translator model described here.

[0026] The computing system 202 can store a training dataset 212 for the machine learning algorithm 210. The training dataset 212 can represent a set of previously constructed data for training the machine learning model 210. The training dataset 212 can be used by the machine learning algorithm 210 to learn weighting factors associated with a neural network algorithm. The training dataset 212 can include a set of source data exhibiting corresponding results or outcomes that the machine learning algorithm 210 attempts to duplicate during the learning process. In this example, the training dataset 212 can include input images containing an object (e.g., a street sign). The input images can represent various scenarios in which the objects are identified.

[0027] The machine learning algorithm 210 can be operated in a learning mode using the training dataset 212 as input. The machine learning algorithm 210 can be executed over a number of iterations using the data from the training dataset 212. With each iteration, the machine learning algorithm 210 can update internal weighting factors based on the results obtained. For example, the machine learning algorithm 210 can compare output results (e.g., a reconstructed or augmented image if image data is the input) with those contained in the training dataset 212. Since the training dataset 212 contains the expected results, the machine learning algorithm 210 can determine when a performance level is acceptable. After the machine learning algorithm 210 has reached a predetermined performance level (e.g.,Once the machine learning algorithm 210 has achieved 100% agreement with the results associated with the training dataset 212, or has reached convergence, it can be run using data not contained in the training dataset 212. It is understood that in this disclosure, "convergence" may mean that a specified (e.g., predetermined) number of iterations have occurred, or that the residual is sufficiently small (e.g., the change in the approximate probability across iterations is less than a threshold), or it may mean other convergence conditions. The trained machine learning algorithm 210 can be applied to new datasets to generate annotated data.

[0028] The machine learning algorithm 210 can be trained to identify a specific feature in the raw source data 216. The raw source data 216 can contain multiple instances or a single input dataset for which supplementary results are desired. For example, the machine learning algorithm 210 can be trained to identify patterns or signatures in audio signals and convert the audio signal into a secondary signal based on the patterns or signatures. The machine learning algorithm 210 can be programmed to process the raw source data 216 to identify the presence of the specific features. The machine learning algorithm 210 can be trained to identify a feature in the raw source data 216 as a predetermined feature. The raw source data 216 can be derived from a variety of sources.For example, the raw source data 216 can be actual input data collected by a machine learning system. The raw source data 216 can be machine-generated for testing the system. As an example, the raw source data 216 can include raw audio signals from a microphone or microphone array.

[0029] Fig. Figure 3A illustrates the signal conversion system 200 in general terms in the form of a diagram. Here, the signal conversion system 200 is configured to convert a primary signal into a secondary signal using machine learning according to one embodiment. A primary signal (e.g., an audio signal) is received as input. A signal converter model 300 (also referred to as a signal translator model, audio translation model, transformer model, e.g., ML model 210) converts the primary signal into a secondary signal. For this, the signal converter model relies on a translator-based model, which is further described in Fig. Figure 3B illustrates this and is further described here. In one embodiment, the primary signal is an audio signal, such as that produced by a microphone. The secondary signal can be one of several desired modalities. For example, the secondary signal can be a torque signal, so that the received audio signal is translated into a torque signal, enabling the system to determine the torque of a device based on its emitted tone.

[0030] Fig. Figure 3B illustrates an exemplary architecture of the Signal Converter Model 300. The Signal Converter Model 300 is designed to transform a first-mode or first-modality input signal into a second-mode or second-modality output signal. In general, the Signal Converter Model 300 can be or include an attention-based neural network that performs machine learning tasks. The attention-based neural network includes a neural encoder network that receives an input sequence and generates a coded representation of the input sequence, outputting high-level representations that are directly processed by a linear transformation layer to generate the secondary signal.

[0031] In 302, a given primary (e.g., audio) signal x ∈ ℝ Tover a time T, which is received (or pre-processed) by a microphone, is divided or spliced ​​into successive frames, each of which has a size τ. Fig. Figure 3C shows an embodiment of frames of an audio input, which are shown in the bounding boxes. The size of each frame can be predetermined and / or can be based on the size (length) of the primary signal. In one embodiment, each frame has an identical size τ. Each of these frames can be designated as x τ ∈ ℝ τ This can be described as a single instance that can be fed into the transformer model.

[0032] Although the raw audio input in Fig. Figure 3B, shown as a "fuel pump tone," is merely an example where a vehicle fuel pump generates a tone for translation purposes. It is understood that the primary signal need not be audio and the secondary signal need not be a torque signal. In fact, it is understood that the models described here can be implemented for translation between any one-dimensional signals, including, but not limited to, sound, vibration, torque, temperature, heart rate variability, seismic waveforms, biological signals (e.g., EEG, EMG, EOG), time-series data, radar reflections, and others.

[0033] At 304, each input audio frame is x τ ∈ ℝ τ first transformed into a vector by a linear transformation layer, which corresponds to ℝ τ×d belongs, where d is the model dimension. This vector represents the input data of each respective frame of audio data and has a dimension V. τ∈ ℝ τ×d Each audio frame is independent and processed separately, with a single feature vector for each audio frame. Each audio frame can have a common size, for example, τ = 512 samples within each frame.

[0034] Position encoding, or positional embedding, is then added at 306. To preserve the temporal order of the audio in the input sequence, positional encoding is added to the embeddings. This allows the model to understand the position of each tone in the audio signal, specifically each position (sample) within each audio frame. In one embodiment, each vector associated with each frame is embedded with information about the relative position of each data (e.g., audio) sample within that vector or frame. Positional encoding allows the signal converter model to perform well on tasks requiring sequence understanding, such as the translation of one-dimensional signals shown here, where the order of frames is significant.By incorporating positional coding, the transformer can effectively capture both the content and the sequence of samples in the input data. In one embodiment, sine and cosine functions with different frequencies can be used for the positional coding, with each dimension of the positional coding corresponding to a sine wave.

[0035] With added positional encoding, the vectors then pass through the transformer encoder at 308, which contains N identical structures. Within each structure, a multi-head attention mechanism (also known as a multi-head attentional machine learning model) is applied to derive the self-attentions of the input vector. An attention mechanism can be designed to map a query and a set of key-value pairs to an output, where the query, key, values, and output are all vectors; the output is calculated as a weighted sum of the values, with the weight assigned to each value calculated by a compatibility function of the query with the corresponding key. The attention mechanism used here is a self-attentional mechanism in that there is only one type of input (e.g., audio).The use of a multi-head self-attention mechanism allows the neural network to compare each individual position (sample) with other positions (samples) within a given frame of input data. For example, with reference to... Fig. 3C compares each sample (e.g., 512 samples per frame) to all other samples within that frame for each frame shown in a bounding box. This allows the translation model to weight the importance or relevance of the samples within each frame. Each brain learns different attention patterns when the samples are compared.

[0036] The multi-head self-attention mechanism allows the transformer model to capture complex relationship dependencies between positions (samples) within each audio frame. One idea behind multi-head self-attention is to have multiple "heads," or different perspectives, regarding how the frames in the input data relate to one another. These heads work together to better understand the entire audio input. Each head focuses on a different aspect of the relationship between the samples. For this to work, the frames in the input data are used to generate three types of vectors: requests, keys, and values. Each head takes these request, key, and value vectors and calculates how much attention each sample should pay to every other sample in the frame. Scores for each sample can then be derived.For example, some parts of the audio data might be more important for translation purposes than others. Attention ratings determine how much each sample should contribute to the output. In other words, some samples are weighted more than others. And with multiple heads, each head learns different attention patterns associated with different samples of input data. After each head has performed its own calculations, their results are combined. Mixing different attention patterns helps the model capture a wide range of relationships between samples within each frame.

[0037] As in Fig. As shown in Figure 3B, the transformer encoder at 308 has one or more add-and-normalize features, or layer normalizations. When applied, these improve the training stability and performance of the model. Adding involves adding the output of each layer (e.g., self-attention layer or feed-forward layer) to the input of the layer after each layer in the transformer. This is called a residual connection or skip connection. This preserves the original input information, simplifying learning for the model. Following the add step, layer normalization is applied, which involves normalizing the values ​​in the output of the layer to have a mean of -1 and a standard deviation of -1.Layer normalization helps to stabilize the learning process by ensuring that the input to each learning layer lies within a similar range of values.

[0038] In short, the self-attention mechanism splices the input into a subspace and then computes a relationship between these splices. The self-attention aspect of the model allows it to compute relationships between each sample (position) and all other samples (positions). The model functions to learn the relationship between the different samples (positions) in the data sequence.

[0039] At 310, vectors pass through H τ ∈ ℝ τxdWith high-level representation output by the self-attention mechanism, a linear transformation layer is used to ensure that the vector has a dimension equal to the secondary signal dimension (e.g., torque signal). Since the primary and secondary signals have different sampling rates, the dimension can be calculated from the sampling rate of the two domain signals. For example, the linear transformation layer provides the framework of a secondary signal (e.g., torque) with a form of ftfaτ out. Here represent, f a and f t the sampling rates of the primary and secondary signals.

[0040] At 312, the secondary signal frames are then chained together to create a reconstructed signal. Rek∈ℝftfaT or to obtain a secondary signal. This reconstructed signal represents an estimated signal or secondary signal based on the primary signal. In one example, the reconstructed signal is a torque signal, indicating the torque associated with the device (e.g., vehicle) emitting the tone used as the input.

[0041] Fig. Figure 3C illustrates an example of frame-based signal-to-signal conversion using the system described above. In this embodiment, the input signal is an audio signal and the output signal is a torque signal. Other input and output signals can be used and derived using the signal converter model disclosed herein. Within each bounding box is a frame of sound, which is the transformer model derived from Fig. 3B is completed.

[0042] Fig. 3D illustrates an example of a predicted secondary signal and an actual secondary signal. The ground truth signal, shown here as the "true torque" signal, is the input to the model, and the reconstructed signal, or secondary signal, is the model's output. The predicted secondary signal is a result of the transformed primary signal. For example, a transformation model performed on the audio signal results in a predicted torque model. The true torque signal is a result of actual torque sensor measurements, such as from a torque sensor. As shown in Fig. Shown in 3D, the predicted secondary signal and the actual secondary signal are extremely close together, indicating a high performance of the transformation model.

[0043] In some embodiments, the initial audio signal (i.e., in the time domain) can undergo a transformation into the frequency domain, such as a spectrogram, before being fed into the signal converter model. In this scenario, a simple process can be used to generate a spectrogram, followed by the subsequent step of transforming or smoothing the resulting two-dimensional spectrograms into a one-dimensional sequence. For example, this sequence can be formed by concatenating each time slice of spectrograms.

[0044] In some implementations, variations of a transformer architecture could be used for modeling, such as a conformer that is a hybrid CNN transformer model, etc.

[0045] The methods and systems disclosed here therefore provide a model based on data-centric and machine learning / deep learning methodologies for converting a one-dimensional signal, such as audio, into another one-dimensional signal modality and vice versa. This model allows for the tracking and potential modification of signal representations, enabling a seamless transition between modalities. It is understood that the secondary signal (e.g., torque) can be converted into the primary signal (e.g., audio) using the same models described here.

[0046] The methods and systems disclosed here can utilize the transformer-based architecture as the core conversion model, which incorporates a flexible frame size for optimal adaptability. The framework simultaneously processes the relationships between two modalities at the frame level. This process ensures high accuracy and speed in the reconstruction of the secondary signal. This facilitates easy extension to other one-dimensional signal-to-signal translations.

[0047] The models disclosed here can be trained with cross-modal data. For example, the input signal during training can be audio, and the resulting secondary signal can be a representative signal for torque, using an actual torque value for comparison. The model can be trained until convergence, i.e., when the secondary signal calculated by the model lies within a threshold of the actual signal produced by a sensor.

[0048] Fig. Figure 4 illustrates a method 400 for transforming a primary signal into a secondary signal of another modality according to one embodiment. The method can be executed by one or more processors described herein, which rely on instructions stored in a memory described herein.

[0049] In the 402 protocol, the processor receives a primary signal. This primary signal can be an audio signal or another one-dimensional signal. The audio signal can, for example, be received from a microphone or pre-processed by an associated processor.

[0050] In 404, the processor splices or segments the primary signal into a plurality of frames. Each frame can have an identical size τ, which can depend on the size of the signal. The signal of each frame can therefore be represented as x τ ∈ ℝ τ be represented.

[0051] At 406, the processor performs a first linear transformation. In embodiments, the linear transformation layer transforms the signal of each frame into a vector that corresponds to ℝ τ × d belongs, where d is the model dimension.

[0052] In the 408, the processor performs position coding to maintain the temporal order of the primary signal. In one embodiment, sine and cosine functions with different frequencies can be used for the position coding, with each dimension of the position coding corresponding to a sine wave.

[0053] In embodiment 410, the processor executes a transformer encoder comprising N structures, each structure having a multi-head self-attention mechanism. In embodiments, the multi-head self-attention mechanisms are configured to map a query and a set of key-value pairs to an output, where the query, key, values, and output are all vectors; the output is computed as a weighted sum of the values, the weight assigned to each value being calculated by a compatibility function of the query with the corresponding key. This results in vectors H τ ∈ ℝ τ × d with high-level representation.

[0054] At 412, the processor performs a second linear transformation on the vectors with high-level representation to ensure that the vector has a dimension equal to the secondary signal dimension.

[0055] In version 414, the processor concatenates the secondary signal frames to obtain a single, reconstructed secondary signal. The secondary signal can be of a different modality than the primary signal.

[0056] Fig. Figure 5 presents a schematic diagram of an interaction between a computer-controlled machine 500 and a control system 502. The computer-controlled machine 500 includes an actuator 504 and a sensor 506. The actuator 504 can include one or more actuators, and the sensor 506 can include one or more sensors. The sensor 506 is configured to detect a condition of the computer-controlled machine 500. The sensor 506 can be configured to encode the detected condition into sensor signals 508 and to transmit sensor signals 508 to the control system 502. The sensor 506 can be one or more of the sensors described above; non-limiting examples of the sensor 506 include a microphone, a camera, a video camera, a radar, a LiDAR, ultrasonic sensors, and motion sensors.In one embodiment, the sensor 506 is a microphone designed to generate audio signals from an environment inside or near the computer-controlled machine 500.

[0057] The control system 502 is configured to receive sensor signals 508 from the computer-controlled machine 500. As explained below, the control system 502 can also be configured to calculate actuator control commands 510 based on the sensor signals and to transmit actuator control commands 510 to the actuator 504 of the computer-controlled machine 500.

[0058] As in Fig. As shown in Figure 5, the control system 502 includes a receiver 512. The receiver 512 can be configured to receive sensor signals 508 from the sensor 506 and to transform these sensor signals 508 into input signals x. In an alternative embodiment, the sensor signals 508 are received directly as input signals x without the receiver 512. Each input signal x can be a part of each sensor signal 508. The receiver 512 can be configured to process each sensor signal 508 to produce each input signal x. The input signal x can contain data corresponding to an image recorded by the sensor 506.

[0059] The control system 502 includes a classifier 514. The classifier 514 can be configured to classify input signals x into one or more labels using a machine learning algorithm, such as a previously described neural network. The classifier 514 is configured to be parameterized by parameters, such as those previously described (e.g., parameter θ). The parameter θ can be stored in and provided by non-volatile memory 516. The classifier 514 is configured to determine output signals y from input signals x. Each output signal y contains information that assigns one or more labels to each input signal x. The classifier 514 can transmit output signals y to the conversion unit 518. The conversion unit 518 is configured to convert output signals y into actuator control commands 510.The control system 502 is configured to transmit actuator control commands 510 to the actuator 504, which is configured to actuate the computer-controlled machine 500 in response to the actuator control commands 510. In another embodiment, the actuator 504 is configured to actuate the computer-controlled machine 500 directly based on output signals y.

[0060] Upon receiving actuator control commands 510, the actuator 504 is configured to perform an action corresponding to the associated actuator control command 510. The actuator 504 may include control logic configured to transform the actuator control commands 510 into a second actuator control command, which is used to control the actuator 504. In one or more embodiments, the actuator control commands 510 can be used to control a display instead of, or in addition to, an actuator.

[0061] In another embodiment, the control system 502 includes a sensor 506 instead of, or in addition to, the computer-controlled machine 500, which includes the sensor 506. The control system 502 can also include the actuator 504 instead of, or in addition to, the computer-controlled machine 500, which includes the actuator 504.

[0062] As in Fig. As shown in Figure 5, the control system 502 also includes the processor 520 and the memory 522. The processor 520 can include one or more processors. The memory 522 can include one or more memory devices. The classifier 514 (e.g., machine learning algorithms, such as those previously described with reference to the pretrained classifier 306) of one or more embodiments can be implemented by the control system 502, which includes the non-volatile memory 516, the processor 520, and the memory 522.

[0063] The non-volatile storage 516 can include one or more persistent data storage devices, such as a hard disk drive, optical drive, tape drive, a non-volatile solid-state device, cloud storage, or any other device capable of persistently storing information. The processor 520 can include one or more devices selected from high-performance computing (HPC) systems, including high-performance cores, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other devices that manipulate signals (analog or digital) based on computer-executable instructions located in the memory 522.The memory 522 can include a single memory device or a series of memory devices, including, but not limited to, random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0064] The processor 520 can be configured to read into memory 522 and execute computer-executable instructions located in non-volatile memory 516, which execute one or more machine learning algorithms and / or methodologies of one or more implementations. The non-volatile memory 516 includes one or more operating systems and applications. The non-volatile memory 516 can store compiled and / or interpreted computer programs generated using a variety of programming languages ​​and / or technologies, including, but not limited to, and either alone or in combination, Java, C, C++, C#, Objective-C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL.

[0065] When executed by the processor 520, the computer-executable instructions of the non-volatile memory 516 can cause the control system 502 to implement one or more of the machine learning algorithms and / or methodologies disclosed herein. The non-volatile memory 516 can also contain machine learning data (including data parameters) that support the functions, features, and processes of one or more embodiments described herein.

[0066] The program code implementing the algorithms and / or methodologies described herein is capable of being distributed individually or collectively as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium containing computer-readable program instructions to cause a processor to execute aspects of one or more embodiments. Computer-readable storage media, which are inherently non-transient, can include volatile and non-volatile, removable and non-removable tangible media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.Computer-readable storage media may also include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technology, portable compact disc read-only memory (CD-ROM) or other optical storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be read by a computer. Computer-readable program instructions may be downloaded from a computer-readable storage medium to a computer, another type of programmable data processing device or other device, or over a network to an external computer or external storage device.

[0067] Computer-readable program instructions stored on a computer-readable medium can be used to instruct a computer, other types of programmable data processing equipment, or other devices to operate in a specific manner, such that the instructions stored on the computer-readable medium produce a manufactured item containing instructions that implement the functions, actions, and / or operations specified in the flowcharts or diagrams. In certain alternative embodiments, the functions, actions, and / or operations specified in the flowcharts and diagrams can be reordered, processed serially, and / or processed simultaneously in accordance with one or more embodiments.Furthermore, any of the flowcharts and / or diagrams may illustrate more or fewer nodes or blocks than those illustrated in accordance with one or more embodiments.

[0068] The processes, procedures or algorithms can be executed wholly or partially using suitable hardware components, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers or other hardware components or devices, or a combination of hardware, software and firmware components.

[0069] The models and teachings provided here can be used and applied in a multitude of situations. Fig. Figure 6 presents a schematic diagram of a control system 502 configured to control a vehicle 600, which may be a partially autonomous vehicle or a partially autonomous robot. The vehicle 600 includes an actuator 504 and a sensor 506. The sensor 506 may include one or more microphones, video sensors, cameras, radar sensors, ultrasonic sensors, LiDAR sensors, and / or position sensors (e.g., GPS). One or more of these specialized sensors may be integrated into the vehicle 600. In the context of sign recognition and processing, as described here, the sensor 506 is a camera mounted in or integrated into the vehicle 600. Alternatively, or in addition to one or more specialized sensors identified above, the sensor 506 may include a software module configured, when executed, to determine the state of an actuator 504.A non-restrictive example of a software module includes a weather information software module trained to determine a present or future state of the weather near the vehicle 600 or some other location.

[0070] The classifier 514 of the control system 502 of the vehicle 600 can be configured to detect objects in the vicinity of the vehicle 600 depending on input signals x. In such an embodiment, the output signal y can contain information characterizing the proximity of the object to the vehicle 600. The actuator control command 510 can be determined according to this information. The actuator control command 510 can be used to avoid collisions with the detected objects.

[0071] In embodiments where the vehicle 600 is at least partially autonomous, the actuator 504 can be implemented in a brake, drive system, motor, powertrain, or steering system of the vehicle 600. Actuator control commands 510 can be determined such that the actuator 504 is controlled to prevent the vehicle 600 from colliding with detected objects. Detected objects can also be classified according to what the classifier 514 considers most likely, such as pedestrians or trees. The actuator control commands 510 can be determined depending on the classification. In a scenario where an enemy attack may occur, the system described above can also be trained to better detect objects or identify changes in lighting conditions or angles for a sensor or camera on the vehicle 600.

[0072] In other embodiments, where the vehicle 600 is an at least partially autonomous robot, the vehicle 600 can be a mobile robot designed to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot can be an at least partially autonomous lawnmower or a at least partially autonomous cleaning robot. In such embodiments, the actuator control command 510 can be determined such that a drive unit, a steering unit, and / or a braking unit of the mobile robot can be controlled in such a way that the mobile robot can avoid collisions with identified objects.

[0073] In another embodiment, the vehicle 600 is an at least partially autonomous robot in the form of a gardening robot. In such an embodiment, the vehicle 600 can use an optical sensor, such as sensor 506, to determine the condition of plants in an environment near the vehicle 600. The actuator 504 can be a nozzle designed to spray chemicals. Depending on an identified species and / or condition of the plants, an actuator control command 510 can be determined to cause the actuator 504 to spray the plants with an appropriate quantity of suitable chemicals.

[0074] The vehicle 600 can be a at least partially autonomous robot in the form of a household appliance. Non-restrictive examples of household appliances include a washing machine, a stove, an oven, a microwave, or a dishwasher. In such a vehicle 600, the sensor 506 can be an optical sensor designed to detect the state of an object that is to be processed by the household appliance. For example, if the household appliance is a washing machine, the sensor 506 can detect the state of the laundry inside the washing machine. The actuator control command 510 can then be determined based on the detected state of the laundry.

[0075] Fig. Figure 7 shows a schematic diagram of a control system 502 configured to control a system 700 (e.g., a manufacturing machine), such as a punching tool, a cutting tool, or a gun drill, or a manufacturing system 702, such as a part of a production line. The control system 502 can be configured to control the actuator 504, which is configured to control the system 700 (e.g., a manufacturing machine).

[0076] The sensor 506 of the system 700 (e.g., a manufacturing machine) can be an optical sensor configured to detect one or more properties of a manufactured product 704. The classifier 514 can be configured to determine a state of the manufactured product 704 from one or more of the detected properties. The actuator 504 can be configured to control the system 700 (e.g., a manufacturing machine) for a subsequent manufacturing step of the manufactured product 704, depending on the determined state of the manufactured product 704. The actuator 504 can be configured to control functions of the system 700 (e.g., a manufacturing machine) on a subsequently manufactured product 706 of the system 700 (e.g., a manufacturing machine), depending on the determined state of the manufactured product 704.

[0077] Fig. Figure 8 shows a schematic diagram of a control system 502 configured to control a power tool 800, such as an electric drill or screwdriver, which has at least a partially autonomous mode. The control system 502 can be configured to control the actuator 504, which is configured to control the power tool 800.

[0078] A sensor 506 of the power tool 800 can be an optical sensor configured to detect one or more properties of a work surface 802 and / or a fastener 804 driven into the work surface 802. The classifier 514 can be configured to determine a state of the work surface 802 and / or the fastener 804 relative to the work surface 802 from one or more of the detected properties. The state can be that the fastener 804 is flush with the work surface 802. Alternatively, the state can be a hardness of the work surface 802. The actuator 504 can be configured to control the power tool 800 such that the drive function of the power tool 800 is adapted depending on the determined state of the fastener 804 relative to the work surface 802 or on one or more detected properties of the work surface 802.For example, the actuator 504 can adjust the drive function if the fastener 804 is flush with the working surface 802. As another non-limiting example, the actuator 504 can apply additional or less torque depending on the hardness of the working surface 802.

[0079] In another embodiment, the sensor 506 is a microphone configured to generate one-dimensional sound data associated with the power tool 800 when it drives a fastener 804 into the work surface 802. The classifier 514 can be configured to determine a torque of the motor of the tool 800 by converting the sound emitted by the tool into a torque signal, as disclosed above.

[0080] Fig. Figure 9 shows a schematic diagram of a control system 502 trained to control an automated personal assistant 900. The control system 502 can be trained to control the actuator 504, which is trained to control an automated personal assistant 900. The automated personal assistant 900 can be trained to control a household appliance, such as a washing machine, stove, oven, microwave, or dishwasher.

[0081] A sensor 506 can be an optical sensor and / or an audio sensor. The optical sensor can be configured to receive video images or gestures 904 from a user 902. The audio sensor can be configured to receive a voice command from a user 902.

[0082] The control system 502 of an automated personal assistant 900 can be trained to determine actuator control commands 510 that are trained for the control system 502. The control system 502 can be trained to determine actuator control commands 510 according to sensor signals 508 from the sensor 506. The automated personal assistant 900 is trained to transmit sensor signals 508 to the control system 502. The classifier 514 of the control system 502 can be trained to execute a gesture recognition algorithm to identify a gesture 904 made by a user 902 in order to determine actuator control commands 510, and to transmit the actuator control commands 510 to the actuator 504.The classifier 514 can be configured to retrieve information from non-volatile storage in response to gesture 904 and to output the retrieved information in a form suitable for reception by user 902.

[0083] Fig. Figure 10 presents a schematic diagram of a control system 502 configured to control a monitoring system 1000. The monitoring system 1000 may be configured to physically control access through a door 1002. A sensor 506 may be configured to detect a scene relevant to a decision on whether to grant access. The sensor 506 may be an optical sensor configured to generate and transmit image and / or video data. Such data may be used by the control system 502 to detect a person's face.

[0084] The classifier 514 of the control system 502 of the monitoring system 1000 can be configured to interpret the image and / or video data by comparing it with the identities of known persons stored in the non-volatile memory 516, thereby determining a person's identity. The classifier 514 can be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504. In this embodiment, the actuator 504 can be configured to lock or unlock a door 1002 in response to the actuator control command 510. In other embodiments, non-physical, logical access control is also possible.

[0085] The monitoring system 1000 can also be an observation system. In such an embodiment, the sensor 506 can be an optical sensor configured to detect a scene under observation, and the control system 502 is configured to control a display 1004. The classifier 514 is configured to determine a classification of a scene, e.g., whether the scene detected by the sensor 506 is suspicious. The control system 502 is configured to transmit an actuator control command 510 to the display 1004 in response to the classification. The display 1004 can be configured to adjust the displayed content in response to the actuator control command 510. For example, the display 1004 can highlight an object that is considered suspicious by the classifier 514. Using an embodiment of the disclosed system, the observation system can predict that objects will appear at certain times in the future.

[0086] Fig. Figure 11 shows a schematic diagram of a control system 502 trained to control an imaging system 1100, for example, an MRI scanner, an X-ray imaging scanner, or an ultrasound scanner. A sensor 506 can, for example, be an imaging sensor. The classifier 514 can be trained to determine a classification of all or part of the acquired image. The classifier 514 can be trained to determine or select an actuator control command 510 in response to the classification obtained by the trained neural network. For example, the classifier 514 can interpret an area of ​​an acquired image as potentially anomalous. In this case, the actuator control command 510 can be determined or selected to cause a display 1102 to display the imaging and highlight the potentially anomalous area.

[0087] This document uses the term "trained" in connection with systems and computer program components. For a system consisting of one or more computers, being trained to perform certain operations or actions means that the system has software, firmware, hardware, or a combination thereof installed that, when operating, causes the system to perform the operations or actions. For one or more computer programs, being trained to perform certain operations or actions means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operations or actions.

[0088] In this document, the term "mechanism" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, a mechanism (e.g., a self-attention mechanism) is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular mechanism; in other cases, multiple mechanisms may be installed and run on the same computer(s). References to a multi-head self-attention mechanism may also be referred to as a multi-head self-attention machine learning model.

[0089] Although exemplary embodiments have been described previously, it is not intended that these embodiments describe all possible forms encompassed by the claims. The words used in this document are descriptive and not limitative, and it is understood that various modifications may be made without departing from the idea and scope of the disclosure. As previously described, the features of different embodiments can be combined to form further embodiments of the invention that may not be explicitly described or illustrated.Although various embodiments may have been described as providing advantages or being preferred over other embodiments or implementations of the prior art with respect to one or more desired characteristics, a person skilled in the art will recognize that one or more features or characteristics may be compromised in order to achieve desired overall system attributes, which depend on the specific application or implementation. These attributes may include, but are not limited to, cost, strength, durability, life-cycle costs, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, and so on.Therefore, insofar as any embodiments are described as less desirable than other embodiments or implementations of the prior art with regard to one or more characteristics, these embodiments are not outside the scope of protection of the disclosure and may be desirable for specific applications.

Claims

[1] A method for transforming an audio signal into a secondary signal of another modality, the method comprising: Receiving an audio signal generated by a microphone; splicing the audio signal into a plurality of frames, each frame comprising a number of samples of audio data; performing a first linear transformation to transform the frames into corresponding vectors; performing position encoding on the vectors to encode relative position information associated with each sample within the vectors; performing a transformer encoder on the vectors with the encoded position information, the transformer encoder comprising a multi-head self-attention mechanism configured to compare a relative importance of the vectors with each other to obtain vectors with high-level representation; Performing a second linear transformation to transform the high-level representation vectors into corresponding secondary signal frames; and Concatenating the corresponding secondary signal frames to form a reconstructed one-dimensional secondary signal with a different modality than the audio signal. [2] The method of claim 1, wherein each frame has an identical size τ. [3] The method of claim 1, wherein the transformer encoder comprises an add-and-normalize feature configured to add an output of a layer of the transformer encoder to an input of the layer and normalize values ​​in the output of the layer. [4] The method of claim 1, wherein the multi-head self-attention mechanism includes an attention function that maps a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. [5] The method of claim 4, wherein the multi-head self-attention mechanism is configured to calculate the output vector as a weighted sum of the values. [6] The method of claim 5, wherein weights assigned to each value are calculated by a compatibility function of the query with a corresponding one of the keys. [7] The method of claim 1, wherein the multi-head self-attention mechanism is configured for each sample to calculate a score for that sample representing the relative importance of that sample relative to the other samples within that frame. [8] The method of claim 7, wherein the scores are associated with how much each sample should contribute to the output of the transformer encoder. [9] The method of claim 1, wherein the secondary signal is a torque signal or a vibration signal. [10] A system for converting a one-dimensional primary signal into a one-dimensional secondary signal of another modality, the system comprising: a processor programmed to execute instructions stored in a memory to: splicing a primary signal into a plurality of consecutive frames; performing a first linear transformation to transform the frames into corresponding vectors; performing position encoding on the vectors to encode relative position information associated with each sample, wherein the relative position information is associated with a sequential position of each sample within its respective frame; Executing a multi-head self-attention mechanism configured to compare a relative importance of the samples with each other within their respective frames to obtain high-level representation vectors; Performing a second linear transformation to transform the high-level representation vectors into corresponding secondary signal frames; and Concatenating the secondary signal frames to form a reconstructed one-dimensional secondary signal with a different modality than the primary signal. [11] The system of claim 10, wherein each frame has an identical size. [12] The system of claim 10, wherein the multi-head self-attention mechanism comprises an add-and-normalize feature configured to add an output of a layer of the transformer encoder to an input of the layer and normalize values ​​in the output of the layer. [13] The system of claim 10, wherein the multi-head self-attention mechanism includes an attention function that maps a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. [14] The system of claim 13, wherein the multi-head self-attention mechanism is configured to calculate the output vector as a weighted sum of the values. [15] The system of claim 14, wherein weights assigned to each value are calculated by a compatibility function of the query with a corresponding one of the keys. [16] The system of claim 10, wherein the multi-head self-attention mechanism is configured for each sample within a respective one of the frames to calculate a score for that sample representing the relative importance of that sample relative to the other samples in that frame. [17] The system of claim 16, wherein the scores are associated with how much each sample should contribute to the output of the transformer encoder. [18] The system of claim 10, wherein the primary signal is an audio signal. [19] A computer-controlled machine comprising the system of claim 10, wherein the computer-controlled machine further comprises an actuator configured to control an operation of the computer-controlled machine based on an output of the system. [20] Computer-controlled machine comprising: at least one microphone configured to generate an audio signal; a control system configured to predict an operating characteristic of the computer-controlled machine by translating the audio signal into a one-dimensional secondary signal representing the operating characteristic, the control system configured to: Splicing the audio signal into a plurality of frames, performing a first linear transformation to transform the frames into corresponding vectors; performing position encoding on the vectors to encode relative position information associated with each sample; performing a transformer encoder on the vectors with the encoded position information, the transformer encoder comprising a multi-head self-attention mechanism configured to compare a relative importance of the samples to each other within the respective vectors to obtain high-level representation vectors; Performing a second linear transformation to transform the high-level representation vectors into corresponding secondary signal frames; and Concatenating the corresponding secondary signal frames to form a reconstructed one-dimensional secondary signal with a different modality than the audio signal.

Images