Signal processing method of computer system

Abstract

The invention relates to multi-mode signal processing, and provides a signal processing method of a computer system. The method comprises the following steps: acquiring different modal signal data by using a plurality of sensors and performing timestamp synchronization; a depth time sequence alignment network and a multi-layer dynamic time warping algorithm are utilized to ensure signal time consistency; multi-modal signal features are extracted and fused through a multi-stream depth feature extraction network and a cross-modal attention mechanism; the system adopts a dynamic convolution kernel generation and meta-learning technology to carry out adaptive signal processing; noise suppression and signal optimization are carried out by using a multi-mode noise reduction auto-encoder and a generative adversarial network, and parameters are optimized and processed through closed-loop feedback control and reinforcement learning strategies. According to the method, synchronization, fusion, adaptive processing, noise suppression and real-time feedback control of multi-mode signals are realized, and the method has the advantages of high precision, high robustness and real-time performance.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal signal processing, and in particular to a method for processing signals by a computer system. Background Art

[0002] With the rapid development of technologies such as the Internet of Things, autonomous driving, intelligent monitoring, and smart healthcare, multimodal signal processing plays an increasingly important role in modern information systems. Multimodal signals refer to various types of signal data collected by different sensors, such as visual signals, audio signals, temperature signals, acceleration signals, etc. These multimodal signals are usually rich in complementarity and can provide a more comprehensive perception of the environment or target. However, due to the large differences in the acquisition frequency, timing characteristics, and data structure of multimodal signals, how to efficiently and accurately process multimodal signals has become a key issue in the current technical field.

[0003] At present, multimodal signal processing systems face the following major challenges in practical applications:

[0004] Signal synchronization and timing alignment

[0005] Multimodal signal acquisition devices often have problems such as asynchronous timestamps, inconsistent data sampling frequencies, and signal delays. Traditional time synchronization methods are difficult to meet the needs of high-precision, multimodal data alignment. Especially in real-time application scenarios, the timing differences of multimodal signals may lead to inaccurate information fusion and affect the overall performance of the system. Therefore, an efficient timing alignment method is needed to enable signals of different modes to be accurately aligned on the same time base, laying the foundation for subsequent feature extraction and fusion.

[0006] Feature extraction and fusion

[0007] Signals of different modes have different feature spaces and data structures. Visual signals have strong spatial characteristics, audio signals have time series characteristics, and sensor signals such as temperature and acceleration mainly show low-frequency changes. Traditional feature extraction methods are difficult to uniformly process the diverse features in multimodal signals. In addition, in terms of feature fusion, how to effectively fuse multimodal signals and how to highlight key features and weaken irrelevant information during the fusion process are difficulties in current signal processing technology. If the fusion method is inappropriate, it may lead to information loss or noise amplification, affecting the final processing effect.

[0008] Environmental adaptability

[0009] In practical applications, multimodal signal processing systems are usually faced with frequent changes in the signal environment. For example, the sensor signals in the autonomous driving system will fluctuate with the changes in the road environment, and the audio and video signals of the monitoring system will be interfered by noise. These dynamic changes put forward higher adaptability requirements for the signal processing system. In the existing technology, fixed parameter processing methods are usually used, which lack the ability to dynamically adapt to the environment and cannot guarantee the signal processing effect in complex and dynamic environments. Therefore, how to enable the system to adaptively adjust parameters to adapt to different environments is an important research direction in multimodal signal processing.

[0010] Noise suppression and signal optimization

[0011] In complex application scenarios, noise interference is an inevitable problem in multimodal signal processing systems. The types and degrees of noise interference to signals of different modes may vary greatly. Visual signals may be affected by changes in lighting, audio signals may be interfered by background noise, and sensor signals may be interfered by vibration and other factors. Traditional noise suppression methods are often targeted at specific modal signals and cannot adapt to the diverse noise characteristics of multimodal signals. At the same time, the dynamic nature of noise increases the difficulty of noise suppression, and more intelligent noise suppression and signal optimization methods are needed to ensure signal quality.

[0012] Real-time control and feedback

[0013] Multimodal signal processing systems usually need to be real-time in applications. For example, in autonomous driving or intelligent monitoring, the system needs to respond to environmental changes within milliseconds. This requires the system to have fast feedback control capabilities during signal processing, monitor the processing effect in real time and make dynamic adjustments. However, traditional control methods are difficult to provide real-time feedback adjustment in complex signal processing processes, which can easily lead to a decrease in signal processing effects as the environment changes. Therefore, how to build a real-time control and feedback mechanism that enables the system to adapt to environmental changes and maintain the optimal state is a key technical problem in multimodal signal processing.

[0014] Therefore, the present invention proposes a method for processing signals in a computer system to solve the deficiencies of the prior art. Summary of the invention

[0015] The present invention aims to solve the technical problems of multimodal signal processing systems in data synchronization, feature extraction, environmental adaptation, noise suppression and real-time feedback control. Through the present invention, high-precision time synchronization, feature fusion and adaptive processing of different modal signals can be achieved, effectively improving the robustness and flexibility of signal processing, and providing a reliable solution for multimodal data processing in complex environments.

[0016] To achieve the above objectives, the present invention is implemented through the following technical solutions: A method for processing signals by a computer system, comprising the following steps:

[0017] Collect multimodal signal data, and collect different modal signal data including time stamps through multiple sensors, the sensors include but are not limited to image sensors, audio sensors, temperature sensors, and acceleration sensors;

[0018] Synchronize and align the collected multi-modal signal data, build a deep time alignment network and combine it with a time warping algorithm to achieve time consistency of different modal signals;

[0019] Extract and fuse the features of multimodal signals, extract the features of each modality through a multi-stream deep feature extraction network, and generate a unified fusion feature representation through a feature fusion strategy;

[0020] Implement adaptive signal processing to adapt to different environmental characteristics through dynamic convolution kernel generation and meta-learning-based adaptive adjustment;

[0021] Perform dynamic noise suppression and signal optimization, and denoise the signal through multimodal noise reduction autoencoders and generative adversarial networks;

[0022] Realize real-time control and feedback, and adjust signal processing parameters based on feedback control and reinforcement learning strategies to optimize the effect of signal processing.

[0023] Preferably, in the step of synchronizing and aligning the multimodal signal data, the precision time protocol or the network time protocol is used to perform preliminary time synchronization on each signal source, and the timestamp t of each modal signal is adjusted. i To achieve basic time alignment of multimodal signals, the time synchronization formula is:

[0024] t′ i =t i +δ

[0025] Among them, t i is the original timestamp of the acquired signal data, t′ i is the timestamp after time synchronization, and δ is the time deviation of devices with different modes.

[0026] Preferably, in the synchronization and alignment step of the multimodal signals, a deep temporal alignment network is constructed, the alignment network includes a bidirectional long short-term memory network and an attention mechanism, which is used to capture the temporal relationship of the multimodal signals, and the alignment loss function is:

[0027]

[0028] Among them, L align is the alignment loss value, α i,jis the attention weight of the i-th and j-th features in the signal sequence, f θ and g φ are the feature mapping functions of mode A and mode B respectively, and are the i-th and j-th signal features of modes A and B respectively.

[0029] Preferably, in the synchronization and alignment step of the modal signal, a multi-layer dynamic time warping algorithm is used to perform multi-resolution feature alignment on signals with different sampling rates, and the distance calculation formula of the multi-layer dynamic time warping is:

[0030]

[0031] Among them, d DTW (X,Y) is the alignment distance of multimodal signals X and Y, π represents the alignment path, represents the set of all possible paths, x i and j are the i-th and j-th eigenvalues ​​of signals X and Y respectively.

[0032] Preferably, in the multimodal feature extraction and fusion step, a multi-stream deep feature extraction network is used to extract features of different modalities, and dynamic weighting is performed at the fusion layer through a cross-modal attention mechanism to achieve fusion of features of different modalities. The cross-modal weighting formula is:

[0033]

[0034] Among them, α i,j is the i-th feature of mode A and the jth feature of mode B The attention weight, exp represents the exponential function, z k represents fusion features, and ⊙ represents element-wise product operation.

[0035] Preferably, in the multimodal feature fusion step, tensor decomposition technology is used to perform high-order fusion of different modal features, and the high-order tensor fusion formula is:

[0036] T=H×1W1×2W2…× n W n

[0037] Among them, T is the fusion feature tensor, H is the multimodal feature matrix, × n represents the tensor product in the nth dimension, W n is the decomposition matrix of the nth dimension.

[0038] Preferably, in the adaptive signal processing step, a convolution kernel is dynamically generated to achieve adaptive adjustment of convolution parameters, and the formula for generating the convolution kernel is:

[0039]

[0040] Where y is the output of the convolution operation, x is the input signal, * represents the convolution operation, W(x) is the dynamic convolution kernel generated according to the input signal x, and θ1 and θ2 are the dynamically generated convolution kernel parameters.

[0041] 8. A method for processing signals by a computer system according to claim 1, characterized in that in the adaptive signal processing step, adaptive adjustment is performed based on meta-learning, and the meta-learning update formula is:

[0042]

[0043] Among them, θ′ is the updated model parameter, θ is the current model parameter, α is the learning rate, represents the gradient of the parameter θ, L meta (θ) is the loss function for meta-learning.

[0044] Preferably, in the dynamic noise suppression and signal optimization step, a generative adversarial network is used to perform denoising optimization on the input signal, and the generative adversarial loss function is:

[0045]

[0046] Among them, L GAN To generate the adversarial loss value, represents the expected value, x is the real data sample, p data represents the real data distribution, D is the discriminator, G is the generator, z is the input noise sample of the generator, and p z is the distribution of noise samples.

[0047] Preferably, a device for processing signals on a computer system for implementing the method comprises:

[0048] A data acquisition module is used to collect multimodal signal data such as images, audio, time series and vibration signals, and assign a timestamp to each signal data;

[0049] Signal synchronization and alignment module, which is used to achieve timing alignment of multimodal signals through deep timing alignment network and dynamic time warping algorithm;

[0050] Feature extraction and fusion module, used to extract features of each modality and achieve fusion through high-order tensor decomposition;

[0051] Adaptive signal processing module, which is used to dynamically generate convolution kernels according to the characteristics of the input signal and perform adaptive adjustments based on meta-learning;

[0052] Dynamic noise suppression module for removing signal noise using multimodal denoising autoencoders and generative adversarial networks;

[0053] Real-time control and feedback module for dynamically optimizing signal processing parameters through closed-loop feedback control and reinforcement learning strategies.

[0054] The present invention provides a method for processing signals by a computer system. The method has the following beneficial effects:

[0055] 1. The present invention introduces a multi-level time synchronization and alignment scheme, including preliminary time synchronization, deep timing alignment network (DTAN) and multi-layer dynamic time warping (DTW) algorithm, to ensure the precise consistency of different modal signals in the time dimension. Traditional signal processing methods often find it difficult to cope with the complexity of multimodal data in terms of sampling frequency, delay and dynamic changes of signals. The synchronization and alignment scheme of the present invention effectively solves these problems, enabling signals from different sensors to achieve microsecond alignment accuracy. This high-precision time alignment not only provides a reliable foundation for the fusion and subsequent processing of multimodal signals, but also improves the system's adaptability to variable signal environments, enabling it to better support application scenarios with strong time dependence, such as medical monitoring and industrial control.

[0056] 2. The present invention adopts a multi-stream deep feature extraction network and a cross-modal attention mechanism, which can achieve effective feature extraction and fusion for signal types of different modalities. The feature extraction module of each modality can extract the spatial, temporal or frequency domain features unique to the modality, ensuring that the fine-grained features of the signal are fully captured. In addition, the cross-modal attention mechanism can adaptively adjust the fusion weights according to the importance of different modalities, dynamically highlight key features, and weaken irrelevant or noise features. This multimodal feature fusion method significantly improves the processing robustness of the system in complex, multimodal environments, brings higher accuracy and comprehensiveness to the overall expression of the signal, and is particularly suitable for scenarios such as autonomous driving that require multi-sensor fusion.

[0057] 3. The present invention endows the system with adaptive processing capabilities under different environmental conditions through dynamic convolution kernel generation and adaptive adjustment technology based on meta-learning. The dynamic convolution kernel generation mechanism automatically adjusts the convolution parameters according to the characteristics of the input signal, so that the system can adapt to the characteristics of different signals in real time, thereby improving the accuracy of feature extraction. In addition, the adaptive adjustment technology based on meta-learning enables the system to quickly update parameters in a new environment, significantly reducing the time and data requirements for retraining, thereby achieving good adaptability under limited data and time conditions. This adaptive capability enables the system to work effectively in different signal backgrounds, whether in high-noise environments, complex outdoor environments, or static indoor environments, and can maintain the efficiency and stability of signal processing.

[0058] 4. By using a multimodal denoising autoencoder and a generative adversarial network (GAN), the present invention greatly improves the noise suppression and optimization effects of the signal. The multimodal denoising autoencoder effectively removes the significant noise components in the signal by encoding and decoding, while the generative adversarial network further optimizes the clarity and signal-to-noise ratio of the signal. The denoising ability of the autoencoder ensures that the system can extract effective signal information in a high-noise environment, while the generative adversarial network can maintain the naturalness and consistency of the signal while denoising. This dual denoising mechanism enables the system to perform well in signal denoising and clarity, and is particularly suitable for application fields such as communications, audio processing, and medical imaging that have high requirements for signal-to-noise ratio.

[0059] 5. The present invention realizes real-time control and adaptive feedback of the signal processing process by introducing closed-loop feedback control and reinforcement learning strategies. When the signal processing effect is not ideal, the closed-loop feedback control mechanism can immediately adjust the parameters according to the feedback information to ensure the real-time and stability of the processing results; the reinforcement learning strategy gradually optimizes the system's processing strategy through multiple interactions to adapt to more complex dynamic environments. The introduction of reinforcement learning strategies not only improves the system's adaptability to changing environments, but also optimizes strategies according to different processing goals, so that the system can be flexibly adjusted according to different task requirements. This real-time feedback mechanism is particularly suitable for scenarios where signal characteristics change frequently, such as automatic monitoring systems in industrial manufacturing or biological signal processing in medical testing, to ensure that the system can maintain stable and efficient operation in a dynamic environment.

[0060] 6. The present invention integrates the entire process from signal acquisition, alignment, feature extraction, fusion, adaptive processing, noise suppression to control feedback to form a complete and efficient multimodal signal processing solution. The close connection and systematic design between the steps ensure that the system has good robustness and adaptability when processing complex signals, enabling it to meet the needs of various application scenarios. This systematic processing solution can improve the overall processing efficiency, reduce information loss and error accumulation in the signal processing process, and is particularly suitable for use in multimodal scenarios with higher complexity such as autonomous driving, intelligent monitoring, and smart medical care, providing the industry with an efficient, stable and easily scalable multimodal signal processing solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Flowchart of a method for processing signals for a computer system. DETAILED DESCRIPTION

[0062] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Example:

[0064] Please refer to the attached Figure 1 The present invention provides a method for processing a signal by a computer system, comprising the following steps:

[0065] Collect multi-modal signal data, and collect different modal signal data including time stamps through multiple sensors;

[0066] In this embodiment, multimodal signal data acquisition includes using multiple sensors to collect signal data of different modes, and assigning a timestamp to each collected signal data to ensure that all modal data has a consistent time stamp when collected. By providing an accurate timestamp, this step provides basic data support for subsequent signal synchronization and alignment.

[0067] Design of multimodal data acquisition module

[0068] In this embodiment, the computer system is equipped with a multimodal data acquisition module for connecting different types of sensors. These sensors include but are not limited to the following types:

[0069] Image sensor: used to collect visual information of the environment and obtain image or video data;

[0070] Audio sensor: used to collect sound signals in the environment and obtain voice or background sound data;

[0071] Temperature sensor: used to detect temperature changes in the environment or target object;

[0072] Accelerometer: used to capture the motion state or vibration signal of an object.

[0073] This module can collect signal data from multiple modalities in parallel and transmit it to the signal processing system through a unified interface, providing a complete multimodal data source for subsequent synchronous alignment and feature extraction.

[0074] Timestamp generation and assignment

[0075] In this embodiment, the timestamp assignment module assigns an accurate timestamp to each piece of collected data to mark the time when the signal was collected. The timestamp generation uses the Precision Time Protocol (PTP) or the Network Time Protocol (NTP). Depending on the sensor device, the time deviation is calculated and the corresponding time stamp is assigned when the data is collected. The specific implementation method is as follows:

[0076] Timestamp generation: When collecting data, the system obtains the clock information of the collection device in real time, corrects the deviation through the protocol, and generates an accurate collection timestamp t i ;

[0077] Timestamp assignment: The system automatically assigns timestamps to i Associated with each piece of collected data i So that each signal data has a unique time stamp.

[0078] Timestamp synchronization mechanism

[0079] In this embodiment, in order to ensure the consistency of timestamps of different modes, the system implements basic synchronization of timestamps in the initial acquisition stage. This mechanism achieves preliminary alignment by adjusting the time deviation δ of different devices. The specific time synchronization formula is:

[0080] t′ i =t i +δ

[0081] Among them, t′ i is the original timestamp, δ is the time deviation, t′ i is the corrected synchronization timestamp. This step ensures that the acquisition time of different modal data is on the same time base, providing preliminary time consistency support for the subsequent timing alignment step.

[0082] Structured storage of multimodal data

[0083] In this embodiment, the collected multimodal data is stored in the data storage module of the system after being given a timestamp. To ensure convenient data access and structured management, the storage module stores each piece of data according to the type of modality, timestamp, data content and other information. The specific structure includes:

[0084] Modality identification: modality information of each piece of data (such as image, audio, temperature);

[0085] Timestamp: Acquisition timestamp t′ i ;

[0086] Data content: contains specific signal data values.

[0087] Through the above steps, unified acquisition, time marking and basic synchronization of multimodal signals are achieved, which ensures the time consistency and structured storage of data in a multimodal environment, and provides efficient basic support for the subsequent synchronization, feature extraction and fusion of signals.

[0088] Synchronize and align the collected multi-modal signal data, build a deep time alignment network and combine it with a time warping algorithm to achieve time consistency of different modal signals;

[0089] This embodiment includes three parts: preliminary time synchronization, deep timing alignment network (DTAN) construction and multi-layer dynamic time warping (DTW) algorithm, which are used to ensure the time consistency of different modal signals and the multi-resolution alignment effect.

[0090] Initial time synchronization

[0091] In this embodiment, before performing deep timing alignment on the signal data, the system first achieves preliminary time synchronization by adjusting the timestamp. The synchronization method is based on the generated timestamp t i and time deviation δ, and complete the preliminary time alignment of multimodal data through the following formula:

[0092] t′ i =t i +δ

[0093] Among them, t i is the original timestamp, δ is the time deviation between each modality device, t′ i It is the timestamp after preliminary synchronization. This step ensures that the signals of different modalities have basic time consistency and provides a reliable time reference for subsequent depth alignment.

[0094] Deep Temporal Alignment Network (DTAN) Construction

[0095] In this embodiment, in order to achieve higher precision time alignment, the system constructs a deep time alignment network (DTAN). The network adopts a combination of a bidirectional long short-term memory network (Bi-LSTM) and an attention mechanism to capture the bidirectional time characteristics of different modal signals and automatically assign alignment weights between signals.

[0096] Bi-LSTM network: Bi-LSTM can accurately align the complex time-dependent characteristics of the signal by considering both the forward and backward timing information of the signal. During the alignment process, the Bi-LSTM network extracts temporal features from the signal sequence of each modality and converts them into feature representations with temporal consistency.

[0097] Attention mechanism: The attention mechanism is used to calculate the similarity weights between different time points and assign higher alignment weights to important time points. This mechanism ensures that during the alignment process, the system can automatically focus on the important temporally relevant information in each modal signal, thereby improving the accuracy of alignment.

[0098] Alignment loss function: To further optimize the alignment effect of DTAN, this embodiment defines an alignment loss function. Given the signal sequences of modality A and modality B and The system minimizes the alignment loss function L align To achieve optimal alignment:

[0099]

[0100] Among them, α i,j is the attention weight, which is used to dynamically adjust the alignment weight f at each time point θ and g φ They are the feature mapping functions of modes A and B respectively; and are the time series characteristics of modes A and B respectively.

[0101] Multi-layer Dynamic Time Warping (DTW) algorithm

[0102] In this embodiment, the system further applies a multi-layer dynamic time warping (DTW) algorithm based on DTAN to achieve multi-resolution timing alignment of signals with different sampling rates. The DTW algorithm can dynamically align signal sequences at multiple scales to ensure that signals of different modes are aligned.

[0103] Consistency in time.

[0104] Multi-layer DTW alignment: In this embodiment, in order to deal with the data sampling frequency differences of different modal signals, the system uses the DTW algorithm on each feature layer to achieve signal alignment layer by layer. Specifically, the system calculates the DTW distance d DTW(X,Y) to obtain the optimal alignment path between different modal signals:

[0105]

[0106] Among them, d DTW (X,Y) represents the alignment distance between signals X and Y, π represents the alignment path, Represents the set of all possible paths.

[0107] Multi-resolution alignment effect: By performing DTW alignment at multiple levels, this embodiment achieves multi-resolution alignment of signal data at different sampling rates, so that each layer of features has temporal consistency, providing an accurate time alignment basis for subsequent feature extraction and fusion steps.

[0108] In summary, this embodiment completes the time synchronization and alignment of multimodal signals through the collaborative processing of preliminary time synchronization, deep timing alignment network (DTAN) and multi-layer dynamic time warping (DTW) algorithm, so that different modal signals have precise consistency in the time dimension, providing a high-quality time reference and alignment basis for subsequent feature extraction and fusion steps.

[0109] Extract and fuse the features of multimodal signals, extract the features of each modality through a multi-stream deep feature extraction network, and generate a unified fusion feature representation through a feature fusion strategy;

[0110] This embodiment includes the construction of a multi-stream deep feature extraction network, the application of a cross-modal attention mechanism, and the use of a high-order tensor fusion method, which are used to extract features and deeply fuse signals of different modalities to achieve a unified expression of multimodal information.

[0111] Multi-stream deep feature extraction network

[0112] In this embodiment, according to the characteristics of different modal signals, the system designs a multi-stream deep feature extraction network to extract the features of each modality separately. Each modal data stream uses a separate feature extraction model, which is selected and designed according to the signal type.

[0113] Visual signal feature extraction: For visual signals such as images and videos, this embodiment uses a convolutional neural network (CNN) to extract spatial features. The multi-layer convolution and pooling operations of CNN can capture local features and global pattern information in the image, ensuring that the feature representation of the visual signal is accurate and reliable.

[0114] Audio signal feature extraction: For audio signals, this embodiment uses a long short-term memory network (LSTM) to extract the temporal features of the audio. The LSTM structure has an advantage in capturing the temporal dependency of audio signals and is suitable for feature extraction of continuous audio signals such as speech and background sound.

[0115] Sensor signal feature extraction: For sensor signals such as temperature and acceleration, a one-dimensional convolutional network (1DCNN) is used in this embodiment to extract time series features.

[0116] 1DCNN can extract local temporal features of signals through convolution operations and is suitable for analyzing pattern changes in sensor signals.

[0117] Feature representation: The feature representation of each modal signal is as follows:

[0118]

[0119] in, and are the feature representations of modes A and B respectively, and is the feature extraction function of modes A and B, θ A and θ B is the model parameter.

[0120] Cross-modal attention mechanism

[0121] In this embodiment, in order to achieve efficient fusion of different modal features, the system designs a cross-modal attention mechanism in the fusion layer. This attention mechanism can dynamically adjust the fusion weight according to the importance of the modal features to highlight the key parts of the multimodal information and weaken irrelevant information.

[0122] Attention weight calculation: given the features of modality A and modality B and The system calculates similarity weights through the attention mechanism

[0123]

[0124] Here, exp represents the exponential function and · represents the dot product operation, which is used to measure the similarity between two features. i,j Represents the fusion weight between the i-th feature of modality A and the j-th feature of modality B.

[0125] Weighted fusion: The system performs weighted fusion of features according to attention weights to generate a unified cross-modal feature representation z k :

[0126]

[0127] Among them, ⊙ represents the element-by-element product, z k is the fused feature representation. Through weighted fusion, the system can achieve effective fusion of multimodal features so that subsequent signal processing is more holistic and consistent.

[0128] High-level tensor fusion

[0129] In order to further enhance the expressive power of fused features and the complementarity of multimodal information, this embodiment also uses a high-order tensor decomposition method to process fused features. Tensor decomposition can capture the nonlinear interaction between different modalities and generate a more global and expressive high-dimensional feature representation.

[0130] Tensor decomposition process: The fused feature matrix H is decomposed into the final feature representation T through high-order tensor decomposition:

[0131] T=H×1W1×2W2…× n W n

[0132] Among them, H is the fusion feature matrix, W n is the decomposition matrix, × n Represents the tensor product operation in the nth dimension. Through multidimensional tensor decomposition, high-order relationships between multimodal features can be extracted, making the fused feature representation more accurate and rich.

[0133] In summary, this embodiment achieves accurate extraction and deep fusion of multimodal signal features through a multi-stream deep feature extraction network, a cross-modal attention mechanism, and high-order tensor fusion technology, providing the system with a high-dimensional, unified multimodal feature representation to meet the input requirements of subsequent adaptive signal processing.

[0134] Implement adaptive signal processing to adapt to different environmental characteristics through dynamic convolution kernel generation and meta-learning-based adaptive adjustment;

[0135] This embodiment includes two parts: dynamic convolution kernel generation and adaptive adjustment based on meta-learning, which are used to adaptively adjust the signal processing capability of the system under different signal environments.

[0136] Dynamic convolution kernel generation

[0137] In this embodiment, in order to enable the system to adaptively adjust the parameters of the convolution operation under different environmental conditions, a dynamic convolution kernel generation mechanism is designed. The dynamic convolution kernel automatically generates convolution kernel parameters according to the input signal characteristics, so that the convolution operation can better extract and characterize key information under different signal characteristics.

[0138] The generation of dynamic convolution kernels is based on the characteristics of the input signal. By learning the convolution kernel parameters under different feature modes, the system can adapt to the changes of different time-varying features in the signal.

[0139] Implementation: For the input signal x, the system dynamically generates the convolution kernel W(x) to perform adaptive convolution operation on the signal. The specific generation formula is as follows:

[0140]

[0141] Where y is the output result of the convolution operation, * represents the convolution operation, θ1 and θ2 are dynamically generated convolution kernel parameters, which are adaptively adjusted based on the characteristics of the input signal x.

[0142] By dynamically generating convolution kernels, the system can automatically adjust the response capability of the convolution according to the characteristics of different input signals, thereby improving the feature extraction effect in complex environments.

[0143] Adaptive Adjustment Based on Meta-Learning

[0144] In order to further enhance the adaptability of the system in a changing environment, this embodiment also adopts an adaptive adjustment technology based on meta-learning. Meta-learning can enable the system to quickly adapt to new environmental requirements within a small amount of data and a limited training cycle, and achieve adaptive adjustment.

[0145] The principle of meta-learning: Meta-learning trains a "learning algorithm" so that the system can quickly update parameters from limited environmental data. This rapid adaptability has advantages in dynamic and unknown environments and can greatly improve the generalization ability and stability of the system.

[0146] In this embodiment, when the system processes a new signal environment, the model parameters are quickly updated through a meta-learning algorithm. Given the current model parameters θ, the adaptive update formula of the meta-learning is:

[0147]

[0148] Among them, θ′ is the updated model parameter, α is the learning rate, represents the gradient of the parameter θ, L meta (θ) is the meta-learning loss function, which is used to measure the adaptability of the current parameters in the new environment. By minimizing L meta (θ), the system can quickly adjust the model parameters to adapt to the changes in signal characteristics in different environments.

[0149] In summary, this embodiment realizes the adaptive signal processing capability of the system in a changing environment through dynamic convolution kernel generation and adaptive adjustment based on meta-learning. By dynamically adjusting the convolution kernel parameters and quickly updating the model parameters, the system can maintain processing accuracy and stability in different signal environments, laying a solid foundation for subsequent noise suppression and signal optimization.

[0150] Perform dynamic noise suppression and signal optimization, and denoise the signal through multimodal noise reduction autoencoders and generative adversarial networks;

[0151] This embodiment includes the use of a multimodal denoising autoencoder and the application of a generative adversarial network (GAN) to achieve efficient dynamic noise suppression and signal optimization in a changing signal environment.

[0152] Multimodal Denoising Autoencoder

[0153] In this embodiment, the system uses a multimodal denoising autoencoder (Denoising Autoencoder) to process the noise signal. The autoencoder removes the noise component from the input signal through feature encoding and reconstruction process, thereby generating a clear signal representation.

[0154] The autoencoder consists of an encoder and a decoder. The encoder is responsible for mapping the input signal to a low-dimensional latent space, thereby compressing and filtering out noise.

[0155] The decoder decodes the compressed features back into the reconstructed signal. This process can maximize the suppression of noise without losing the effective signal.

[0156] In the noise reduction process, the input signal x is passed through the encoder f enc Extract the low-dimensional feature representation z and then pass it through the decoder f dec Reconstructing into a clear signal The specific process is as follows:

[0157]

[0158] Among them, z is the low-dimensional representation of the signal, is the reconstructed signal after denoising. In order to improve the denoising effect, this embodiment defines a reconstruction loss function L recon , to optimize the denoising performance of the autoencoder:

[0159]

[0160] This loss function is used to measure the difference between the reconstructed signal and the original input signal. The system minimizes L recon Improve the reconstruction quality and make the denoised signal closer to the original signal characteristics.

[0161] Generative Adversarial Network (GAN) Denoising

[0162] In order to further enhance the noise reduction capability of the system, a generative adversarial network (GAN) is also used in this embodiment for signal optimization. GAN consists of a generator and a discriminator. The generator generates a noise-free signal and the discriminator judges the quality of the signal. The generator and the discriminator continuously improve the signal denoising effect through adversarial training.

[0163] The generator of GAN learns to generate a noise-free signal that is close to reality from a noisy environment, while the discriminator is used to determine whether the input signal is a real and clear signal. Through adversarial training between the generator and the discriminator, the generator can gradually learn the signal pattern to remove noise.

[0164] In the GAN denoising process, the generator G receives the noise signal input z and generates a denoised signal G(z). The discriminator D distinguishes the generated signal G(z) from the real signal x. To optimize the denoising ability of the generator, this embodiment defines the generative adversarial loss function L GAN :

[0165]

[0166] Among them, L GAN To generate the adversarial loss value, represents the expected value, x is the real data sample, p data represents the real data distribution, D(x) is the output of the discriminator for the real signal, G(z) is the denoised signal generated by the generator, and p z is the distribution of the noise signal. The system minimizes L GAN The denoising capability of the generator is improved, and the generated signal gradually approaches the characteristics of a noise-free signal.

[0167] In summary, this embodiment achieves efficient dynamic noise suppression and signal optimization of multimodal signals through the combination of multimodal denoising autoencoder and generative adversarial network (GAN). The multimodal denoising autoencoder is responsible for removing significant noise components in the signal, while GAN further optimizes the quality of the denoised signal, enabling the system to output clearer and more accurate signals, providing high-quality signal input for subsequent real-time control and feedback.

[0168] Realize real-time control and feedback, and adjust signal processing parameters based on feedback control and reinforcement learning strategies to optimize the effect of signal processing.

[0169] This embodiment includes two parts: closed-loop feedback control and strategy optimization based on reinforcement learning. By real-time monitoring of signal processing effects and adjusting system parameters, it ensures that the system can adaptively optimize signal processing effects in a dynamic environment.

[0170] Closed-loop feedback control

[0171] In this embodiment, the system monitors the signal processing process in real time and adjusts parameters through closed-loop feedback control. The closed-loop feedback mechanism can automatically detect the signal output quality and adjust the parameters of the processing module according to the feedback to optimize the signal processing effect.

[0172] The core of closed-loop feedback control is to use real-time feedback information to automatically adjust signal processing parameters to ensure that the system is always in the optimal processing state. Through feedback control, the system can quickly adjust when the signal processing effect deviates from expectations, thereby avoiding a decrease in processing quality.

[0173] The system monitors the processing output in each signal processing cycle, measures the deviation between the actual output signal y and the expected output r, and adjusts the signal processing parameter θ through the following feedback control formula:

[0174]

[0175] Among them, Δθ is the parameter adjustment amount, K p , K i , K d are the proportional, integral and differential gain coefficients, r is the desired output, and y is the actual output.

[0176] Through the PID control formula, the system can fine-tune the parameters according to the proportional, integral and differential control to optimize the signal processing effect. This closed-loop feedback mechanism ensures that the system can dynamically respond to external changes and maintain the stability of the signal processing effect in a complex environment.

[0177] Strategy optimization based on reinforcement learning

[0178] In a complex signal processing environment, the system may encounter unpredictable environmental changes, so this embodiment introduces a strategy optimization method based on reinforcement learning. The reinforcement learning strategy enables the system to learn through interaction with the environment and gradually form the optimal processing strategy to cope with different environmental changes.

[0179] The reinforcement learning strategy allows the system to try different actions in the environment (such as adjusting processing parameters) and learn based on feedback rewards, thereby gradually optimizing the processing strategy. This method is particularly suitable for dealing with dynamic and uncertain environments.

[0180] In this embodiment, the system models the signal processing parameter adjustment problem as a reinforcement learning task. t Indicates the current processing effect, action a t represents the parameter adjustment operation, and the reward r t Indicates the improvement of processing effect. The system optimizes the signal processing strategy by maximizing the cumulative reward. The specific optimization formula is as follows:

[0181]

[0182] Among them, J(π) is the long-term reward of the strategy, π is the current strategy, and γ is the discount factor used to balance short-term and long-term rewards. The system gradually adjusts the processing strategy through reinforcement learning, so that the signal processing parameters can be adaptively optimized, improving the system's processing effect in uncertain environments.

[0183] In summary, this embodiment realizes real-time control and feedback adjustment of the signal processing system through closed-loop feedback control and strategy optimization based on reinforcement learning. Closed-loop feedback control ensures the signal processing effect of the system within the normal fluctuation range, and reinforcement learning strategy optimization enables the system to cope with more complex and unpredictable environmental changes and form the optimal processing strategy. This real-time control and feedback mechanism enables the system to adaptively adjust parameters in a dynamic environment to ensure the continuity and stability of signal processing.

[0184] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for processing a signal by a computer system, characterized in that: The following steps are involved: Collect multimodal signal data, and collect different modal signal data including time stamps through multiple sensors, the sensors include but are not limited to image sensors, audio sensors, temperature sensors, and acceleration sensors; Synchronize and align the collected multi-modal signal data, build a deep time alignment network and combine it with a time warping algorithm to achieve time consistency of different modal signals; Extract and fuse the features of multimodal signals, extract the features of each modality through a multi-stream deep feature extraction network, and generate a unified fusion feature representation through a feature fusion strategy; Implement adaptive signal processing to adapt to different environmental characteristics through dynamic convolution kernel generation and meta-learning-based adaptive adjustment; Perform dynamic noise suppression and signal optimization, and denoise the signal through multimodal noise reduction autoencoders and generative adversarial networks; Realize real-time control and feedback, and adjust signal processing parameters based on feedback control and reinforcement learning strategies to optimize the effect of signal processing.

2. A method for processing signals by a computer system according to claim 1, characterized in that: In the step of synchronizing and aligning the multimodal signal data, the precision time protocol or the network time protocol is used to perform preliminary time synchronization on each signal source, and the timestamp t of each modal signal is adjusted. i To achieve basic time alignment of multimodal signals, the time synchronization formula is: t′ i =t i +δ Among them, t i is the original timestamp of the acquired signal data, t′ i is the timestamp after time synchronization, and δ is the time deviation of devices with different modes.

3. The method for processing a signal by a computer system according to claim 1, characterized in that: In the synchronization and alignment step of the multimodal signals, a deep temporal alignment network is constructed. The alignment network includes a bidirectional long short-term memory network and an attention mechanism to capture the temporal relationship of the multimodal signals. The alignment loss function is: Among them, L align is the alignment loss value, α i,j is the attention weight of the i-th and j-th features in the signal sequence, f θ and g φ are the feature mapping functions of mode A and mode B respectively, and are the i-th and j-th signal features of modes A and B respectively.

4. The method for processing a signal by a computer system according to claim 1, characterized in that: In the synchronization and alignment step of the modal signal, a multi-layer dynamic time warping algorithm is used to perform multi-resolution feature alignment on signals with different sampling rates. The distance calculation formula of the multi-layer dynamic time warping is: Among them, d DTW (X,Y) is the alignment distance of multimodal signals X and Y, π represents the alignment path, represents the set of all possible paths, x i and j are the i-th and j-th eigenvalues ​​of signals X and Y respectively.

5. The method for processing a signal by a computer system according to claim 1, characterized in that: In the multimodal feature extraction and fusion step, a multi-stream deep feature extraction network is used to extract features of different modalities, and dynamic weighting is performed at the fusion layer through a cross-modal attention mechanism to achieve fusion of features of different modalities. The cross-modal weighting formula is: Among them, α i,j is the i-th feature of mode A and the jth feature of mode B The attention weight, exp represents the exponential function, z k represents fusion features, and ⊙ represents element-wise product operation.

6. The method for processing a signal by a computer system according to claim 1, characterized in that: In the multimodal feature fusion step, tensor decomposition technology is used to perform high-order fusion of different modal features. The high-order tensor fusion formula is: T=H×1W1×2W2…× n W n Among them, T is the fusion feature tensor, H is the multimodal feature matrix, × n represents the tensor product in the nth dimension, W n is the decomposition matrix of the nth dimension.

7. The method for processing a signal by a computer system according to claim 1, characterized in that: In the adaptive signal processing step, a convolution kernel is dynamically generated to achieve adaptive adjustment of convolution parameters. The formula for generating the convolution kernel is: Where y is the output of the convolution operation, x is the input signal, * represents the convolution operation, W(x) is the dynamic convolution kernel generated according to the input signal x, and θ1 and θ2 are the dynamically generated convolution kernel parameters.

8. The method for processing a signal by a computer system according to claim 1, characterized in that: In the adaptive signal processing step, adaptive adjustment is performed based on meta-learning, and the meta-learning update formula is: Among them, θ′ is the updated model parameter, θ is the current model parameter, α is the learning rate, represents the gradient of the parameter θ, L meta (θ) is the loss function for meta-learning.

9. The method for processing a signal by a computer system according to claim 1, characterized in that: In the dynamic noise suppression and signal optimization step, a generative adversarial network is used to perform denoising optimization on the input signal, and the generative adversarial loss function is: Among them, L GAN To generate the adversarial loss value, represents the expected value, x is the real data sample, p data represents the real data distribution, D is the discriminator, G is the generator, z is the input noise sample of the generator, and p z is the distribution of noise samples.

10. A computer system signal processing device for implementing the method according to any one of claims 1 to 9, comprising: A data acquisition module is used to collect multimodal signal data such as images, audio, time series and vibration signals, and assign a timestamp to each signal data; Signal synchronization and alignment module, which is used to achieve timing alignment of multimodal signals through deep timing alignment network and dynamic time warping algorithm; Feature extraction and fusion module, used to extract features of each modality and achieve fusion through high-order tensor decomposition; Adaptive signal processing module, which is used to dynamically generate convolution kernels according to the characteristics of the input signal and perform adaptive adjustments based on meta-learning; Dynamic noise suppression module for removing signal noise using multimodal denoising autoencoders and generative adversarial networks; Real-time control and feedback module for dynamically optimizing signal processing parameters through closed-loop feedback control and reinforcement learning strategies.

Images