Signal processing method and computer equipment

Abstract

The invention relates to the technical field of computers, in particular to a signal processing method and computer equipment, and the method comprises the steps: collecting signal data from a multi-mode signal source, and carrying out compressed sensing processing on the collected data to reduce the data dimension and redundancy; extracting time-frequency characteristics of the signals through short-time Fourier transform or wavelet transform, and capturing local dynamic characteristics; spatial features are extracted by using a convolutional neural network, weighted fusion is performed on multi-modal features through an attention mechanism, and complementarity of information between modals is optimized; dynamically adjusting filter parameters and a task allocation strategy based on deep reinforcement learning, and realizing signal adaptive processing in a complex dynamic environment; the model parameters are optimized online through a closed-loop feedback mechanism, and the signal processing precision and efficiency are continuously improved. According to the method, the problems of low processing efficiency, poor adaptability, insufficient multi-modal fusion capability and lack of a feedback optimization mechanism in the prior art are solved.

Description

[0001] Existing methods usually perform modal fusion through simple splicing, weighted averaging, etc., and fail to fully exploit the relevance and complementarity between different modal signals, resulting in limited fusion effect. For example, in the multi-modal analysis combining computer vision and natural language processing, the simple feature splicing method is difficult to capture the deep semantic association between modalities, affecting the overall performance of the model.

[0002] Further, most signal processing models use static process design and lack feedback mechanisms to optimize processing performance. In existing technologies, the results of signal processing cannot be fed back to model parameter adjustment or optimization strategy update, so that the system cannot dynamically improve according to the processing results. This design without feedback mechanism limits the intelligent development of signal processing. In addition, most existing signal processing frameworks are designed for a single signal type and lack of universality, making it difficult to meet the needs of multi-modal complex signal processing.

[0003] Therefore, the present application provides a signal processing method and a computer device to solve the problems of the prior art. SUMMARY

[0004] In view of the problems of the prior art in the signal processing process, such as lack of efficiency, poor adaptability, weak universality and lack of feedback optimization mechanism, the present application provides a signal processing method and a computer device. Through the design of an innovative signal processing framework, the present application aims to realize efficient processing, adaptive optimization and dynamic feedback adjustment of multi-modal signals, and adapt to complex and variable signal processing requirements.

[0005] To achieve the above purpose, the present application realizes the following technical scheme: a signal processing method, comprising the following steps:

[0006] Collecting signal data from a multi-modal signal source, and performing compressive sensing processing and preprocessing on the collected signal data;

[0007] Extracting time-frequency features of the signal, including performing short-time Fourier transform or wavelet transform on the signal data;

[0008] Using a deep learning model to extract spatial features from the time-frequency features, and performing multi-modal feature fusion through an attention mechanism;

[0009] Based on the feature fusion result, using deep reinforcement learning to optimize signal processing parameters, including filter parameters and task allocation strategy;

[0010] Outputting the signal processing result and dynamically adjusting the signal processing model according to the feedback.

[0011] Preferably, the compressive sensing processing comprises the following steps

[0012] The signal data is reduced in dimension using a sampling matrix to generate a compressed signal:

[0013] y = Φx

[0014] where x is the original signal with dimension N; Φ is the sampling matrix with dimension M x N, where M << N; y is the compressed signal with dimension M; a sparse basis matrix is used to recover the sparse representation of the signal:

[0015]

[0016] where Ψ is the sparse basis matrix used to transform the signal into sparse representation; α is the sparse coefficient vector with dimension N; ∈ is the error tolerance used to control the accuracy of signal recovery.

[0017] Preferably, the preprocessing includes:

[0018] Noise reduction processing: an adaptive filter is used to reduce noise in the signal data, and the filter parameter update formula is:

[0019]

[0020] where w n is the filter weight vector with dimension d, where d is the dimension of the signal feature; μ is the learning rate, which is in the range of (0, 1]; d n is the expected signal, which is the target output value at the current time step; x n is the input signal vector at the current time step with dimension d.

[0021] Normalization processing: the signal data is normalized to have a range of [0, 1], and the normalization formula is:

[0022]

[0023] where x i is the i-th signal sample; N is the number of signal samples; μ is the mean of the signal, which is used to measure the central tendency of the signal; σ is the standard deviation of the signal, which is used to measure the dispersion of the signal; x′ i is the normalized signal sample, whose value is limited to the range of [0, 1].

[0024] Preferably, the time-frequency feature extraction includes:

[0025] Performing short-time Fourier transform on the signal data:

[0026]

[0027] where X STFT(t,f) is the time-frequency feature matrix of the signal, t represents the time window position, and f represents the frequency; x[n] is the value of the input signal at the nth sampling point; w[n-t] is the window function used to locally limit the signal, and common window functions include rectangular window, Hamming window, and Gaussian window; e -j2πfn is the complex exponential basis function used to calculate the amplitude and phase distribution of the signal in the frequency domain; n is the sampling point index of the signal,

[0028] Alternatively, a wavelet transform is performed on the signal data:

[0029]

[0030] where X DWT (t,s) is the wavelet feature matrix of the signal, t represents the time position, and s represents the scale factor; x[n] is the value of the input signal at the nth sampling point; is the mother wavelet function, common mother wavelets include Morlet wavelet, Haar wavelet, and Daubechies wavelet, used to realize multi-scale decomposition of the signal; t is the time index, controlling the translation of the wavelet function; s is the scale factor, controlling the compression or expansion of the wavelet function, used to adjust the time and frequency resolution.

[0031] Preferably, the spatial feature extraction is realized by a convolutional neural network, and the convolution operation is defined as:

[0032]

[0033] where, is the convolution output feature, representing the activation value at position i,j after convolution operation; X is the input feature matrix, with dimensions HxW, where H and W are the height and width of the input matrix, respectively; is the convolution kernel, with size MxN, where M and N are the height and width of the convolution kernel, respectively; K m,n is the weight of the convolution kernel at position m,n; b is the bias term, which is a scalar value used to add an offset to the convolution result; σ is the activation function, common activation functions include:

[0034] ReLU is σ(x) = max(0,x), ReLU: Rectified Linear Unit;

[0035] Sigmoid:

[0036] Tanh:

[0037] In addition, i and j are the row and column indexes of the feature matrix after convolution operation, and the dimension of the output feature matrix is (H-M+1) x (W-N+1), which is determined by the dimension of the input matrix and the size of the convolution kernel.

[0038] m and n are the row and column indexes within the convolution kernel, used for pixel-by-pixel convolution calculation.

[0039] Preferably, the multi-modal feature fusion is realized through an attention mechanism, and the fusion formula is:

[0040]

[0041] where F fusion is the fused global feature matrix, representing the result of multi-modal feature fusion, with a dimension of d f , where d f is the output feature dimension; M is the number of modalities, i.e., the number of different modalities participating in fusion; is the feature matrix of the i-th modality, with a dimension of d i , where d i is the feature dimension of modality i; W i is the weight matrix of modality i, with a dimension of d f x d i , used to measure the importance of modality i features in the fusion process; softmax is a normalization function that normalizes the fused feature values into a probability distribution, defined as:

[0042]

[0043] where z k is the k-th element in the feature matrix, z j is the set of all feature values; is an exponential function used to amplify differences; i is the modality index, taking values in the range 1, 2, …, M; W i is the weight value determined through training of the deep learning model, used to dynamically allocate the importance of different modalities features; the feature fusion result F fusion retains the multi-modal feature correlation, suitable for subsequent deep reinforcement learning optimization module.

[0044] Preferably, the deep reinforcement learning optimization signal processing parameters include:

[0045] Define the current state s t , action a t and immediate reward r t :

[0046] s t: represents the characteristic state of the current signal, including the time-frequency characteristics and the multi-modal feature fusion results of the signal, with a dimension of d s , wherein d s is the dimension of the state vector;

[0047] a i : represents the action taken under the current state s t , including filter parameter adjustment and task allocation strategy update, with a dimension of d a , wherein d a is the dimension of the action vector;

[0048] r t : is the immediate reward, defined as the improvement of signal processing effect:

[0049] r t = SNR out - SNR in - λ · C

[0050] , wherein SNR out is the signal-to-noise ratio after signal processing; SNR in is the signal-to-noise ratio before signal processing; λ is a weight factor of calculation cost, used to balance processing performance and calculation overhead; C is the calculation cost of the current signal processing;

[0051] Update the Q value function of the deep Q network:

[0052]

[0053] , wherein Q(s t , a t ): represents the value function of taking action a t in state s t , measuring the long-term cumulative income of the action; γ: is a discount factor, with a value range of [0, 1], used to balance the weight of immediate reward and future income; s t+1 : is the next state of the current state s t after action a t ; a t+1 : is the possible action set of the next state s t+1 ;

[0054] Optimize the parameters of the deep Q network using the gradient descent algorithm:

[0055]

[0056] , wherein Θ: is the parameter of the deep Q network; η: is the learning rate, controlling the step size of parameter update; is the loss function, used to minimize the error between the actual Q value and the target Q value, defined as:

[0057]

[0058] where N: is the number of samples;

[0059] Q target (s i ,a i )=r i +γmax a′ Q(s i+1 ,a′),for the target Q value.

[0060] Preferably, the task allocation strategy is implemented by optimizing the following objective function:

[0061]

[0062] where P = {P1, P2, …, P N}: is the task allocation strategy, representing a specific scheme of allocating N tasks to different computing resources; P i : is the allocation scheme of the i-th task, taking values in the set of available computing resources; N: is the total number of tasks; C i (P i ): is the computing cost of the i-th task after being allocated to resource P i , calculated by the formula:

[0063]

[0064] where W i is the computing load of the i-th task, measured in operations; is the computing power of resource P i , measured in operations per second; L(P): is the total scheduling delay of task allocation, calculated by the formula:

[0065]

[0066] where T start,i is the start-up time of task i on the computing resource; T compute,i is the computing time of task i, defined as:

[0067]

[0068] T comm,i is the communication time of task i, representing the data transfer time required after task allocation to the target computing resource; λ: is the weight factor of delay cost, used to balance the optimization objective between computing cost and delay, taking values in the range [0, 1].

[0069] Preferably, the signal processing model is optimized by pruning and quantization, with the optimization formula being:

[0070]

[0071] wherein Θ: is a parameter set of the signal processing model, with dimension d Θ , wherein d Θ is the total number of model parameters; is a loss function of the model, used to measure the error between the input signal X and the model predicted output and the true label y, defined as:

[0072]

[0073] wherein X={x1,x2,…,x N}: is a set of input signal samples, containing N samples; is the model predicted output of the i-th sample; y i : is the true label of the i-th sample; ||Θ||0: is the number of non-zero terms of the model parameters, used to measure the model complexity; β: is a regularization weight factor, with a value range of [0,∞), used to balance the weight between the loss function and the model complexity.

[0074] Preferably, a computer device comprises a processor, a memory, and a computer program stored on the memory and executable on the processor, wherein the computer program is executed by the processor to implement the steps of the signal processing method.

[0075] The present application provides a signal processing method and a computer device. The present application has the following advantages:

[0076] 1. The present application introduces the compressive sensing technology to reduce the dimensionality of the multi-modal signal, combines the sparse matrix representation and the optimization algorithm, and greatly reduces the redundancy of data storage and transmission, thereby realizing efficient signal processing. Compared with the prior art, the present application effectively solves the problem of high resource occupation and significantly improves the real-time performance of signal processing, and is suitable for resource-limited embedded devices or dynamic complex signal environments.

[0077] 2. The present application dynamically optimizes the signal processing parameters (such as filter parameters and task allocation strategies) through deep reinforcement learning, adjusts the system processing strategy in real time according to the current signal characteristics, and ensures the optimization of the processing effect. Compared with the prior art, the present application overcomes the problem of unstable performance in complex dynamic signal environment, and makes the system have stronger robustness and environmental adaptability.

[0078] 3、The application combines the spatial feature extraction method based on convolutional neural network and the multi-modal feature fusion strategy of attention mechanism, effectively improving the feature extraction and fusion performance of multi-modal signals. Compared with the simple splicing or weighted average fusion mode of multi-modal signals in the prior art, the application can give dynamic weights to modal features, fully excavate the correlation and key features between signals, thereby solving the problem of insufficient information interaction between modes in the prior art, and greatly improving the signal fusion precision and processing effect.

[0079] 4、The application designs a signal processing closed-loop system based on feedback optimization, which continuously optimizes the processing model through online learning and model pruning technology, and realizes the continuous self-regulation and resource optimization of signal processing in a dynamic environment. Compared with the prior art which lacks feedback mechanism and cannot dynamically adjust the system according to environmental changes, the application can optimize the signal processing flow in real time according to the processing result, and solve the problem of signal processing flow staticity. At the same time, the application realizes the compatibility of multi-modal signals such as audio, video and biological signals through a unified framework, providing strong support for the universality and expansibility of signal processing. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 A signal processing method flowchart. DETAILED DESCRIPTION

[0081] The technical solutions of the application will be described in detail below with reference to the drawings of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0082] Please refer to the accompanying Figure 1 The application provides a signal processing method, and the steps of the method will be described in detail below.

[0083] S1, collecting signal data from a multi-modal signal source, and performing compressive sensing processing and preprocessing on the collected signal data

[0084] Firstly, this step involves obtaining signal data from a multi-modal signal source, and performing compressive sensing processing and preprocessing on the obtained data, laying a foundation for subsequent feature extraction and signal processing. Multi-modal signal sources are diverse, including but not limited to audio signals, video signals and biological signals. On the basis of ensuring signal integrity, this step reduces the amount of data through compressive sensing technology, and improves the processing quality of signals through noise reduction and normalization and other operations.

[0085] In practical applications, due to the significant differences in the collection methods and characteristics of multi-modal signals, it is necessary to design appropriate collection and processing schemes for different modalities. Generally, these processing steps can be completed by hardware sensors combined with edge computing devices, and the data can be processed in real time or offline through software algorithms.

[0086] Specifically, in this embodiment, the collection of signal data includes the following contents:

[0087] In one possible implementation, the audio signal can be collected by a microphone, and its output is one-dimensional waveform data, represented as where T1 is the number of sampling points. The sampling frequency can be set according to actual needs, for example, a sampling rate of 16 kHz is usually used in speech signal processing.

[0088] For the primary frequency signal, generally, multiple frames of high-frequency data can be collected by a camera, and its output is multi-channel frame data, represented as where T2 is the number of frames, H, W, and C represent the height, width, and channel number of the frame, respectively.

[0089] As an option, the video signal can be down-sampled to reduce the amount of data, for example, reducing the frame rate from 60 fps to 30 fps. For biological signals, such as electrocardiogram (ECG) or electroencephalogram (EEG), wearable sensors can be used for collection, and their output is usually one-dimensional time series signals, represented as x bio ∈ where T3 is the number of sampling points.

[0090] Specifically, the sampling frequency is set according to the characteristics of the signal, for example, ECG is usually 500 Hz, and EEG is 1000 Hz.

[0091] After the above collection is completed, the signal data needs to be processed by compressive sensing to reduce storage and transmission costs.

[0092] In this embodiment, the specific steps of compressive sensing processing include:

[0093] First, a sampling matrix is constructed where M is the dimension of the compressed signal, N is the dimension of the original signal, and M << N. The sampling matrix Φ can be constructed using a Gaussian random matrix or a sparse random matrix.

[0094] Then, the original signal is reduced in dimension, and the dimension reduction formula is:

[0095] y = Φx

[0096] where, is the compressed signal.

[0097] In some embodiments, to decode the original signal, sparse representation decoding of the signal can be performed using a sparse reconstruction method. Specifically, the sparse coefficients of the signal are recovered by solving the following optimization problem:

[0098]

[0099] where Ψ is a sparse basis matrix, a is the sparse representation of the signal, and ∈ is an error tolerance.

[0100] After completing the compressive sensing process, the signal needs to be preprocessed to further improve the signal quality.

[0101] Generally, preprocessing includes two parts: noise reduction and normalization.

[0102] In this embodiment, signal noise reduction can be achieved through an adaptive filter, and the parameter dynamic adjustment formula is:

[0103]

[0104] where w n is the filter weight vector, with a dimension of d, where d is the signal feature dimension; μ is the learning rate, with a value range of (0, 1]; d n is the expected signal, and y n is the target output value at the current time step; x is the input signal vector at the current time step, with a dimension of d.

[0105] Normalization processing is used to standardize the amplitude range of the signal. In one possible implementation, the normalization formula is:

[0106]

[0107] where x i is the i-th signal sample; N is the number of signal samples; μ is the mean of the signal, which measures the central tendency of the signal; σ is the standard deviation of the signal, which measures the dispersion of the signal, and the calculation formula is x′ i is the normalized signal sample, whose value is limited to the range [0, 1].

[0108] Specifically, for signal data with large amplitude fluctuations, normalization can effectively improve feature extraction and training effect of subsequent deep learning models.

[0109] As an option, for signals with more low-frequency components (such as ECG), a filter can be further combined to eliminate power frequency interference and low-frequency drift, thereby improving signal quality.

[0110] Through the above steps, the signal data is collected, compressed and sensed, and preprocessed to obtain high-quality signal data suitable for subsequent feature extraction and analysis. The specific implementation of the above steps can be appropriately adjusted according to the signal source and application scenario, for example, an anti-aliasing filter can be added for high-frequency signals to prevent spectral aliasing.

[0111] S2, extracting time-frequency features of the signal, including performing short-time Fourier transform or wavelet transform on the signal data

[0112] First, the goal of this step is to extract time-frequency features from the signal data to provide input basis for subsequent deep learning model feature analysis and processing. The extraction of time-frequency features mainly relies on the joint representation of signals in time and frequency domains, which can effectively capture the local time-frequency dynamic characteristics of signals. In this invention, short-time Fourier transform (STFT) and wavelet transform (DWT) are used to extract time-frequency features according to the characteristics of different types of signals.

[0113] Generally, STFT is suitable for time-frequency characteristic analysis of stationary signals or approximately stationary signals, while DWT is more suitable for non-stationary signals and multi-scale analysis scenarios. As an option, the appropriate method can be selected according to the frequency range and time variation characteristics of the signal. For example, for audio signals, STFT is often used to analyze their spectral characteristics; while for non-stationary biological signals (such as electrocardiogram), DWT is more suitable for capturing the dynamic changes of low and high frequency components.

[0114] In this embodiment, the specific implementation of short-time Fourier transform includes the following contents:

[0115] In one possible implementation, after the input signal is subjected to STFT, a time-frequency feature matrix X STFT (t,f) is generated, which is defined as:

[0116]

[0117] where X STFT (t,f) is the time-frequency feature matrix of the signal, t represents the time window position, and f represents the frequency; x[n] is the value of the input signal at the nth sampling point; w[n-t] is a window function used to limit the signal locally, common window functions include rectangular window, Hamming window and Gaussian window; e -j2πfn is a complex exponential basis function used to calculate the amplitude and phase distribution of the signal in the frequency domain; n is the sampling point index of the signal,

[0118] In practical implementation, the window length L and the sliding step S of the STFT need to be adjusted according to the characteristics of the signal. Generally, the window length can be set to 10% to 20% of the sampling frequency of the signal to balance between time and frequency resolution. For example, for a 16 kHz sampled speech signal, the window length can be set to 512 points and the sliding step to 128 points.

[0119] As an option, the spectral resolution of the signal can be further optimized by adjusting the shape and length of the window function. For example, when the frequency components of the signal are low, a longer window can be used to improve the frequency resolution; while when the signal frequency changes rapidly, a shorter window can be used to enhance the time resolution.

[0120] In this embodiment, the specific implementation of the wavelet transform includes the following contents:

[0121] The core idea of the wavelet transform is to decompose the signal into different scale details and approximations, generating a multi-resolution time-frequency feature matrix X DWT (t, s), which is defined as:

[0122]

[0123] where X DWT (t, s) is the wavelet feature matrix of the signal, t represents the time position, and s represents the scale factor; x[n] is the value of the input signal at the nth sampling point; is the mother wavelet function, commonly used mother wavelets include Morlet wavelet, Haar wavelet and Daubechies wavelet, which are used to realize multi-scale decomposition of the signal; t is the time index, which controls the translation of the wavelet function; s is the scale factor, which controls the compression or expansion of the wavelet function, and is used to adjust the time and frequency resolution.

[0124] In one possible implementation, the time resolution and frequency resolution of the wavelet function can be adjusted by setting different scale factors s. Generally, when s is small, the wavelet function has high time resolution and low frequency resolution; when s is large, it has high frequency resolution and low time resolution.

[0125] As an option, the number of decomposition layers J of the wavelet transform can be set according to the frequency range and sampling rate of the signal. For example, for an electrocardiogram (ECG), 5 to 6 layers of decomposition can be selected to separate low-frequency drift and high-frequency noise.

[0126] Specifically, the time-frequency feature extraction in the present application not only can capture the global characteristics of the signal, but also can mine the local dynamic changes of the signal through joint analysis of the time-frequency domain, thereby providing more rich feature input for the subsequent deep learning model.

[0127] Through the above steps, the time-frequency feature extraction module of the signal can generate high-quality time-frequency feature data for subsequent feature analysis, with the advantages of strong applicability and high flexibility. In application scenarios, STFT or DWT can be flexibly selected for time-frequency feature extraction according to the specific type and characteristics of the signal, thereby improving the adaptability and robustness of the system.

[0128] S3, spatial feature extraction of time-frequency features using a deep learning model, and multi-modal feature fusion through an attention mechanism

[0129] This step aims to further extract spatial features from the extracted time-frequency features, and use multi-modal feature fusion technology to uniformly process data from different modalities, providing high-quality input features for subsequent signal optimization and decision-making. Spatial feature extraction mainly relies on convolutional neural networks (CNN) to model local and global patterns of input features, while multi-modal feature fusion assigns dynamic weights to different modal features through an attention mechanism, thereby achieving joint optimization of heterogeneous data.

[0130] Generally, spatial feature extraction is applicable to all input signals with time-frequency feature matrix form, such as audio spectrum, video frame data, and multi-dimensional spectrum of biological signals. Multi-modal feature fusion is applicable to data scenarios containing multiple modal inputs simultaneously. Through weight adjustment of each modal feature, the quality of the fusion result can be dynamically optimized.

[0131] In this embodiment, the specific implementation of spatial feature extraction includes the following:

[0132] In one possible implementation, a convolutional neural network is used to perform convolution operations on the time-frequency feature matrix. The mathematical formula for convolution is:

[0133]

[0134] where, is the convolution output feature, representing the activation value at position i,j after convolution operation; X is the input feature matrix, with dimensions HxW, where H and W are the height and width of the input matrix, respectively; is the convolution kernel, with size MxN, where M and N are the height and width of the convolution kernel, respectively; K m,n is the weight of the convolution kernel at position m,n; b is the bias term, which is a scalar value used to add an offset to the convolution result; σ is the activation function, commonly used activation functions include:

[0135] ReLU is σ(x) = max(0,x), ReLU: Rectified Linear Unit;

[0136] Sigmoid:

[0137] Tanh:

[0138] The result of the convolution operation is a feature matrix F with dimensions (H-M+1) x (W-N+1). As an option, the features can be reduced in dimension and filtered after the convolution operation by a pooling layer to retain key features and reduce computational complexity. Specifically, in the pooling operation, maximum pooling or average pooling can be selected. For example, the formula for maximum pooling is:

[0139] P i,j = max(F m,n ), m e [i, i+p), n e [j, j+q)

[0140] wherein:

[0141] P i,j is the pooling result; p and q are the height and width of the pooling window, respectively.

[0142] In some embodiments, the pooling layer can effectively reduce the dimension of the convolution features, for example, reducing a 32x32 feature matrix to 16x16, thereby reducing the complexity of subsequent calculations.

[0143] In this embodiment, the specific implementation of multi-modal feature fusion includes the following:

[0144] In one possible implementation, the feature matrices of different modalities are weighted and fused by an attention mechanism. The formula for multi-modal feature fusion is:

[0145]

[0146] wherein F fusion is the fused global feature matrix, representing the result of multi-modal feature fusion, with dimensions d f , wherein d f is the output feature dimension; M is the number of modalities, i.e., the number of different modalities participating in fusion; is the feature matrix of the i-th modality, with dimensions d i , wherein d i is the feature dimension of modality i; W i is the weight matrix of modality i, with dimensions d f x d i , used to measure the importance of modality i features in the fusion process; softmax is a normalization function that normalizes the fused feature values into a probability distribution, defined as:

[0147]

[0148] wherein zk z k is the k-th element in the feature matrix, z j Z is the set of all feature values; is an exponential function used to amplify the difference; i is the modal index, taking values in the range of 1, 2, …, M; W i The weight value of z is determined through the training of the deep learning model, which is used to dynamically allocate the importance of different modal features; the feature fusion result F fusion retains the multi-modal features correlation, which is suitable for the subsequent deep reinforcement learning optimization module.

[0149] As an option, the calculation of modal weights can be further optimized by introducing a self-attention mechanism. Specifically, the weight calculation formula of self-attention is:

[0150]

[0151] where q i and k j are the query vector and key vector of modal i and modal j, respectively; a i,j is the correlation weight of modal i and modal j.

[0152] In some embodiments, the fused feature matrix F can be further linearly transformed to unify the feature dimension, and the transformation formula is:

[0153] F final = W·F fusion +b

[0154] where W is the transformation matrix with dimension d f ×d f ; b is the bias vector.

[0155] Through the above steps, the spatial feature extraction and multi-modal feature fusion module of the present application can effectively process high-dimensional multi-modal signals, providing more comprehensive and accurate feature input for subsequent deep reinforcement learning, with high robustness and expansibility.

[0156] S4, based on the feature fusion result, using deep reinforcement learning to optimize signal processing parameters, including filter parameters and task allocation strategy

[0157] The goal of this step is to dynamically optimize signal processing parameters based on the results of multi-modal feature fusion using deep reinforcement learning, to adapt to complex and variable signal processing environments. Through reinforcement learning method, the system can dynamically adjust the filter parameters and task allocation strategy according to the current feature state, so as to achieve the optimal signal processing effect under different signal conditions.

[0158] Generally speaking, the deep reinforcement learning optimization process mainly includes state representation, action definition, immediate reward calculation, and Q-value function update. As an option, the Deep Q Network (DQN) can be introduced to further improve learning efficiency and use the target network to stabilize the optimization process.

[0159] In this embodiment, the specific implementation of deep reinforcement learning optimization includes the following:

[0160] In one possible implementation, the state s of reinforcement learning is t Represents the multimodal feature fusion result F fusion , whose dimension is d f , that is, the dimension of the fused feature. State s t Provides the dynamic characteristics of the current signal. Action a t Defined as adjustments to signal processing parameters, including but not limited to:

[0161] Filter parameter adjustment, such as the learning rate μ or filter order N of the adaptive filter;

[0162] Dynamic update of task allocation strategies, such as the allocation method P of computing resources;

[0163] Instant Rewards t is a direct measure of the effect of signal processing and is defined as:

[0164] r t =SNR out -SNR in -λ·C

[0165] Among them, SNR out SNR is the signal-to-noise ratio after signal processing; in is the signal-to-noise ratio before signal processing; λ is the weight factor of the computational cost, which is used to balance processing performance and computational overhead; C is the computational cost of the current signal processing, including processing time and resource consumption.

[0166] Specifically, the reward function dynamically evaluates the quality of the current action. For example, if the signal-to-noise ratio (SNR) is significantly improved after signal processing and the computational cost is low, the reward value is high; conversely, if the computational cost is too high or the SNR is not significantly improved, the reward value is low.

[0167] In this embodiment, the update formula of the Q value function is:

[0168]

[0169] Among them, Q(s t ,a t ): means in state s t Next take action at The value function of , which measures the long-term cumulative benefits of the action; γ: is the discount factor, ranging from [0, 1], which is used to balance the weight of immediate rewards and future benefits; s t+1 : is the current state s t In action a t The next state after a t+1 : is the next state s t+1 The set of possible actions.

[0170] In one possible implementation, a deep Q-network (DON) is used to approximate the Q-value function. DQN uses a neural network as a function approximator, whose input is the state s t , the output is the Q value of each possible action. The optimization of the network parameters Θ is done by gradient descent

[0171] Algorithm implementation, the loss function is defined as:

[0172]

[0173] Among them, y i =r i +γmax a′ Q(s i+1 ,a′;Θ - ) is the target Q value; Θ - is the parameter of the target network, which is updated periodically synchronously with the main network parameter Θ; N is the number of mini-batch samples.

[0174] As an option, the training efficiency can be improved by using the experience replay mechanism. Specifically, the four-tuple (s t ,a t ,r t ,s t+1 ) to the experience pool and randomly sample a batch of samples for gradient update during each training, thereby breaking the data correlation and improving the convergence of the model.

[0175] In this embodiment, a dual deep Q network (DoubleDQN) can also be combined to reduce the overestimation problem. The target value update formula of the dual DQN is:

[0176]

[0177] Through the above method, the present invention can dynamically optimize signal processing parameters through deep reinforcement learning under different signal conditions, making the system adaptable and robust in complex environments. This optimization process can be performed even with limited computing resources, and computational efficiency can be further improved through parameter sharing or model pruning.

[0178] S5, output the signal processing result, and dynamically adjust the signal processing model according to the feedback.

[0179] The purpose of this step is to output the optimized signal feature result, and dynamically adjust the signal processing model according to the output result to realize the closed-loop optimization of the system. Through the feedback mechanism, the application can adaptively adjust the key parameters in the signal processing process, so that it maintains good performance in a dynamic environment. Specifically, the implementation of this step includes the generation of signal results, the dynamic adjustment of model parameters, and the continuous iteration of model optimization.

[0180] Generally, the output of the signal processing result can include various forms, such as an enhanced version of the signal, a processed feature vector, or a decision result. As an option, different feedback strategies can be designed for different application scenarios, such as prioritizing delay in real-time communication or prioritizing accuracy in medical signal processing.

[0181] In this embodiment, the output of the signal processing result includes the following contents:

[0182] In one possible implementation, the signal feature matrix F fusion processed by multi-modal feature fusion and deep reinforcement learning optimization is further processed into a structured output result. Specifically, the output result can include a signal quality indicator (such as signal-to-noise ratio SNR), waveform data of an enhanced signal, or a high-dimensional feature vector after feature extraction.

[0183] For audio signals, in some embodiments, the processed signal can be remapped back to the time domain to generate an enhanced audio waveform. The mapping formula is:

[0184] x output [n]=ISTFT(X STFT )

[0185] where ISTFT is the inverse short-time Fourier transform operation, X STFT is the time-frequency feature matrix of the audio signal. For video signals, the output result can include a sequence of noise-removed frames or an enhanced image feature matrix. In some embodiments, the video frame rate can be further improved by inter-frame interpolation, such as filling in the missing information between adjacent frames using a specific interpolation algorithm, to generate a smoother output video.

[0186] In another possible implementation, for biological signals (such as ECG or EEG), the output result can include a denoised signal waveform, extracted feature values, or diagnostic assistance information. For example, for ECG signals, the features of P-wave, QRS complex, and T-wave can be extracted, and heart rate variability analysis results can be provided based on these features.

[0187] After the signal result is generated, it needs to be fed back to the model for dynamic adjustment. In this embodiment, the feedback optimization includes the following:

[0188] Specifically, the feedback mechanism dynamically adjusts the model parameters based on the processing results, such as optimizing the convolution kernel weights of feature extraction or retraining the policy network of deep reinforcement learning. In one possible implementation, the model update can be achieved through the following online learning formula

[0189]

[0190] where Θ t is the parameter of the current model; η is the learning rate; is the loss function, which is used to measure the gap between the model prediction value and the target value.

[0191] In some embodiments, the prompt function can be designed according to the application scenario. For example, in the signal denoising task, mean square error (MSE) can be used as the loss function, which is defined as:

[0192]

[0193] where x output,i is the output signal of the i-th sample; x target,i is the target signal of the i-th sample.

[0194] As an option, the model complexity can be optimized through pruning and quantization to reduce the consumption of computing resources. For example, for a convolutional neural network, redundant parameters can be reduced through the following pruning formula:

[0195]

[0196] where ||Θ||0 represents the number of non-zero elements of the model parameter; β is the regularization coefficient.

[0197] In addition, to further improve the adaptability of the model, an adaptive learning rate algorithm such as the Adam optimization algorithm can be introduced in some embodiments to dynamically adjust the learning rate to speed up convergence.

[0198] In practical applications, the feedback mechanism of this step can be combined with the real-time requirements of signal processing. For example, in communication signal processing, the sampling rate of compressed sensing can be adjusted through feedback to optimize data transmission efficiency; in medical signal processing, the denoising strategy of the signal can be optimized based on the diagnosis result to improve the accuracy of diagnosis.

[0199] Through the above steps, the present application realizes the generation and feedback loop of signal processing results, ensuring that the system can continuously optimize the signal processing parameters, thereby providing high-quality signal output results in different application scenarios.

[0200] The application also provides a computer device comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the signal processing method.

[0201] Although embodiments of the present application have been shown and described, it would be appreciated by those skilled in the art that changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A signal processing method, characterized in that: The following steps are involved: Collect signal data from multimodal signal sources, and perform compressed sensing processing and preprocessing on the collected signal data; Extracting the time-frequency characteristics of the signal, including performing short-time Fourier transform or wavelet transform on the signal data; Use deep learning models to extract spatial features from time-frequency features, and use attention mechanisms to fuse multimodal features; Based on the feature fusion results, deep reinforcement learning is used to optimize signal processing parameters, including filter parameters and task allocation strategies; Output signal processing results and dynamically adjust the signal processing model based on feedback.

2. A signal processing method according to claim 1, characterized in that: The compressed sensing process includes the following steps Use the sampling matrix to reduce the dimension of the signal data and generate a compressed signal: y=Φ Where x is the original signal with dimension N; Φ is the sampling matrix with dimension M×N, where M<<N; y is the compressed signal with dimension M. The sparse representation of the signal is restored using the sparse basis matrix: Among them, Ψ is the sparse basis matrix, which is used to transform the signal into a sparse representation; α is the sparse coefficient vector with dimension N; ∈ is the error tolerance, which is used to control the accuracy of signal recovery.

3. A signal processing method according to claim 1, characterized in that: The pretreatment includes: Noise reduction processing: Use an adaptive filter to reduce the noise of the signal data. The filter parameter update formula is: Among them, w n is the filter weight vector with dimension d, where d is the signal feature dimension; μ is the learning rate with a value range of (0, 1]; d n is the expected signal, is the target output value of the current time step; n Input signal vector for the current time step, dimension d; Normalization: Normalize the signal data to make its range normalized to [0, 1]. The normalization formula is: Among them, x i is the i-th signal sample; N is the number of signal samples; μ is the mean of the signal, which is used to measure the central tendency of the signal; σ is the standard deviation of the signal, which is used to measure the dispersion of the signal; x′ i is the normalized signal sample, whose value is limited to the range of [0, 1].

4. A signal processing method according to claim 1, characterized in that: The time-frequency feature extraction includes: Perform a short-time Fourier transform on the signal data: Among them, X STFT (t,f) is the time-frequency feature matrix of the signal, t represents the time window position, and f represents the frequency; x[n] is the value of the input signal at the nth sampling point; w[nt] is the window function, which is used to locally limit the signal. Common window functions include rectangular window, Hamming window and Gaussian window; e -j2πfn is a complex exponential basis function used to calculate the amplitude and phase distribution of the signal in the frequency domain; n is the sampling point index of the signal, Alternatively, perform a wavelet transform on the signal data: Among them, X DWT (t,s) is the wavelet characteristic matrix of the signal, t represents the time position, s represents the scale factor; x[n] is the value of the input signal at the nth sampling point; is the mother wavelet function. Common mother wavelets include Morlet wavelet, Haar wavelet and Daubechies wavelet, which are used to realize multi-scale decomposition of signals. t is the time index, which controls the translation of the wavelet function. s is the scale factor, which controls the compression or expansion of the wavelet function and is used to adjust the time and frequency resolution.

5. A signal processing method according to claim 1, characterized in that: The spatial feature extraction is achieved through a convolutional neural network, and its convolution operation is defined as: in, is the convolution output feature, which represents the activation value of position i, j after the convolution operation; X is the input feature matrix with dimension H×W, where H and W are the height and width of the input matrix respectively; is the convolution kernel, the size is M×N, where M and N are the height and width of the convolution kernel respectively; K m,n is the weight of the convolution kernel at position m,n; b is the bias term, which is a scalar value used to add an offset to the convolution result; σ is the activation function. Common activation functions include: ReLU is σ(x)=max(0,x), ReLU: rectified linear unit; Sigmoid: Fishy: In addition, i and j are the row and column indices of the feature matrix after the convolution operation, respectively. The dimension of the output feature matrix is ​​(H-M+1)×(W-N+1), which is determined by the input matrix dimension and the convolution kernel size. m and n are the row and column indices within the convolution kernel, respectively, used for pixel-by-pixel convolution calculations.

6. A signal processing method according to claim 1, characterized in that: The multimodal feature fusion is achieved through the attention mechanism, and the fusion formula is: Among them, F fusion is the fused global feature matrix, which represents the result of multimodal feature fusion, with a dimension of d f , where d f is the output feature dimension; M is the number of modalities, that is, the number of different modalities involved in fusion; is the characteristic matrix of the i-th mode, with dimension d i , where d i is the characteristic dimension of mode i; W i is the weight matrix of mode i, with dimension d f ×d i , used to measure the importance of modal i features in the fusion process; softmax is a normalization function, which uses the eigenvalues ​​after dry paddle fusion to normalize the probability distribution, which is defined as: Among them, z k is the kth element in the feature matrix, z j is the set of all eigenvalues; is an exponential function used to amplify differences; i is the modal index, ranging from 1, 2, …, M; W i The weight value is determined by the training of the deep learning model and is used to dynamically assign the importance of different modal features; the feature fusion result F fusion Preserves multimodal features Empty correlation, suitable for subsequent deep reinforcement learning optimization modules.

7. A signal processing method according to claim 1, characterized in that: The deep reinforcement learning optimization signal processing parameters include: Define the current state s t 、Action a t and instant rewards t : s t : Indicates the characteristic state of the current signal, including the time-frequency characteristics of the signal and the fusion results of multimodal features, with a dimension of d s , where d s is the dimension of the state vector; a i : Indicates the current state s t The actions taken under , including filter parameter adjustment and task allocation strategy update, dimension is d a , where d a is the dimension of the action vector; r t : is an immediate reward, defined as the improvement in signal processing effect: r t =SNR out -SNR in -λ·C Among them, SNR out is the signal-to-noise ratio after signal processing; SNR in is the signal-to-noise ratio before signal processing; λ is the weight factor of the computational cost, which is used to balance processing performance and computational overhead; C is the computational cost of the current signal processing; Update the Q-value function of the deep Q network: Among them, Q(s t ,a t ): means in state s t Next take action a t The value function of , which measures the long-term cumulative benefits of the action; γ: is the discount factor, ranging from [0, 1], which is used to balance the weight of immediate rewards and future benefits; s t+1 : is the current state s t In action a t The next state after a t+1 : is the next state s t+1 The set of possible actions; Optimize the parameters of the deep Q network using the gradient descent algorithm: Among them, Θ: is the parameter of the deep Q network; η: is the learning rate, which controls the step size of parameter update; Is the loss function, used to minimize the error between the actual Q value and the target O value, defined as: Where, N: is the number of samples; Q target (s i ,a i )=r i +γmax a′ Q(s i+1 ,a′), is the target Q value.

8. A signal processing method according to claim 1, characterized in that: The task allocation strategy is achieved by optimizing the following objectives: Where P = {P1, P2, ..., P N }: is the task allocation strategy, which represents the specific plan for allocating N tasks to different computing resources; P i : is the allocation scheme for the i-th task, and its value range is the set of available computing resources; N: is the total number of tasks; C i (P i ):For the i-th task assigned to resource P i The calculation cost after , is calculated as: Among them, W i is the computational load of the i-th task, in units of operations; For resource P i The computing power, in units of operations / second; L(P): is the total scheduling delay assigned to the task, calculated as: Among them, T start,i is the startup time of task i on computing resources; T compute,i is the computation time of task i, defined as: T comm,i is the communication time of task i, which means the data transmission time required after the task is assigned to the target computing resource; λ is the weight factor of the delay cost, which is used to balance the optimization goal between computing cost and delay, and its value range is [0, 1].

9. A signal processing method according to claim 1, characterized in that: The signal processing model is optimized through pruning and quantization, and its optimization formula is: Among them, Θ: is the parameter set of the signal processing model, the dimension is d Θ , where d Θ is the total number of model parameters; is the loss function of the model, which is used to measure the input signal X and the model prediction output The error between the true label y is defined as: Where X={x1,x2,…,x N }: is the input signal sample set, containing N samples; The model prediction output for the i-th sample; y i : is the true label of the i-th sample; ||Θ||0: is the number of non-zero items of the model parameters, which is used to measure the complexity of the model; β: is the regularization weight factor, with a value range of [0,∞), which is used to balance the weight between the loss function and the model complexity.

10. A computer device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the signal processing method according to any one of claims 1 to 9 when executed by the processor.

Images