Fourier transform denoising method and system based on hybrid mask strategy

Abstract

The application belongs to the technical field of data processing, and specifically discloses a Fourier transform denoising method and system based on a hybrid mask strategy. The method comprises the following steps: Fourier transform is used on original time series data to obtain frequency domain components; square sum operation is performed on the real part and the imaginary part of the frequency domain components to obtain the amplitude of the frequency components; based on a Top-k strategy, the first k frequency components with the largest amplitude are selected from all the frequency components, and the remaining frequency components are discarded to obtain a Top-k mask vector; a learning strategy based on a neural network is set to dynamically adjust the amplitude of the frequency components to obtain a learning mask vector; the Top-k mask vector and the learning mask vector are fused to obtain a final mask vector; and Hadamard product operation is performed on the mask vector and the frequency domain components, and then inverse Fourier transform is performed to restore the denoised time series data. The technical scheme improves the generalization of the Fourier transform denoising method based on the hybrid mask strategy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and relates to a Fourier transform denoising method and system based on a hybrid mask strategy. BACKGROUND

[0002] Time series data denoising is an important step in data preprocessing, which involves removing or reducing noise from time series data to improve data quality and accuracy of subsequent analysis. Time series data is a sequence of data points ordered by time, which can be stock prices, temperature records, website visits, or any data that changes over time.

[0003] Noise is ubiquitous in time series data, which can be caused by various factors such as measurement errors, random fluctuations in data collection process, external interference, etc. These noises can mask the real patterns and trends in the data, leading to inaccurate data analysis results. Therefore, denoising is a key step in the time series data analysis process, which directly affects the quality and reliability of data analysis results.

[0004] Existing time series data denoising methods can be roughly divided into two categories: time domain analysis-based denoising methods and frequency domain analysis-based denoising methods. Time domain analysis-based denoising methods refer to directly analyzing the time series of time series data, such as using moving average filters, median filters for denoising, etc. Frequency domain analysis-based denoising methods refer to converting signals from spatial domain to frequency domain and analyzing frequency spectrum in the transformed domain, such as using wavelet transform, Fourier transform, etc. For example, the patent with publication number CN114036454A proposes a wavelet transform denoising method based on a soft threshold function.

[0005] Currently, one of the main difficulties in using Fourier transform denoising methods is how to adapt to the characteristics of different data sets to achieve stronger generalization ability. Since the noise characteristics, signal frequency distribution, and the relationship between signal and noise of different data sets may differ significantly, how to make the noise algorithm flexibly identify and handle these changes is a technical problem that needs to be solved. SUMMARY

[0006] The purpose of the present application is to provide a Fourier transform denoising method and system based on a hybrid mask strategy to improve the generalization of the Fourier transform denoising method.

[0007] To achieve the above purpose, the basic scheme of the present application is as follows: a Fourier transform denoising method based on a hybrid mask strategy, comprising the following steps:

[0008] Obtain the original time series data;

[0009] The Fourier transform is used on the original time series data to obtain frequency domain components;

[0010] The real part and the imaginary part of the frequency domain components are subjected to square sum operation to obtain the amplitude of the frequency components;

[0011] Based on the Top-k strategy, the top k with the largest amplitude are selected from all the frequency components, and the remaining frequency components are discarded to obtain a Top-k mask vector;

[0012] A learning strategy based on a neural network is set to dynamically adjust the amplitude of the frequency components to obtain a learning mask vector;

[0013] The two strategies are used comprehensively to perform fusion operation on the Top-k mask vector and the learning mask vector to obtain a final mask vector;

[0014] After Hadamard product operation of the mask vector and the frequency domain components, inverse Fourier transform is performed to restore the denoised time series data.

[0015] The working principle and beneficial effects of the basic scheme are that the technical scheme is based on a hybrid mask strategy, uses Top-k strategy and learning strategy to obtain corresponding mask vectors, combines the two mask vectors through logical or operation to form a final mask vector that comprehensively considers conservative and flexible methods, and more comprehensively selects frequency domain components that are crucial to signal denoising or feature extraction. This can effectively remove noise in time series data to obtain clearer and more accurate signals and improve the generalization of the Fourier transform denoising method.

[0016] Further, when it is required to retain as much important information as possible and reduce information loss, the fusion operation adopts or operation;

[0017] When it is required to retain only the most important information recognized by multiple strategies, the fusion operation adopts and operation.

[0018] Different fusion operations can be selected according to different needs, which is more practical.

[0019] Further, the method for obtaining the learning mask vector by using a discrete gating strategy as the learning strategy and dynamically adjusting the amplitude of the frequency components is as follows:

[0020]

[0021] Gate = Dropout(Sigmoid(W o Amp(X)))

[0022] Where Gate is the vector obtained after dynamically adjusting the amplitude of the frequency component, which is calculated by a learnable weight parameter W and amplitude Amp(X) through Hadamard product; W is a learnable parameter responsible for dynamically adjusting the amplitude of the component during the training process, and the parameter a converts the output into a Boolean value; i represents the i-th frequency domain component; T represents the length of the time series; Sigmoid is an activation function responsible for mapping the input value to between 0 and 1; Dropout is a dropout operation, which is a commonly used regularization technique in deep learning, used to prevent overfitting; and is a Hadamard product operation.

[0023] The discrete gating strategy is suitable for time series signals, ensuring that components with small but important amplitudes are selected, while components with large but unimportant amplitudes, which may be noise, are discarded.

[0024] Further, a fusion operation is performed on the Top-k mask vector and the learned mask vector to obtain the final mask vector Mask, which is:

[0025] Mask = Mask top-k ∨Mask learn

[0026] Where Mask top-k represents the mask vector obtained through the top-k strategy, i.e., the Top-k mask vector; and Mask learn represents the mask vector obtained through the learning strategy, i.e., the learned mask vector.

[0027] The mask vectors obtained by the two strategies are combined through logical OR operation to form a final mask vector that comprehensively considers the conservative and flexible methods, so as to more comprehensively select the frequency domain components that are crucial for signal denoising or feature extraction.

[0028] Further, after performing Hadamard product operation on the mask vector and the frequency domain component, and then through inverse Fourier transform IFFT, the denoised time series data X d is restored, which is:

[0029] X d = IFFT(FFT(X) ⊙ Mask)

[0030] Where Mask top-k represents the mask vector obtained through the top-k strategy, and Mask learn represents the mask vector obtained through the learning strategy.

[0031] In this way, the noise in the time series data can be effectively removed, and a clearer and more accurate signal can be obtained.

[0032] The application also provides a Fourier transform denoising system based on a hybrid mask strategy, comprising a data acquisition module and a processing module, the data acquisition module is used to acquire original time series data, and the output end of the data acquisition module is connected with the input end of the processing module.

[0033] The processing module executes the method of the application to perform time series data denoising.

[0034] The system utilizes the data acquisition module and the processing module, and improves the generalization of the Fourier transform denoising method based on the hybrid mask strategy. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a flowchart of the Fourier transform denoising method based on the hybrid mask strategy of the application. DETAILED DESCRIPTION

[0036] The embodiments of the application are described in detail below, and examples of the embodiments are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are only used to explain the application, and cannot be understood as a limitation of the application.

[0037] In the description of the application, it should be understood that the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the application and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the application.

[0038] In the description of the application, unless otherwise specified and limited, it should be noted that the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be mechanical connection or electrical connection, or the communication between two elements, or direct connection, or indirect connection through an intermediate medium, and the specific meaning of the above terms can be understood by those skilled in the art according to the specific circumstances.

[0039] The application discloses a Fourier transform denoising method based on a hybrid mask strategy, as shown in Figure 1 The method comprises the following steps:

[0040] Obtaining original time series data;

[0041] Using Fourier transform (FFT) on the original time series data to obtain frequency domain components, denoted as FREQ;

[0042] Square and operate on the real part and the imaginary part of the frequency domain component to obtain the amplitude of the frequency component;

[0043] Based on the Top-k strategy, the top k with the largest amplitude are selected from all the frequency components, and the remaining frequency components are discarded to obtain a Top-k mask vector, so as to eliminate unimportant noise components;

[0044] A learning strategy based on a neural network (such as a multilayer perceptron, an attention mechanism, etc.) is set, and the amplitudes of the frequency components are dynamically adjusted to obtain a learning mask vector;

[0045] The two strategies are comprehensively used, and a fusion operation is performed on the Top-k mask vector and the learning mask vector to obtain a final mask vector; wherein the fusion operation can adopt or operation or and operation. The or operation can retain the components selected by the two strategies, and as much important information as possible is retained to reduce information loss; the and operation only retains the most important information that is commonly recognized by multiple strategies. After the mask vector is subjected to Hadamard product operation with the frequency domain component, the denoised time series data is restored through inverse Fourier transform.

[0046] In a preferred scheme of the present application, when it is required to retain as much important information as possible and reduce information loss, the fusion operation adopts or operation;

[0047] When it is required to only retain the most important information that is commonly recognized by multiple strategies, the fusion operation adopts and operation.

[0048] In a preferred scheme of the present application, a discrete gating strategy suitable for time series signals is used as the learning strategy, and the method for dynamically adjusting the amplitudes of the frequency components to obtain the learning mask vector is as follows:

[0049]

[0050] Gate=Dropout(Sigmoid(E⊙Amp(X)))

[0051] Wherein, Gate is a vector obtained after dynamically adjusting the amplitudes of the frequency components, and is obtained by Hadamard product calculation through a learnable weight parameter W and the amplitude Amp(X); W is a learnable parameter responsible for dynamically adjusting the amplitudes of the components during the training process, and the parameter a converts the output into a Boolean value. i represents the i-th frequency domain component; T represents the length of the time series; Sigmoid is an activation function responsible for mapping the input value to between 0 and 1; Dropout is a dropout operation, which is a commonly used regularization technique in deep learning, used to prevent overfitting; and is a Hadamard product operation. This ensures that components with small but important amplitudes are selected, while components with large but unimportant amplitudes are discarded, which may be noise.

[0052] In a preferred scheme of the present application, the logical OR operation is performed on the Top-k mask vector and the learning mask vector to obtain the final mask vector Mask, which is:

[0053] Mask=Mask top-k ∨Mask learn

[0054] wherein Mask top-k represents the mask vector obtained by the top-k strategy, i.e., the Top-k mask vector; Mask learn represents the mask vector obtained by the learning strategy, i.e., the learning mask vector. Mask is a binary vector containing only 0 and 1, wherein 0 represents discarding the corresponding component, and 1 represents selecting the corresponding component. Considering that the important information selected by the combination of the two strategies helps to improve the performance of the prediction algorithm, the logical OR operation is specifically adopted in this embodiment.

[0055] The top-k strategy is a conservative method that ensures the retention of important components through an empirical threshold in the denoising process. The learning strategy is a flexible method that can select components with a smaller amplitude threshold but equal importance according to the characteristics of different data sets.

[0056] The mask vectors obtained by the two strategies are combined through the logical OR operation to form a final mask vector that comprehensively considers the conservative and flexible methods, so as to more comprehensively select the frequency domain components that are crucial for signal denoising or feature extraction.

[0057] In a preferred scheme of the present application, after the Hadamard product operation is performed on the mask vector and the frequency domain component, the inverse Fourier transform IFFT is performed to restore the denoised time series data X d , which is:

[0058] X d =IFFT(FFT(X)⊙Mask)

[0059] wherein Mask top-k represents the mask vector obtained by the top-k strategy, and Mask learn represents the mask vector obtained by the learning strategy.

[0060] For example, the public data sets Solar-Energy and Traffic are adopted. Solar-Energy contains solar energy data collected from the Renewable Energy Laboratory. In the experiment, each data set is divided into a training set, a validation set, and a test set in a ratio of 6:2:2.

[0061] For evaluation indicators, the relative square root error (RSE), root mean square error (RMSE) and correlation coefficient (CORR) are used as evaluation indicators for the prediction performance of the method of the present application in the time series short-term prediction task. For RSE and RMSE, the smaller the value of the indicator, the better the prediction performance. For CORR, the larger the value of the indicator, the better the prediction performance. This paper compares the following four methods:

[0062] 1. GRU: directly using the GRU model for prediction;

[0063] 2. GRU+DFTG-G: only using the denoising method based on the Top-k strategy to denoise the time series data, and then using GRU for prediction;

[0064] 3. GRU+DFTG-T: only using the denoising method based on the learning strategy to denoise the time series data, and then using GRU for prediction;

[0065] 4. GRU+DFTG: comprehensively considering the denoising method of Top-k strategy and learning strategy to denoise the time series data, and then using GRU for prediction.

[0066] As shown in Table 1, the results show that the denoising method considering the Top-k strategy and the learning strategy can make all three indicators optimal in the time series short-term prediction task. The experimental results show the effectiveness of the time series data denoising method based on mixed frequency domain enhancement proposed by the present application.

[0067] Table 1 Experimental results

[0068]

[0069] The present application also provides a Fourier transform denoising system based on a mixed mask strategy, comprising a data acquisition module and a processing module, the data acquisition module being used to acquire original time series data, and the output end of the data acquisition module being electrically connected with the input end of the processing module.

[0070] The processing module executes the method of the present application to denoise the time series data. The system uses the data acquisition module and the processing module, and improves the generalization of the Fourier transform denoising method based on the mixed mask strategy.

[0071] In the description of the specification, reference to "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that a particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the application. The appearances of the above expressions in various places in the specification are not necessarily referring to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0072] Although embodiments of the application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions, and alterations can be made hereto without departing from the spirit and scope of the application, which is defined by the following claims and their equivalents.

Claims

1. A Fourier transform denoising method based on a hybrid mask strategy, characterized in that: The steps include: Get the original time series data; Apply Fourier transform to the original time series data to obtain the frequency domain components; Perform square sum operation on the real and imaginary parts of the frequency domain component to obtain the amplitude of the frequency component; Based on the Top-k strategy, the top k frequency components with the largest amplitude are selected from all frequency components, and the remaining frequency components are discarded to obtain the Top-k mask vector; Set up a neural network-based learning strategy to dynamically adjust the amplitude of the frequency component to obtain a learning mask vector; The Top-k mask vector and the learned mask vector are fused to obtain the final mask vector; After performing the Hadamard product operation on the mask vector and the frequency domain component, the denoised time series data is restored through inverse Fourier transform.

2. The Fourier transform denoising method based on a hybrid mask strategy according to claim 1, wherein: When it is required to retain as much important information as possible and reduce information loss, the fusion operation adopts the OR operation; When it is required to retain only the most important information that is recognized by multiple strategies, the fusion operation uses the AND operation.

3. The Fourier transform denoising method based on a hybrid mask strategy according to claim 1, wherein: Using discrete gating strategy as the learning strategy, the amplitude of the frequency component is dynamically adjusted to obtain the learning mask vector: Gate=Dropout(Sigmoid(W⊙Amp(X))) Among them, Gate is the vector obtained after dynamically adjusting the amplitude of the frequency component, which is obtained by calculating the Hadamard product through a learnable weight parameter W and the amplitude Amp(X); W is a learnable parameter responsible for dynamically adjusting the amplitude of the component during training, and the parameter α converts the output into a Boolean value; i represents the i-th frequency domain component; T represents the length of the time series; Sigmoid is the activation function responsible for mapping the input value to between 0 and 1; Dropout is a discarding operation, which is a commonly used regularization technique in deep learning to prevent overfitting; ⊙ is the Hadamard product operation.

4. The Fourier transform denoising method based on a hybrid mask strategy according to claim 1, wherein: The Top-k mask vector and the learned mask vector are fused to obtain the final mask vector Mask, which is: Mask=Mask top-k ∨Mask learn Among them, Mask top-k Represents the mask vector obtained by the top-k strategy, that is, the Top-k mask vector; Mask learn Represents the mask vector obtained by the learning strategy, that is, the learning mask vector.

5. The Fourier transform denoising method based on a hybrid mask strategy according to claim 1, wherein: After performing Hadamard product operation on the mask vector and the frequency domain component, the denoised time series data X is restored by inverse Fourier transform IFFT. d ,for: X d =IFFT(FFT(X)⊙Mask) Among them, Mask top-k Represents the mask vector obtained by the top-k strategy, Mask learn represents the mask vector obtained by learning the strategy.

6. A Fourier transform denoising system based on a hybrid masking strategy, characterized in that: It includes a data acquisition module and a processing module, wherein the data acquisition module is used to obtain original time series data, and the output end of the data acquisition module is connected to the input end of the processing module; The processing module executes the method according to any one of claims 1 to 5 to perform time series data denoising.

Images