Data noise reduction system based on deep neural network

Abstract

The invention discloses a data noise reduction system based on a deep neural network, and relates to the technical field of data noise reduction, and the method comprises the steps: carrying out the noise reduction of a plurality of target objects in different types of affiliation objects, obtaining an evaluation value of a noise reduction model for the processing effect of the plurality of target objects in the affiliation objects, and carrying out the noise reduction of the plurality of target objects; according to the discrete degree of the evaluation value, determining the evaluation reduction range of the noise reduction model on the affiliation object processing effect, and obtaining all evaluation reduction ranges of any affiliation object on the affiliation object processing by means of a plurality of noise reduction models; a primary selection model is determined according to the upper limit value and the lower limit value of the evaluation reduction range, then a correlation model is determined according to the coincidence degree of the evaluation reduction ranges of other noise reduction models and the evaluation reduction range of the primary selection model, and primary selection models and correlation models of all affiliation objects are obtained; according to the method, different models can be adopted to intelligently process different types of noise data, meanwhile, the processing results of the selected models can be fused, and a better processing effect can be obtained.

Description

Technical Field

[0001] This invention belongs to the field of noise reduction technology, specifically a data noise reduction system based on deep neural networks. Background Technology

[0002] In today's era of big data and artificial intelligence, high-quality data is the core engine driving scientific discovery, industrial intelligence, and precise decision-making. However, collected data is inevitably subject to various forms of noise contamination, such as sensor errors, transmission interference, and environmental disturbances. This noise severely distorts the true structure and inherent information of the data, leading to a significant decline in the performance of subsequent analyses (such as pattern recognition, predictive modeling, and feature extraction), and even resulting in erroneous conclusions. Therefore, data denoising, as a crucial step in data preprocessing, directly determines the reliability and value of data analysis.

[0003] For example, Chinese patent CN117134966A discloses a data denoising and classification method, apparatus, and electronic device, relating to the technical field of information security. The method includes: acquiring a set of security events to be processed; performing data filtering on the set of security events to be processed through a data denoising unit to determine a target set of security events, wherein the target set of security events is a set of real alarm security events; and performing alarm classification processing on the target set of security events through a data classification unit to determine the classification label of each target security event. By using the data denoising unit and the data classification unit, deep denoising and alarm classification are performed on security events, thereby significantly improving the efficiency of identifying and analyzing real alarms.

[0004] However, with the increasing richness of denoising models and the extreme diversity of noise scenarios, a key challenge has become increasingly prominent: how to intelligently select the most suitable and efficient model from a large number of pre-trained denoising models, each with its own strengths, for a given set of noisy data (whose noise characteristics may be unknown or complex and variable). This is a problem. Based on this, a data denoising system is provided for different denoising models and different types of noise data. Summary of the Invention

[0005] This invention aims to solve at least one of the technical problems existing in the prior art;

[0006] Therefore, this invention proposes a data denoising system based on deep neural networks, comprising:

[0007] The model library is used to perform noise reduction on several target objects representing different types of belonging objects by using several trained noise reduction models. The target objects are the noisy data.

[0008] The absolute values ​​of the differences in the relevant features of the noise data in the target object before and after are obtained. After being assigned different weights, they are added together to obtain the evaluation value representing the processing effect of the denoising model on several target objects in the target object. The reduction range of the denoising model on the target object is determined according to the dispersion of the evaluation value, so as to obtain the total reduction range of any target object by using several denoising models.

[0009] The initial model is determined based on the upper and lower limits of the noise reduction range. Then, the associated model is determined based on the overlap between the noise reduction range of the other noise reduction models and the noise reduction range of the initial model, thus obtaining the initial model and associated model for all the objects.

[0010] The model selection unit is used to automatically match the corresponding preliminary model and associated model from the model library to process the acquired real-time target object based on the object to which the real-time target object belongs.

[0011] Furthermore, it also includes a data acquisition unit for acquiring real-time target objects;

[0012] The noise feature extraction unit is used to extract the type of noise from the real-time target object and determine the object to which it belongs.

[0013] Furthermore, after the model selection unit selects the initial model and the associated model for processing the real-time target object, it processes the target object sequentially using the initial model and the associated model, and then selects the processed result with the lowest noise intensity as the output.

[0014] Furthermore, after the model selection unit selects the initial model and the associated model for processing the real-time target object, it processes the target object sequentially using the initial model and the associated model, and retains the common parts of all processed real-time target objects, marking them as the output results.

[0015] Furthermore, the relevant features of the target object are extracted by the noise feature extraction unit, and the relevant features include the time domain, frequency domain, and spatial domain features of the target object noise.

[0016] Furthermore, the method for determining the range of downgrade based on the dispersion of evaluation values ​​is as follows:

[0017] Several evaluation values ​​are obtained and labeled as Gi, i=1, ..., n. The mean value P of Gi is automatically obtained. Gi is filtered several times using the filtering condition |Gi-P|≤X1 to obtain all Gi that meet the filtering condition. Here, X1 is a preset value.

[0018] If the proportion of the filtered Gi to the total number of Gi does not exceed the preset value B1, the range from the minimum to the maximum value of the remaining Gi will be automatically marked as the downgrade range.

[0019] Furthermore, if the proportion of the filtered Gi values ​​to the total number of Gi values ​​exceeds the preset value B1, and at the same time the proportion of the number of filtered Gi values ​​exceeding the P value to the total number of Gi values ​​exceeds the set proportion B2, the median value between the maximum value of the remaining Gi and the maximum value of the filtered Gi is marked as the upper limit of the devaluation range, and the lower limit of the devaluation range is the minimum value of the remaining Gi that meets the screening conditions.

[0020] Furthermore, if the proportion of the filtered Gi values ​​to the total number of Gi values ​​exceeds the preset value B1, and at the same time the proportion of the number of filtered Gi values ​​exceeding the P value to the total number of Gi values ​​does not exceed the set proportion B2, the range from the minimum value of the remaining Gi to the maximum value of the remaining Gi is marked as the downgrade range after multiplying the minimum value of the remaining Gi by 0.9.

[0021] Furthermore, the method for determining the initial model is as follows:

[0022] The upper and lower limits of the noise reduction range of all noise reduction models are assigned a weight and then summed to obtain the core evaluation value. The model with the largest core evaluation value is marked as the initial selection model; the upper limit value has a greater weight than the lower limit value.

[0023] Furthermore, the method for determining the association model based on the initial model is as follows:

[0024] Obtain the evaluation reduction range of the initial selection model, then select the evaluation reduction range of any other denoising model, and obtain the numerical length of their overlap. Next, subtract the minimum value from the maximum value of the initial selection model's evaluation reduction range and mark this as the initial selection length. Divide the numerical length by the initial selection length to obtain the numerical value, which is marked as the overlap degree. Denoising models with an overlap degree exceeding B3 with the initial selection model are marked as associated models.

[0025] Compared with the prior art, the beneficial effects of the present invention are:

[0026] This application uses several trained denoising models to denoise several target objects representing different types of belonging objects, obtains the evaluation value of the denoising model's processing effect on several target objects in the belonging objects, determines the denoising range of the denoising model's processing effect on the belonging objects based on the dispersion of the evaluation value, and thus obtains the total denoising range of any belonging object processed by several denoising models.

[0027] The initial model is determined based on the upper and lower limits of the noise reduction range. Then, the associated model is determined based on the overlap between the noise reduction range of the other noise reduction models and the noise reduction range of the initial model, thus obtaining the initial model and associated model for all the objects.

[0028] This application enables the use of different models for intelligent processing of different types of noise data, and can also integrate the processing results of various selected models to achieve better processing results. This application is simple, effective, and easy to use. Attached Figure Description

[0029] Figure 1 This is a system block diagram of the present invention;

[0030] Figure 2 This is a flowchart illustrating the process of obtaining the evaluation value of the target object in this invention. Detailed Implementation

[0031] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Example 1:

[0033] Please see Figure 1 This application provides a data denoising system based on deep neural networks;

[0034] As an embodiment of the present invention, it includes:

[0035] The model library is used to perform noise reduction on several target objects representing different types of belonging objects by using several trained noise reduction models. The target objects are the noisy data.

[0036] The absolute values ​​of the differences in the relevant features of the noise data in the target object before and after are obtained. After being assigned different weights, they are added together to obtain the evaluation value representing the processing effect of the denoising model on several target objects in the target object. The reduction range of the denoising model on the target object is determined according to the dispersion of the evaluation value, so as to obtain the total reduction range of any target object by using several denoising models.

[0037] The initial model is determined based on the upper and lower limits of the noise reduction range. Then, the associated model is determined based on the overlap between the noise reduction range of the other noise reduction models and the noise reduction range of the initial model, thus obtaining the initial model and associated model for all the objects.

[0038] The model selection unit is used to automatically match the corresponding preliminary model and associated model from the model library to process the acquired real-time target object based on the object to which the real-time target object belongs.

[0039] Example 2:

[0040] Data denoising systems based on deep neural networks specifically include:

[0041] The data acquisition unit is used to acquire the target object, which is the data that needs to be denoised, that is, the data content containing noise. For ease of description, it is referred to as the target object throughout the text.

[0042] It also includes a noise feature extraction unit.

[0043] A lightweight CNN+Transformer hybrid network is used to extract relevant features of the target object. These relevant features are the temporal, frequency, and spatial features of the corresponding noise, including spectral energy distribution, pixel value variance, autocorrelation characteristics, etc.

[0044] The noise classification unit is used to identify the noise type of the corresponding target object. The noise types include Gaussian, salt and pepper, Poisson, compression artifact, and ambient background noise, and quantifies the noise intensity. Several noise types are obtained and they are marked as object belonging, and the corresponding noise intensity is marked as the belonging value. There is a one-to-one correspondence between the object belonging and the belonging value for the same target object.

[0045] The model selection unit is used to automatically match the initial selection model and associated model corresponding to the identified noise type in the real-time target object, i.e., the object to which it belongs, from the model library.

[0046] The real-time target object is processed sequentially using the initial selection model and the correlation model, and then the processed result with the lowest noise intensity is selected as the output.

[0047] The model library contains pre-selected models and associated models for each corresponding object;

[0048] Output unit, used to output results.

[0049] Example 3:

[0050] As a third embodiment of this application, this embodiment is implemented based on the second embodiment. The difference is that in this embodiment, after processing the real-time target object with the help of the initial selection model and the association model, several processed target objects are obtained. Then, only the common parts of all processed real-time target objects are retained and marked as the result output.

[0051] Example 4:

[0052] like Figure 2 As shown, this is the fourth embodiment of the present application. This embodiment is based on the second embodiment, but the difference is that the initial selection model and the associated model corresponding to each belonging object in the model library in this embodiment are obtained by correlation analysis. The specific method of correlation analysis is as follows:

[0053] Choose any noise reduction model;

[0054] Choose any object to belong to, and obtain any target object within that object's category;

[0055] Then, the selected trained denoising model is used to denoise the target object. After denoising, the relevant features of the target object are re-collected.

[0056] Obtain the absolute value of the difference between the values ​​of the relevant features before and after noise reduction, and mark it as a reduction.

[0057] All depreciation values ​​are assigned different weights and then summed to obtain a numerical value, which is then labeled as the evaluation value.

[0058] The same processing is applied to the remaining target objects to which this type of object belongs, and the evaluation value of the denoising model on the remaining target objects is obtained, resulting in several evaluation values ​​Gi, i=1,...,n, indicating that there are n target objects that have been processed by the same denoising model;

[0059] The mean of Gi is then automatically obtained and labeled as P. Gi is then filtered using the condition |Gi-P|≤X1, where X1 is a preset value.

[0060] Filter out the Gi that meets the criteria and delete the ones that do not.

[0061] Then, the remaining Gi are substituted into the filtering conditions again for filtering, and the Gi that does not meet the filtering conditions is repeatedly deleted until all Gi meet the filtering conditions.

[0062] Then, the number of Gi items deleted is automatically obtained, and the value obtained by dividing it by n is marked as the rejection ratio;

[0063] When the rejection ratio does not exceed the preset value B1, which is generally 20%, the range from the minimum to the maximum value of the remaining Gi is automatically marked as the reduction range of the target object to which the corresponding denoising model belongs.

[0064] If the rejection ratio exceeds B1, the rejected Gi values ​​are obtained, and the number of Gi values ​​that are rejected exceeding the P value is obtained and marked as the upper value. After dividing the upper value by n, it is marked as the rejection ratio. Here, exceeding the P value means exceeding the mean value of the Gi that was rejected.

[0065] If the screening ratio exceeds the set ratio B2, the median value of the maximum value of the remaining Gi and the maximum value of the screened Gi will be marked as the upper limit value.

[0066] Mark the range from the minimum to the maximum value in the remaining Gi as the range for demotion;

[0067] If the screening ratio does not exceed B2, then the range from the minimum value of the remaining Gi to the maximum value of the remaining Gi is marked as the range for de-evaluation.

[0068] The corresponding noise reduction model is used to determine the range of degraded ratings for the object's attribution.

[0069] Then the same process is applied to the remaining object classifications to obtain the range of noise reduction when the denoising model is applied to different object classifications.

[0070] Then the remaining noise reduction models are processed in the same way to obtain the noise reduction range of each noise reduction model for all objects.

[0071] In other words, each object belongs to a range of noise reduction models.

[0072] The data fitting method is as follows: The reduction range of different object classifications is processed using different noise reduction models.

[0073] Choose any object to belong to, obtain the reduction range of all denoising models for that object, obtain the reduction range of all denoising models, obtain the maximum and minimum values ​​in the reduction range, and mark them as the upper and lower reduction values ​​respectively.

[0074] For each noise reduction model, the upper and lower noise reduction values ​​for that object are obtained. The core score is calculated according to the formula: Core score = 0.66 × upper noise reduction value + 0.34 × lower noise reduction value.

[0075] Obtain all kernel evaluation values ​​and mark the denoising model with the highest kernel evaluation value as the initial selection model;

[0076] Then, the overlap between the degradation range of the remaining denoising models and the degradation range of the initial selection model is obtained. The overlap is obtained as follows:

[0077] Obtain the evaluation reduction range of the initial model, then select the evaluation reduction range of any noise reduction model, and obtain the numerical length of the overlap between the two. The data length is the maximum value minus the minimum value of the same part of the two. Then obtain the value from the maximum value minus the minimum value of the evaluation reduction range of the initial model, and mark it as the initial length. Divide the numerical length by the initial length to obtain the numerical mark as the degree of overlap.

[0078] Here is a specific example:

[0079] If the initial model's rating reduction range is C1 to C2, the selected denoising model's rating reduction range is C3 to C4;

[0080] If the overlapping portion of C1 to C2 and C3 to C4 is C5 to C6; here C1 <C2,C3<C4,C5<C6;

[0081] Therefore, the degree of overlap here is equal to: (C6-C5) / (C2-C1);

[0082] Obtain the overlap between each denoising model and the initial selection model, and mark those with an overlap exceeding B3 as related models;

[0083] The initial selection model and several association models corresponding to the belonging objects are obtained;

[0084] The same processing is applied to all attribution objects to obtain the initial selection model and association model of all attribution objects;

[0085] The noise reduction model mentioned here is obtained through pre-training. Of course, specialized network models can be used for different types of noise.

[0086] For image types, U-Net (medical noise), DnCNN (Gaussian), and GAN (complex artifacts) can be used.

[0087] For audio types, Conv-TasNet (speech separation) and DiffWave (diffusion model) can be used.

[0088] For time-series types, Informer (long sequence denoising) can be used.

[0089] The data in the above formula are all calculated by removing the dimensions and taking the numerical values. The formula is the closest to the real situation obtained by software simulation of a large amount of collected data. The preset parameters and preset thresholds in the formula are set by those skilled in the art according to the actual situation or obtained through simulation of a large amount of data.

[0090] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.

Claims

1. A data denoising system based on deep neural networks, characterized in that, include: The model library is used to perform noise reduction on several target objects representing different types of belonging objects by using several trained noise reduction models. The target objects are the noisy data. The absolute values ​​of the differences between the relevant features of the noise data in the target object before and after processing by the denoising model are obtained. After being assigned different weights, they are added together to obtain the evaluation value representing the processing effect of the denoising model on the target object. The reduction range of the denoising model on the target object is determined according to the dispersion of the evaluation value, so as to obtain the total reduction range of any target object processed by several denoising models. The initial model is determined based on the upper and lower limits of the noise reduction range. Then, the associated model is determined based on the overlap between the noise reduction range of the other noise reduction models and the noise reduction range of the initial model, thus obtaining the initial model and associated model for all the objects. The model selection unit is used to automatically match the corresponding preliminary model and associated model from the model library to process the acquired real-time target object based on the object to which the real-time target object belongs.

2. The data denoising system based on a deep neural network according to claim 1, characterized in that, Also includes: The data acquisition unit is used to acquire real-time target objects; The noise feature extraction unit is used to extract the type of noise from the real-time target object and determine the corresponding object to which it belongs.

3. The data denoising system based on a deep neural network according to claim 1, characterized in that, After selecting the initial model and associated model for processing the real-time target object through the model selection unit, the target object is processed sequentially using the initial model and associated model, and the processed result with the lowest noise intensity is selected as the output.

4. The data denoising system based on a deep neural network according to claim 1, characterized in that, After selecting the initial model and associated model for processing the real-time target object through the model selection unit, the target object is processed sequentially using the initial model and associated model. The common parts of all processed real-time target objects are retained and marked as the results output.

5. A data denoising system based on a deep neural network according to claim 1, characterized in that, The relevant features of the target object are extracted by the noise feature extraction unit. The relevant features include the time domain, frequency domain, and spatial domain features of the target object noise.

6. A data denoising system based on a deep neural network according to claim 1, characterized in that, The method for determining the range of downgrade based on the dispersion of evaluation values ​​is as follows: The obtained evaluation values ​​are labeled as Gi, i=1, ..., n. The mean value P of Gi is automatically obtained. Gi is filtered several times using the filtering condition |Gi-P|≤X1 to obtain all Gi that meet the filtering condition. X1 is a preset value. If the proportion of the filtered Gi to the total number of Gi does not exceed the preset value B1, the range from the minimum to the maximum value of the remaining Gi will be automatically marked as the downgrade range.

7. A data denoising system based on a deep neural network according to claim 6, characterized in that, If the proportion of the filtered Gi values ​​to the total number of Gi values ​​exceeds the preset value B1, and at the same time the proportion of the number of filtered Gi values ​​exceeding the P value to the total number of Gi values ​​exceeds the set proportion B2, the median value between the maximum value of the remaining Gi and the maximum value of the filtered Gi is marked as the upper limit of the de-evaluation range, and the lower limit of the de-evaluation range is the minimum value of the remaining Gi values ​​that meet the screening conditions.

8. A data denoising system based on a deep neural network according to claim 6, characterized in that, If the proportion of the filtered Gi values ​​to the total number of Gi values ​​exceeds the preset value B1, and at the same time the proportion of the number of filtered Gi values ​​exceeding the P value to the total number of Gi values ​​does not exceed the set proportion B2, the range from the minimum value of the remaining Gi to the maximum value of the remaining Gi is marked as the downgrade range.

9. A data denoising system based on a deep neural network according to claim 1, characterized in that, The method for determining the initial model is as follows The upper and lower limits of the noise reduction range of all noise reduction models are assigned a weight and then summed to obtain the core evaluation value. The model with the largest core evaluation value is marked as the initial selection model; the upper limit value has a greater weight than the lower limit value.

10. A data denoising system based on a deep neural network according to claim 1, characterized in that, The method for determining the association model based on the initial selected model is as follows: Obtain the degradation range of the initial model, then select the degradation range of any other denoising model, obtain the numerical length of the overlap between the two, then obtain the value of the maximum value minus the minimum value of the degradation range of the initial model, mark it as the initial length, divide the numerical length by the initial length, and mark the numerical value as the overlap degree; mark the denoising model with an overlap degree of more than B3 with the initial model as the associated model.

Images