A machine learning-based data denoising and reconstruction processing method

Abstract

This invention discloses a data denoising and reconstruction method based on machine learning, comprising the following steps: acquiring the original data to be processed and performing preprocessing; calculating the invertible latent representation vector through the forward encoding path of the invertible mapping network; performing structure-perturbation decomposition to generate a set of structural latent variables and a set of perturbation latent variables; generating a set of structural reconstruction latent variables and combining it with the set of perturbation latent variables for encoding; generating preliminary denoising and reconstruction results; performing difference to obtain a backtracking bias vector and generating an observation residual set; performing closed-loop correction on the reconstructed combined latent variable set; performing latent space error correction and performing error correction decoding update; generating potential function convergence judgment results and syndrome convergence judgment results to obtain the final denoising and reconstruction data results. This invention uses a combination of invertible latent representation and structural perturbation decomposition with a closed-loop error correction mechanism to achieve data denoising and reconstruction, possessing the advantages of backtracking self-checking, localization and repair, and high output reliability.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a data denoising and reconstruction method based on machine learning. Background Technology

[0002] Existing data denoising and reconstruction methods mostly employ filtering, denoising autoencoders, or generative reconstruction models to perform end-to-end regression on the input data. Some methods introduce latent representations for missing data completion and distortion recovery, and typically rely on fixed loss and a single inference to obtain the results.

[0003] In scenarios where the type and intensity of noise are unknown and accompanied by local missing or distorted noise, the above methods are difficult to stably distinguish between structural information and disturbance components, and are prone to details being erased, artifact reconstruction, or structural drift. They lack a backtracking and self-checking closed-loop correction mechanism and a localizable repair mechanism, making it difficult to locate the error position within the algorithm and perform error correction and updates, and making it difficult to determine and control the reliability of the output. Summary of the Invention

[0004] One objective of this invention is to propose a data denoising and reconstruction method based on machine learning. This invention employs a combination of reversible latent representation and structural perturbation decomposition with a closed-loop error correction mechanism to achieve data denoising and reconstruction, which has the advantages of being able to backtrack and self-check, locate and repair, and output with high reliability.

[0005] A data denoising and reconstruction method based on machine learning according to an embodiment of the present invention includes the following steps:

[0006] Obtain the raw data to be processed and perform preprocessing to generate standardized data;

[0007] Standardized data is input into the forward encoding path of the invertible mapping network to calculate the invertible latent representation vector and generate a set of invertible latent representations.

[0008] Perform structure-perturbation decomposition on the set of invertible latent representations to generate a set of structural latent variables and a set of perturbation latent variables;

[0009] The set of structural latent variables is reconstructed to generate a set of structural reconstruction latent variables, and then combined and encoded with the set of disturbance latent variables to generate a set of reconstructed combined latent variables.

[0010] The reconstructed set of latent variables is input into the reverse decoding path, and fragment splicing and boundary continuity correction are performed to generate preliminary denoising and reconstruction results.

[0011] The initial denoising and reconstruction results are input into the forward coding path to calculate the backtracking latent representation vector, which is then differiated with the reconstructed combined latent variable set to obtain the backtracking bias vector, generating the observation residual set.

[0012] Based on the set of observed residuals, a closed-loop correction is performed on the reconstructed combined latent variable set to generate a closed-loop corrected structural latent variable set.

[0013] Perform latent space error correction on the closed-loop modified structure latent variable set, perform error correction decoding update, and generate the error correction structure latent variable set;

[0014] The set of latent variables for error correction structure and the set of latent variables for perturbation are combined and encoded, input into the inverse decoding path, and the potential function convergence judgment result and the syndrome convergence judgment result are generated. The final denoising and reconstructed data result is generated based on the potential function convergence judgment result and the syndrome convergence judgment result.

[0015] Optionally, the preprocessing includes time alignment, amplitude normalization, missing segment marking, and outlier removal.

[0016] Optionally, the generation of the reversible latent representation set specifically includes:

[0017] Standardized data is divided into sets of standardized data segments based on data segment identifiers;

[0018] The standardized data fragment set is input into the forward encoding path of the invertible mapping network, and the invertible mapping is performed on the standardized data fragment set to generate an invertible latent representation vector that corresponds one-to-one with the identifier of each data fragment.

[0019] The reversible latent representation vectors are aggregated according to the data segment identifiers to form a set of reversible latent representations.

[0020] Optionally, the generation of the set of structural latent variables and the set of disturbance latent variables includes:

[0021] Read the set of invertible latent representations and group the invertible latent representation vectors according to the data fragment identifiers to generate a fragment-level invertible latent representation subset;

[0022] Perform a structure-perturbation decomposition on each invertible latent representation vector in the fragment-level invertible latent representation subset to generate a structural latent variable vector and a perturbation latent variable vector;

[0023] Perform structural consistency constraint on the structural latent variable vector to generate a constrained structural latent variable vector;

[0024] Perturbation isolation is performed on the perturbation latent variable vector, and energy proportion allocation and intensity normalization are performed on the perturbation latent variable vector according to the data segment identifier to generate a normalized perturbation latent variable vector;

[0025] The constrained structural latent variable vector and the normalized perturbation latent variable vector are organized according to data segment identifiers to form a set of structural latent variables and a set of perturbation latent variables.

[0026] Optionally, the generation of the reconstructed combination latent variable set includes:

[0027] Read the set of structural latent variables and group the structural latent variable vectors according to the data segment identifiers to generate a segment structural latent variable subset;

[0028] Reconstruction is performed on the subset of latent variables of the fragment structure. Component linear mapping is performed on the latent variable vector of the structure according to the data channel identifier to generate a structure mapping vector. Nonlinear transformation is performed on the structure mapping vector to generate a structure activation vector. Component aggregation is then performed on the structure activation vector to generate a structure reconstruction latent variable vector. The results are then summarized to obtain the set of latent variables for structure reconstruction.

[0029] Read the set of disturbance latent variables and pair them with the set of structure reconstruction latent variables according to the data segment identifier to generate a segment-level pairing set;

[0030] Perform combinatorial encoding on the fragment-level paired set to generate a reconstructed combinatorial latent variable vector, and summarize it according to the data fragment identifier to form a reconstructed combinatorial latent variable set;

[0031] The generation of the reconstructed combined latent variable vector includes: reading the structural reconstructed latent variable vector and the perturbation latent variable vector corresponding to the same data segment identifier; calculating the structural significance score of the structural reconstructed latent variable vector and the perturbation intensity score of the perturbation latent variable vector; generating the interleaving step size value and the interleaving start position index based on the structural significance score and the perturbation intensity score, and generating the structural component position set and the perturbation component position set accordingly; writing the components of the structural reconstructed latent variable vector into the corresponding positions of the reconstructed combined latent variable vector according to the structural component position set, and writing the components of the perturbation latent variable vector into the corresponding positions of the reconstructed combined latent variable vector according to the perturbation component position set, thus generating the reconstructed combined latent variable vector.

[0032] Optionally, the generation of the preliminary denoising and reconstruction results specifically includes:

[0033] Read the set of reconstructed combination latent variables and organize the vector of reconstructed combination latent variables into a subset of fragment reconstructed combination latent variables based on the data fragment identifier;

[0034] The fragment reconstruction combined latent variable subset is input into the inverse decoding path of the inverse mapping network, and inverse decoding is performed to generate fragment decoding data;

[0035] Perform fragment splicing on the decoded fragment data to form preliminary spliced ​​data;

[0036] Perform boundary continuity correction on the initial spliced ​​data, generate a correction transition window, and write back the correction transition window to obtain boundary continuity corrected data;

[0037] The boundary continuity correction data is output according to the data channel identifier to form the preliminary denoising and reconstruction results.

[0038] Optionally, the generation of the observation residual set specifically includes:

[0039] The preliminary denoising and reconstruction results are organized into a preliminary result fragment set according to the data fragment identifier;

[0040] The preliminary result fragment set is input into the forward encoding path of the invertible mapping network to generate backtracking latent representation vectors, which are then summarized to form a backtracking latent representation set.

[0041] Read the reconstructed set of combined latent variables and align it with the backtracked set of latent representations to obtain the fragment alignment set;

[0042] Perform component-wise differencing on each backtracking latent representation vector in the fragment alignment set and the corresponding reconstructed combined latent variable vector to generate a backtracking bias vector, and bind the backtracking bias vector with the data fragment identifier to form a backtracking bias set;

[0043] Aggregate the backtracking deviation set according to the data channel identifier. Calculate the sum of the absolute values ​​of the components of the backtracking deviation vector corresponding to the same data channel identifier to obtain the channel residual value. Then, summarize the channel residual values ​​according to the data segment identifier to form the observation residual set.

[0044] Optionally, the generation of the closed-loop modified structure latent variable set specifically includes:

[0045] Read the set of observation residuals and establish an observation residual index based on the data segment identifier and data channel identifier to form a segment-channel residual matrix;

[0046] Generate a set of combined latent variable correction vectors based on the segment channel residual matrix;

[0047] Read the reconstructed set of combined latent variables and align it with the set of modified vectors of combined latent variables according to the data segment identifier. Perform modification and update on each reconstructed set of combined latent variables to generate a closed-loop modified set of combined latent variables.

[0048] Structural component extraction is performed on the closed-loop modified combined latent variable set to determine the set of structural component locations, and the closed-loop modified structural latent variable vectors are extracted and summarized to form the closed-loop modified structural latent variable set.

[0049] Optionally, the generation of the latent variable set of the error correction structure specifically includes:

[0050] Read the set of latent variables of the closed-loop modified structure, and organize the vector of latent variables of the closed-loop modified structure according to the data segment identifier to generate a subset of latent variables of the closed-loop modified structure of the segment.

[0051] Perform check syndrome calculation on each closed-loop modified structure latent variable vector in the subset of fragment closed-loop modified structure latent variables to generate a check syndrome vector;

[0052] The sum of the absolute values ​​of the components of the verification syndrome vector is used to obtain the syndrome intensity value, and the syndrome intensity value is compared with the preset syndrome threshold to generate a syndrome abnormality score;

[0053] Based on the abnormal scores of the syndrome, error location is performed on the latent variable vector of the closed-loop modified structure to determine the set of error locations;

[0054] Error correction decoding updates are performed on the components of the closed-loop correction structure latent variable vector corresponding to the set of error locations to generate an error correction update vector, and additive updates are performed on the closed-loop correction structure latent variable vector to generate an error correction structure latent variable vector.

[0055] The error correction structure latent variable vector is bound to the data segment identifier and written into the error correction structure latent variable set.

[0056] Optionally, the generation of the final denoised and reconstructed data results specifically includes:

[0057] Read the latent variable set of error correction structure and the latent variable set of disturbance, and establish a one-to-one segment pairing relationship based on the data segment identifier to form a segment error correction pairing set;

[0058] Perform combined encoding on the fragment error correction pairing set to generate an error correction combined latent variable vector, and summarize it according to the data fragment identifier to form an error correction combined latent variable set;

[0059] The set of error correction combined latent variables is input into the inverse decoding path of the invertible mapping network to generate closed-loop denoising and reconstruction data, and the fragment splicing is performed according to the data time index to generate closed-loop denoising and reconstruction results.

[0060] Read the observation residual set corresponding to the current loop, sum the channel residual values ​​in the observation residual set according to the data segment identifier to obtain the segment potential function value, and form a potential function value set by combining the segment potential function values ​​corresponding to each data segment identifier;

[0061] The potential function value is compared with the preset potential function convergence threshold to generate a potential function convergence determination result.

[0062] The syndrome intensity value is obtained by calculating the sum of the absolute values ​​of the verification syndrome vector components, and the syndrome intensity value is compared with the preset syndrome threshold to generate the syndrome convergence determination result;

[0063] A joint judgment is performed on the potential function convergence judgment result and the syndrome convergence judgment result to generate an output judgment result. When the output judgment result is satisfied, the closed-loop denoising and reconstruction result is used as the final denoising and reconstruction data result. When the output judgment result is not satisfied, the closed-loop update process is triggered and the observation residual set, the error correction structure latent variable set, the error correction combined latent variable set, the closed-loop denoising and reconstruction result, the potential function convergence judgment result and the syndrome convergence judgment result are regenerated until the output judgment result is satisfied or the preset maximum number of loops is reached, and then the final denoising and reconstruction data result is output.

[0064] When the potential function value is not greater than the preset potential function convergence threshold, the potential function convergence determination result is satisfied; when the potential function value is greater than the preset potential function convergence threshold, the potential function convergence determination result is not satisfied. When the syndrome intensity value is not greater than the preset syndrome threshold, the syndrome convergence determination result is satisfied; when the syndrome intensity value is greater than the preset syndrome threshold, the syndrome convergence determination result is not satisfied. Only when both the potential function convergence determination result and the syndrome convergence determination result are satisfied, is the output determination result satisfied.

[0065] The beneficial effects of this invention are:

[0066] This invention constructs a reversible latent representation set through a reversible mapping network, maintaining a one-to-one mapping relationship between standardized data and forward encoding and reverse decoding, thus forming a traceable representation foundation during denoising and reconstruction. Based on the reversible latent representation set, structure-perturbation decomposition is performed to generate a set of structural latent variables and a set of perturbation latent variables. In the subsequent reconstruction stage, the set of structural latent variables is reconstructed to generate a set of reconstructed structural latent variables, explicitly isolating effective structural information from noise and random perturbation components within the latent space. This reduces the probability of structural details being mistakenly erased or noise being mistakenly reconstructed as details in end-to-end denoising methods. Furthermore, this invention employs an interleaved writing method to perform combinatorial encoding, generating a reconstructed combinatorial latent variable set and performing reverse decoding to obtain preliminary denoising and reconstruction results. Finally, fragment splicing and boundary continuity correction enable continuous output of multiple data fragments on a time index, improving the overall reconstruction stability and result usability under complex data input conditions.

[0067] Furthermore, this invention introduces a backtracking self-checking and closed-loop correction mechanism. The initial denoising and reconstruction results are forward-encoded again to obtain a backtracking latent representation set. This set is then differentially processed with the reconstructed combined latent variable set to generate a backtracking deviation set, which is then aggregated to obtain an observation residual set, forming a computable consistency deviation measure for the current reconstruction result. Based on the observation residual set, a segment channel residual matrix is ​​constructed, and a combined latent variable correction vector set is generated. A component-wise differential update is performed on the reconstructed combined latent variable set to obtain a closed-loop corrected combined latent variable set. Structural components are then extracted to form a closed-loop corrected structural latent variable set, enabling the denoising and reconstruction results to have iterative self-correcting capabilities and avoiding error solidification caused by a single output. Simultaneously, this invention introduces a latent space error correction mechanism on the closed-loop corrected structural latent variable set. A check matrix is ​​used to calculate a check syndrome vector, generating a syndrome anomaly score. Further, the error location set is located through the component contribution score sequence. An error correction update vector is generated according to the error location set, and additive updates are performed on the structural latent variables to obtain an error-corrected structural latent variable set. This achieves localized repair of structural damage, reducing structural drift caused by local missing or anomaly issues.

[0068] Ultimately, this invention uses the convergence determination result of the potential function and the convergence determination result of the syndrome to perform joint gating. The final denoised and reconstructed data result is output only when both types of determination results are satisfied, and a closed-loop update process is triggered when they are not satisfied. This improves the reliability, stability and controllability of the output result from a mechanism perspective. Attached Figure Description

[0069] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0070] Figure 1 This is a flowchart of a data denoising and reconstruction method based on machine learning proposed in this invention;

[0071] Figure 2 This is a schematic diagram illustrating the error location of a data denoising and reconstruction processing method based on machine learning proposed in this invention. Detailed Implementation

[0072] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0073] refer to Figure 1 and Figure 2 A data denoising and reconstruction method based on machine learning includes the following steps:

[0074] Obtain the raw data to be processed and perform preprocessing to generate standardized data;

[0075] Standardized data is input into the forward encoding path of the invertible mapping network to calculate the invertible latent representation vector and generate a set of invertible latent representations.

[0076] Perform structure-perturbation decomposition on the set of invertible latent representations to generate a set of structural latent variables and a set of perturbation latent variables;

[0077] The set of structural latent variables is reconstructed to generate a set of structural reconstruction latent variables, and then combined and encoded with the set of disturbance latent variables to generate a set of reconstructed combined latent variables.

[0078] The reconstructed set of latent variables is input into the reverse decoding path, and fragment splicing and boundary continuity correction are performed to generate preliminary denoising and reconstruction results.

[0079] The initial denoising and reconstruction results are input into the forward coding path to calculate the backtracking latent representation vector, which is then differiated with the reconstructed combined latent variable set to obtain the backtracking bias vector, generating the observation residual set.

[0080] Based on the set of observed residuals, a closed-loop correction is performed on the reconstructed combined latent variable set to generate a closed-loop corrected structural latent variable set.

[0081] Perform latent space error correction on the closed-loop modified structure latent variable set, perform error correction decoding update, and generate the error correction structure latent variable set;

[0082] The set of latent variables for error correction structure and the set of latent variables for perturbation are combined and encoded, input into the inverse decoding path, and the potential function convergence judgment result and the syndrome convergence judgment result are generated. The final denoising and reconstructed data result is generated based on the potential function convergence judgment result and the syndrome convergence judgment result.

[0083] In this embodiment, preprocessing includes time alignment, amplitude normalization, missing segment marking, and outlier removal.

[0084] In this embodiment, the generation of the reversible latent representation set specifically includes:

[0085] Standardized data is divided into sets of standardized data segments based on data segment identifiers;

[0086] Each data point in the standardized data records a data time index, a data channel identifier, and a data segment identifier;

[0087] The standardized data fragment set is input into the forward encoding path of the invertible mapping network, and the invertible mapping is performed on the standardized data fragment set to generate an invertible latent representation vector that corresponds one-to-one with the identifier of each data fragment.

[0088] The reversible latent representation vectors are aggregated according to the data segment identifiers to form a set of reversible latent representations;

[0089] The reversible mapping network uses a reversible neural network, which includes a forward encoding path and a reverse decoding path. The forward encoding path is composed of multiple reversible mapping layers connected in series. The forward encoding path outputs a reversible latent representation vector. The reverse decoding path is also composed of multiple reversible mapping layers connected in series, but the execution order is reversed from that of the forward encoding path. The reverse restoration operation of each reversible mapping layer is executed sequentially from the last reversible mapping layer to the first reversible mapping layer in the forward encoding path, and the segment decoded data is output.

[0090] Training a reversible mapping network includes: collecting and preprocessing data fragments to form a set of standardized data segments; inputting the set of standardized data segments into the forward encoding path of the reversible mapping network to generate a reversible latent representation vector; inputting the reversible latent representation vector into the reverse decoding path of the reversible mapping network to generate reconstructed data segments; calculating the reconstruction error between the reconstructed data segments and the standardized data segments as the training loss and performing gradient backpropagation on the training loss to update the parameters of the reversible mapping network; iteratively updating until the training loss converges and then fixing the parameters of the reversible mapping network.

[0091] In this embodiment, the generation of the set of structural latent variables and the set of disturbance latent variables includes:

[0092] Read the set of reversible latent representations and group the reversible latent representation vectors according to the data fragment identifiers to generate fragment-level reversible latent representation subsets, maintaining a one-to-one correspondence between the fragment-level reversible latent representation subsets and the data time index and data channel identifier;

[0093] Perform a structure-perturbation decomposition on each invertible latent representation vector in the fragment-level invertible latent representation subset to generate a structural latent variable vector and a perturbation latent variable vector;

[0094] Read the reversible latent representation vector bound to the data segment identifier, and determine the channel block boundary index of the reversible latent representation vector according to the data channel identifier. Divide the reversible latent representation vector into structural candidate block vectors and perturbation candidate block vectors according to the channel block boundary index. Calculate the absolute value of the difference between adjacent components of the structural candidate block vectors according to component position index i to obtain the structural difference amplitude sequence. Calculate the mean amplitude sequence within the window according to a fixed sliding window length on the structural difference amplitude sequence to obtain the structural local mean amplitude sequence. Simultaneously, calculate the maximum amplitude sequence within the window to obtain the structural local maximum amplitude sequence. Perform a weighted fusion operation on the i-th component of the structural local mean amplitude sequence and the i-th component of the structural local maximum amplitude sequence according to component position index i to generate a fused component, and assemble it into a structural stability score sequence according to the component position index order. Perform normalization on the structural stability score sequence to generate a normalized structural stability sequence. Normalize the structural stability. The following steps are performed: First, a structural weight vector is generated by mapping the permutation sequence component by component. Then, a component-wise weighted sum is applied to the structural candidate block vector and the structural weight vector to obtain the structural latent variable vector. Next, the absolute value of the difference between adjacent components is calculated for the perturbation candidate block vector according to component position index i, resulting in a perturbation difference amplitude sequence. Then, the amplitude variance sequence within the window is calculated on the perturbation difference amplitude sequence with a fixed sliding window length to obtain the perturbation local fluctuation variance sequence. Simultaneously, the amplitude mean sequence within the window is calculated to obtain the perturbation local fluctuation mean sequence. Based on component position index i, the i-th component of the perturbation local fluctuation variance sequence and the i-th component of the perturbation local fluctuation mean sequence are weighted and fused to generate a fused component. These components are then arranged in order of component position index to form a perturbation intensity score sequence. The perturbation intensity score sequence is normalized to generate a perturbation intensity normalized sequence. Finally, the perturbation intensity normalized sequence is mapped component by component to generate a perturbation weight vector. Finally, a component-wise weighted sum is applied to the perturbation candidate block vector and the perturbation weight vector to obtain the perturbation latent variable vector.

[0095] In structure-perturbation decomposition, the structural latent variable vector is a latent variable vector extracted from the invertible latent representation vector and used to characterize the effective structural information of the data. The perturbation latent variable vector is a latent variable vector extracted from the invertible latent representation vector and used to characterize noise and random perturbation components.

[0096] Structural consistency constraints are applied to the structural latent variable vectors, and amplitude boundary constraints are applied to the structural latent variable vectors according to the data channel identifiers to generate constrained structural latent variable vectors.

[0097] The process of performing amplitude boundary constraints includes: performing amplitude boundary constraint processing on each component of the structural latent variable vector; replacing the component value with the lower amplitude threshold when the component value is less than the lower amplitude threshold; replacing the component value with the upper amplitude threshold when the component value is greater than the upper amplitude threshold; generating a constrained structural latent variable vector and binding it to the data channel identifier for output.

[0098] Perturbation isolation is performed on the perturbation latent variable vector, and energy proportion allocation and intensity normalization are performed on the perturbation latent variable vector according to the data segment identifier to generate a normalized perturbation latent variable vector;

[0099] The generation of the normalized perturbation latent variable vector specifically includes: reading the perturbation latent variable vectors and forming a segment perturbation latent variable group based on the data segment identifier; calculating the sum of the absolute values ​​of the components of each perturbation latent variable vector in the segment perturbation latent variable group to obtain the perturbation energy value, and summing all the perturbation energy values ​​in the segment perturbation latent variable group to obtain the total perturbation energy value of the segment; calculating the ratio of each perturbation energy value to the total perturbation energy value of the segment to obtain the perturbation energy proportion score, and using the perturbation energy proportion score as the energy proportion weight value of the perturbation latent variable vector; calculating the sum of squares of the components of each perturbation latent variable vector to obtain the perturbation intensity value, and normalizing the perturbation latent variable vector using the perturbation intensity value to obtain the intensity-normalized perturbation latent variable vector; and performing a component-wise weighted summation of the intensity-normalized perturbation latent variable vector and the energy proportion weight value to obtain the normalized perturbation latent variable vector.

[0100] The constrained structural latent variable vector and the normalized perturbation latent variable vector are organized according to data segment identifiers to form a set of structural latent variables and a set of perturbation latent variables.

[0101] In this embodiment, the generation of the reconstructed set of latent variables includes:

[0102] Read the set of structural latent variables and group the structural latent variable vectors according to the data segment identifiers to generate a segment structural latent variable subset;

[0103] Reconstruction is performed on the subset of latent variables of the fragment structure. Component linear mapping is performed on the latent variable vector of the structure according to the data channel identifier to generate a structure mapping vector. Nonlinear transformation is performed on the structure mapping vector to generate a structure activation vector. Component aggregation is then performed on the structure activation vector to generate a structure reconstruction latent variable vector. The results are then summarized to obtain the set of latent variables for structure reconstruction.

[0104] Read the set of disturbance latent variables and pair them with the set of structure reconstruction latent variables according to the data segment identifier to generate a segment-level pairing set;

[0105] Perform combinatorial encoding on the fragment-level paired set to generate a reconstructed combinatorial latent variable vector, and summarize it according to the data fragment identifier to form a reconstructed combinatorial latent variable set;

[0106] The generation of the reconstructed combined latent variable vector includes: reading the structural reconstructed latent variable vector and the perturbation latent variable vector corresponding to the same data segment identifier; calculating the structural significance score of the structural reconstructed latent variable vector and the perturbation intensity score of the perturbation latent variable vector; generating the interleaving step size value and the interleaving start position index based on the structural significance score and the perturbation intensity score, and generating the structural component position set and the perturbation component position set accordingly; writing the components of the structural reconstructed latent variable vector into the corresponding positions of the reconstructed combined latent variable vector according to the structural component position set, and writing the components of the perturbation latent variable vector into the corresponding positions of the reconstructed combined latent variable vector according to the perturbation component position set, thereby generating the reconstructed combined latent variable vector.

[0107] The generation of structural significance score and disturbance intensity score includes: reading the latent variable vector of structural reconstruction and calculating the absolute value sequence of differences between adjacent components according to component position index; summing the absolute value sequence of differences to obtain the structural change amplitude value; summing the absolute values ​​of the components of the latent variable vector of structural reconstruction to obtain the total structural amplitude value; and weighted summing the structural change amplitude value and the total structural amplitude value to obtain the structural significance score; calculating the sum of squares of the components of the latent variable vector of disturbance to obtain the disturbance energy value; simultaneously calculating the absolute value sequence of differences between adjacent components according to component position index of the latent variable vector of disturbance and summing it to obtain the disturbance fluctuation amplitude value; and weighted summing the disturbance energy value and the disturbance fluctuation amplitude value to obtain the disturbance intensity score.

[0108] The process of generating interleaving step size values ​​and interleaving start position indices, and generating sets of structural component positions and perturbation component positions includes: reading the structural saliency score and perturbation intensity score corresponding to the same data segment identifier, summing the structural saliency score and perturbation intensity score to obtain the total score; using the ratio of the structural saliency score to the total score as the structural proportion coefficient, and the ratio of the perturbation intensity score to the total score as the perturbation proportion coefficient; reading the structural reconstruction latent variable vector and perturbation latent variable vector corresponding to the same data segment identifier, calculating the total number of structural reconstruction latent variable vector components and the total number of perturbation latent variable vector components, and summing them to obtain the total number of combined components; calculating the structural occupancy quantity value based on the structural proportion coefficient, where the structural occupancy quantity value is a positive integer obtained by rounding the product of the total number of combined components and the structural proportion coefficient, and subtracting the structural occupancy quantity value from the total number of combined components to obtain the perturbation occupancy quantity value; calculating the interleaving step size value based on the total number of combined components and the structural occupancy quantity value, and interleaving... The weaving step size is the result of dividing the total number of combined components by the number of structural occupants. If the divisor is less than 1, the weaving step size is replaced with 1. An interweaving start position index is generated based on the relationship between the structural significance score and the disturbance intensity score. When the structural significance score is greater than the disturbance intensity score, the interweaving start position index is set as the first component position index; when the structural significance score is less than the disturbance intensity score, the interweaving start position index is set as the second component position index. The structural component position set and the disturbance component position set are initialized to empty sets. Candidate structural position indices are generated recursively from the interweaving start position index according to the interweaving step size and written into the structural component position set one by one. The recursion stops when the number of position indices in the structural component position set reaches the number of structural occupants. All component position indices corresponding to the total number of combined components are traversed, and component position indices not yet written into the structural component position set are written into the disturbance component position set in the order of component position indices.

[0109] In this embodiment, the generation of the preliminary denoising and reconstruction results specifically includes:

[0110] Read the set of reconstructed combined latent variables, and organize the vector of reconstructed combined latent variables into a subset of fragment reconstructed combined latent variables according to the data fragment identifier, while maintaining the correspondence between the subset of fragment reconstructed combined latent variables and the data time index and data channel identifier;

[0111] The fragment reconstruction and combination latent variable subset is input into the inverse decoding path of the inverse mapping network, and inverse decoding is performed to generate fragment decoding data that corresponds one-to-one with the data fragment identifier.

[0112] Perform fragment splicing on the decoded fragment data to form preliminary spliced ​​data;

[0113] Boundary continuity correction is performed on the initial spliced ​​data. Boundary transition windows are extracted at the splicing positions of adjacent segments. The difference sequence of adjacent sampling points within the transition window is calculated and boundary discontinuity scores are generated. When the boundary discontinuity scores exceed the preset continuity threshold, linear interpolation is performed on the sampling points within the transition window and a corrected transition window is generated. The corrected transition window is written back to obtain boundary continuity correction data.

[0114] The boundary continuity correction data is output according to the data channel identifier to form the preliminary denoising and reconstruction results.

[0115] In this embodiment, the generation of the observation residual set specifically includes:

[0116] The preliminary denoising and reconstruction results are organized into a preliminary result fragment set according to the data fragment identifier;

[0117] The preliminary result fragment set is input into the forward encoding path of the invertible mapping network to generate backtracking latent representation vectors, which are then summarized to form a backtracking latent representation set.

[0118] Read the reconstructed set of combined latent variables and align it with the backtracked set of latent representations to obtain the fragment alignment set;

[0119] Perform component-wise differencing on each backtracking latent representation vector in the fragment alignment set and the corresponding reconstructed combined latent variable vector to generate a backtracking bias vector, and bind the backtracking bias vector with the data fragment identifier to form a backtracking bias set;

[0120] Aggregate the backtracking deviation set according to the data channel identifier. Calculate the sum of the absolute values ​​of the components of the backtracking deviation vector corresponding to the same data channel identifier to obtain the channel residual value. Then, summarize the channel residual values ​​according to the data segment identifier to form the observation residual set.

[0121] In this embodiment, the generation of the latent variable set of the closed-loop modified structure specifically includes:

[0122] Read the set of observation residuals and establish an observation residual index based on the data segment identifier and data channel identifier to form a segment-channel residual matrix;

[0123] The observation residual index uses the data segment identifier and the data channel identifier as keys to locate the positional mapping relationship between each channel residual value in the observation residual set and the corresponding data segment and the corresponding data channel. The segment-channel residual matrix is ​​a two-dimensional residual value matrix filled with the channel residual values ​​in the observation residual set, with the data segment identifier as the row index and the data channel identifier as the column index.

[0124] Generate a set of combined latent variable correction vectors based on the segment channel residual matrix;

[0125] Read the channel residual matrix of the data segment and select the channel residual value vector corresponding to the current data segment identifier according to the data segment identifier; initialize the residual filling vector with the same total number of combined components as an all-zero vector; read the channel residual value one by one according to the data channel identifier, obtain the component position index interval corresponding to the data channel identifier according to the data channel identifier, and assign the channel residual value to the residual filling vector at the component position index corresponding to the component position index interval to generate the residual filling vector; calculate the sum of the absolute values ​​of the components of the residual filling vector to obtain the total residual value. When the total residual value is zero, the combined latent variable correction vector is set to an all-zero vector. When the total residual value is not zero, normalize and update the residual filling vector with the total residual value to generate the combined latent variable correction vector, and form a set of combined latent variable correction vectors.

[0126] Read the reconstructed set of combined latent variables and align it with the set of modified vectors of combined latent variables according to the data segment identifier. Perform modification and update on each reconstructed set of combined latent variables to generate a closed-loop modified set of combined latent variables.

[0127] Read the set of reconstructed combined latent variables and the set of modified combined latent variable vectors. Establish a one-to-one alignment relationship based on the data segment identifier to generate a segment alignment set. Perform a component-by-component update operation on each set of reconstructed combined latent variable vectors and modified combined latent variable vectors in the segment alignment set. Subtract the corresponding component of the modified combined latent variable vector from each component of the reconstructed combined latent variable vector according to the component position index to obtain the modified component. Assemble all the modified components in the order of the component position index to obtain the closed-loop modified combined latent variable vector. Bind the closed-loop modified combined latent variable vector to the data segment identifier and write it into the closed-loop modified combined latent variable set.

[0128] Structural component extraction is performed on the closed-loop modified combined latent variable set to determine the set of structural component locations, and the closed-loop modified structural latent variable vectors are extracted and summarized to form the closed-loop modified structural latent variable set.

[0129] Read the closed-loop correction combined latent variable set and select the closed-loop correction combined latent variable vector one by one according to the data segment identifier; read the structural component position set corresponding to the same data segment identifier. The structural component position set is generated by the combination coding mapping relationship and stored in conjunction with the data segment identifier; extract the corresponding components from the closed-loop correction combined latent variable vector in sequence according to the component position index in the structural component position set, and assemble them according to the index order of the structural component position set to generate the closed-loop correction structural latent variable vector; bind the closed-loop correction structural latent variable vector with the data segment identifier and write it into the closed-loop correction structural latent variable set. The combination coding mapping relationship refers to the component position index mapping rule that determines the structural component position set and the disturbance component position set during the combination coding process and writes the structural reconstruction latent variable vector components and disturbance latent variable vector components into the corresponding positions of the reconstruction combined latent variable vector.

[0130] In this embodiment, the generation of the latent variable set of the error correction structure specifically includes:

[0131] Read the set of latent variables of the closed-loop modified structure, and organize the vector of latent variables of the closed-loop modified structure according to the data segment identifier to generate a subset of latent variables of the closed-loop modified structure of the segment.

[0132] Perform check syndrome calculation on each closed-loop modified structure latent variable vector in the subset of closed-loop modified structure latent variable vectors of the segment to generate a check syndrome vector that corresponds one-to-one with the closed-loop modified structure latent variable vector.

[0133] Read the closed-loop modified structure latent variable vectors from the subset of data segment closed-loop modified structure latent variables, and call the check matrix bound to the data channel identifier according to the data segment identifier; read the check matrix row vectors one by one according to the row index, and read the closed-loop modified structure latent variable vectors one by one according to the component position index; perform component-by-component multiplication operation on each check matrix row vector and the closed-loop modified structure latent variable vector, and sum all the product results to obtain the row check value; assemble the row check values ​​in the row index order to generate the check syndrome vector.

[0134] The verification matrix is ​​a matrix parameter used to perform verification syndrome calculation on the closed-loop modified structural latent variable vector. Each row of the verification matrix corresponds to a verification constraint and is associated with the component position index of the structural latent variable vector. The verification matrix is ​​generated as a sparse matrix according to the dimension of the structural latent variable vector during the initialization stage, and the non-zero position index of each row is configured as a pair of adjacent component position indexes or a pair of fixed span component position indexes. At the same time, the values ​​of the non-zero elements are configured as preset constants and bound and stored according to the data channel identifier.

[0135] The sum of the absolute values ​​of the components of the verification syndrome vector is used to obtain the syndrome intensity value, and the syndrome intensity value is compared with the preset syndrome threshold to generate a syndrome abnormality score;

[0136] Read the verification syndrome vector and generate a syndrome absolute value sequence by taking the absolute value of each component according to the component position index; sum the syndrome absolute value sequence to obtain the syndrome intensity value; read the preset syndrome threshold, compare the syndrome intensity value with the preset syndrome threshold and generate a syndrome abnormality score. When the syndrome intensity value is not greater than the preset syndrome threshold, the syndrome abnormality score is set to zero. When the syndrome intensity value is greater than the preset syndrome threshold, the syndrome abnormality score is set to the difference between the syndrome intensity value and the preset syndrome threshold.

[0137] Based on the abnormal scores of the syndrome, error location is performed on the latent variable vector of the closed-loop modified structure to determine the set of error locations;

[0138] When determining the error location set, the latent variable vector of the closed-loop correction structure, the check syndrome vector, and the syndrome abnormality score are read, and the component contribution score sequence is initialized to an all-zero sequence. The check matrix is ​​read row by row, and the non-zero component position index set of the check matrix is ​​located. At the same time, the row check value corresponding to the row in the check syndrome vector is read and its absolute value is calculated to obtain the row contribution value. The row contribution value is assigned to the corresponding component position index of the component contribution score sequence according to the non-zero component position index set and accumulated and updated to obtain the component contribution score sequence. The preset positioning threshold is read, and the syndrome abnormality score is read to generate a threshold scaling factor. The threshold scaling factor is the ratio of the syndrome abnormality score to the preset syndrome threshold. The preset positioning threshold is multiplied and scaled using the threshold scaling factor to obtain the positioning threshold. The component contribution score sequence is traversed. When the component contribution score corresponding to a certain component position index is greater than the positioning threshold, the component position index is written into the error location set to form the error location set.

[0139] During the initialization phase, the preset syndrome threshold and preset location threshold are calculated and verified based on historical standardized data according to the data channel identifier, and the distribution of syndrome intensity value and component contribution score is verified. The fixed quantile values ​​of the corresponding distributions are taken as the thresholds.

[0140] Error correction decoding updates are performed on the components of the closed-loop correction structure latent variable vector corresponding to the set of error locations to generate an error correction update vector, and additive updates are performed on the closed-loop correction structure latent variable vector to generate an error correction structure latent variable vector.

[0141] Read the latent variable vector of the closed-loop correction structure and the set of error locations, and initialize the error correction update vector as a zero vector according to the component location index; traverse each component location index in the error location set, read the component contribution score and the location threshold, and perform difference on the component contribution score and the location threshold to obtain the over-threshold deviation value; multiply the over-threshold deviation value with the preset update coefficient to obtain the scalar value of the error correction update quantity, and write the scalar value of the error correction update quantity into the error correction update quantity vector at the corresponding position of the component location index; perform additive update on the latent variable vector of the closed-loop correction structure according to the component location index, and add the latent variable vector of the closed-loop correction structure and the error correction update quantity vector component by component to obtain the latent variable vector of the error correction structure;

[0142] The error correction structure latent variable vector is bound to the data segment identifier and written into the error correction structure latent variable set.

[0143] In this embodiment, the generation of the final denoised and reconstructed data results specifically includes:

[0144] Read the latent variable set of error correction structure and the latent variable set of disturbance, and establish a one-to-one segment pairing relationship based on the data segment identifier to form a segment error correction pairing set;

[0145] Perform combined encoding on the fragment error correction pairing set to generate an error correction combined latent variable vector, and summarize it according to the data fragment identifier to form an error correction combined latent variable set;

[0146] The combined encoding performed on the error correction structural latent variable set and the perturbation latent variable set is the same as the "combined encoding performed on the fragment-level pairing set". Both are based on the structural component position set and perturbation component position set corresponding to the same data fragment identifier. The structural vector components and perturbation vector components are interleaved and written into the corresponding positions of the combined latent variable vector according to the component position index to generate the combined latent variable vector. The only difference is that in this step, the structural reconstruction latent variable vector is replaced with the error correction structural latent variable vector. The generation and usage of the structural component position set and the perturbation component position set remain unchanged.

[0147] The set of error correction combined latent variables is input into the inverse decoding path of the reversible mapping network to generate closed-loop denoising and reconstruction data that correspond one-to-one with the data segment identifiers. The segment splicing is then performed according to the data time index to generate the closed-loop denoising and reconstruction results.

[0148] Read the observation residual set corresponding to the current loop, sum the channel residual values ​​in the observation residual set according to the data segment identifier to obtain the segment potential function value, and form a potential function value set by combining the segment potential function values ​​corresponding to each data segment identifier;

[0149] The potential function value is compared with the preset potential function convergence threshold to generate a potential function convergence determination result.

[0150] The preset potential function convergence threshold is calculated during the initialization phase based on the data channel identifier and the historical observation residual set to calculate the segment potential function value distribution and take the fixed quantile value of the distribution as the threshold.

[0151] The syndrome intensity value is obtained by calculating the sum of the absolute values ​​of the verification syndrome vector components, and the syndrome intensity value is compared with the preset syndrome threshold to generate the syndrome convergence determination result;

[0152] A joint judgment is performed on the potential function convergence judgment result and the syndrome convergence judgment result to generate an output judgment result. When the output judgment result is satisfied, the closed-loop denoising and reconstruction result is used as the final denoising and reconstruction data result. When the output judgment result is not satisfied, the closed-loop update process is triggered and the observation residual set, the error correction structure latent variable set, the error correction combined latent variable set, the closed-loop denoising and reconstruction result, the potential function convergence judgment result and the syndrome convergence judgment result are regenerated until the output judgment result is satisfied or the preset maximum number of loops is reached, and then the final denoising and reconstruction data result is output.

[0153] When the potential function value is not greater than the preset potential function convergence threshold, the potential function convergence determination result is satisfied; when the potential function value is greater than the preset potential function convergence threshold, the potential function convergence determination result is not satisfied. When the syndrome intensity value is not greater than the preset syndrome threshold, the syndrome convergence determination result is satisfied; when the syndrome intensity value is greater than the preset syndrome threshold, the syndrome convergence determination result is not satisfied. Only when both the potential function convergence determination result and the syndrome convergence determination result are satisfied, is the output determination result satisfied.

[0154] Example 1: To verify the feasibility of this invention in practice, it was applied to a multi-channel vibration and current acquisition data processing scenario in an industrial equipment operation monitoring system. This system is deployed long-term in the production workshop of a manufacturing enterprise in East China, collecting vibration, current, and temperature signals generated by the equipment during continuous operation. Due to factors such as mechanical shock, electromagnetic interference, and sensor aging in the field environment, the acquired raw data contains abrupt amplitude changes, locally missing segments, and random high-frequency noise. Existing processing methods often employ filtering and conventional autoencoder reconstruction models for data repair. While this can reduce the overall noise amplitude, it easily leads to missmoothing of structures or erroneous amplification of abnormal peaks at locally missing locations. Furthermore, it cannot internally determine which locations have experienced structural drift, nor can it perform targeted repairs at erroneous locations.

[0155] In this scenario, the collected raw data is first time-aligned, amplitude-standardized, and missing segment marked to form standardized data, and the data time index, data channel identifier, and data segment identifier are recorded. Then, the standardized data is input into the forward encoding path of the reversible mapping network to generate a reversible latent representation set. Structure-perturbation decomposition is then performed within the latent space to obtain a set of structural latent variables and a set of perturbation latent variables. By performing reconstruction generation processing on the set of structural latent variables and combining it with the set of perturbation latent variables, a reconstructed combined latent variable set is generated. Inverse decoding yields the preliminary denoising and reconstruction results. For the local abnormal fluctuations still present in the preliminary results, this invention further performs backtracking latent representation calculation to generate an observation residual set. Based on the observation residual set, a segment channel residual matrix is ​​constructed, and closed-loop correction is performed on the reconstructed combined latent variable set. Subsequently, check syndrome calculation and error location are performed on the closed-loop corrected structural latent variable set. Error correction decoding updates are performed on the located structural components, and the components are combined and decoded again to generate the closed-loop denoising and reconstruction results.

[0156] In actual operation, the system continuously records the processing time and processing result status of the daily collected data. Through joint gating of the potential function convergence judgment result and the syndrome convergence judgment result, the final denoising and reconstructed data result is output only when both types of judgment results are satisfied.

[0157] To verify the performance of the present invention in practice, a comparative experiment was conducted, and the results are shown in Table 1.

[0158] Table 1. Comparison of Noise Reduction and Reconstruction Performance of Multi-channel Industrial Operation Data

[0159] Method Name Structural retention rate (%) Noise suppression rate (%) Mean reconstruction error Error localization accuracy (%) False repair rate (%) Low-pass filtering method 78.4 62.7 0.148 12.3 21.6 Denoising autoencoder 85.9 74.5 0.102 26.8 14.9 Generative Reconstruction Model 88.7 81.3 0.086 31.4 11.2 Method of the present invention 94.6 89.8 0.041 83.7 6.3

[0160] As shown in Table 1, under the same data acquisition period and operating conditions, the method of this invention achieves a structure retention rate of 94.6%, which is approximately 5.9 percentage points higher than the generative reconstruction model and approximately 8.7 percentage points higher than the denoising autoencoder. This difference indicates that, under conditions of local missing values ​​and strong interference noise, this invention separates the structural latent variables from the perturbation latent variables in the latent space and maintains the stability of the structural component positions through combined encoding, effectively reducing the probability of structural information being mistakenly erased.

[0161] In terms of noise suppression rate, this invention achieves 89.8%, which is more than 27 percentage points higher than the traditional low-pass filtering method and about 15 percentage points higher than the denoising autoencoder. This improvement mainly comes from the closed-loop correction mechanism and the observation residual-driven update mechanism, which enable the model to perform secondary corrections on noise components that still exist after reconstruction, rather than outputting results all at once.

[0162] The mean reconstruction error of this invention is only 0.041, significantly lower than other methods. The reason for this decrease is that the reversible mapping network provides a backtrackable potential representation, which allows for re-encoding and calculation of the backtracking bias after each decoding, and gradually approximating the original structure distribution based on the differential update mechanism.

[0163] In terms of error location accuracy, this invention achieves 83.7%, far exceeding other methods, while reducing the false correction rate to 6.3%. Traditional methods typically cannot explicitly locate error positions and can only adjust the overall output. This invention generates a sequence of component contribution scores by calculating the check matrix and check syndrome, thereby achieving precise location of the error location set within the latent space, and then performing targeted error correction updates. Therefore, the location accuracy is improved while avoiding over-correction.

[0164] The data above show that the present invention has advantages in terms of structure preservation capability, noise suppression effect, error location accuracy and error repair control. Its performance improvement mainly comes from the synergistic effect of the backtrackable characteristics of the reversible latent representation, the separation mechanism between structure and disturbance, the closed-loop correction mechanism based on observation residuals, and the latent space error correction mechanism based on verification syndrome.

[0165] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A data denoising and reconstruction method based on machine learning, characterized in that, Includes the following steps: Obtain the raw data to be processed and perform preprocessing to generate standardized data; Standardized data is input into the forward encoding path of the invertible mapping network to calculate the invertible latent representation vector and generate a set of invertible latent representations. Perform structure-perturbation decomposition on the set of invertible latent representations to generate a set of structural latent variables and a set of perturbation latent variables; The set of structural latent variables is reconstructed to generate a set of structural reconstruction latent variables, and then combined and encoded with the set of disturbance latent variables to generate a set of reconstructed combined latent variables. The reconstructed set of latent variables is input into the reverse decoding path, and fragment splicing and boundary continuity correction are performed to generate preliminary denoising and reconstruction results. The initial denoising and reconstruction results are input into the forward coding path to calculate the backtracking latent representation vector, which is then differiated with the reconstructed combined latent variable set to obtain the backtracking bias vector, generating the observation residual set. Based on the set of observed residuals, a closed-loop correction is performed on the reconstructed combined latent variable set to generate a closed-loop corrected structural latent variable set. Perform latent space error correction on the closed-loop modified structure latent variable set, perform error correction decoding update, and generate the error correction structure latent variable set; The set of latent variables for error correction structure and the set of latent variables for perturbation are combined and encoded, input into the inverse decoding path, and the potential function convergence judgment result and the syndrome convergence judgment result are generated. The final denoising and reconstructed data result is generated based on the potential function convergence judgment result and the syndrome convergence judgment result.

2. The data denoising and reconstruction method based on machine learning according to claim 1, characterized in that, The preprocessing includes time alignment, amplitude normalization, missing segment marking, and outlier removal.

3. The data denoising and reconstruction method based on machine learning according to claim 1, characterized in that, The generation of the reversible latent representation set specifically includes: Standardized data is divided into sets of standardized data segments based on data segment identifiers; The standardized data fragment set is input into the forward encoding path of the invertible mapping network, and the invertible mapping is performed on the standardized data fragment set to generate an invertible latent representation vector that corresponds one-to-one with the identifier of each data fragment. The reversible latent representation vectors are aggregated according to the data segment identifiers to form a set of reversible latent representations.

4. The data denoising and reconstruction method based on machine learning according to claim 1, characterized in that, The generation of the set of structural latent variables and the set of disturbance latent variables includes: Read the set of invertible latent representations and group the invertible latent representation vectors according to the data fragment identifiers to generate a fragment-level invertible latent representation subset; Perform a structure-perturbation decomposition on each invertible latent representation vector in the fragment-level invertible latent representation subset to generate a structural latent variable vector and a perturbation latent variable vector; Perform structural consistency constraint on the structural latent variable vector to generate a constrained structural latent variable vector; Perturbation isolation is performed on the perturbation latent variable vector, and energy proportion allocation and intensity normalization are performed on the perturbation latent variable vector according to the data segment identifier to generate a normalized perturbation latent variable vector; The constrained structural latent variable vector and the normalized perturbation latent variable vector are organized according to data segment identifiers to form a set of structural latent variables and a set of perturbation latent variables.

5. The data denoising and reconstruction method based on machine learning according to claim 1, characterized in that, The generation of the reconstructed combination latent variable set includes: Read the set of structural latent variables and group the structural latent variable vectors according to the data segment identifiers to generate a segment structural latent variable subset; Reconstruction is performed on the subset of latent variables of the fragment structure. Component linear mapping is performed on the latent variable vector of the structure according to the data channel identifier to generate a structure mapping vector. Nonlinear transformation is performed on the structure mapping vector to generate a structure activation vector. Component aggregation is then performed on the structure activation vector to generate a structure reconstruction latent variable vector. The results are then summarized to obtain the set of latent variables for structure reconstruction. Read the set of disturbance latent variables and pair them with the set of structure reconstruction latent variables according to the data segment identifier to generate a segment-level pairing set; Perform combinatorial encoding on the fragment-level paired set to generate a reconstructed combinatorial latent variable vector, and summarize it according to the data fragment identifier to form a reconstructed combinatorial latent variable set; The generation of the reconstructed combined latent variable vector includes: reading the structural reconstructed latent variable vector and the perturbation latent variable vector corresponding to the same data segment identifier; calculating the structural significance score of the structural reconstructed latent variable vector and the perturbation intensity score of the perturbation latent variable vector; generating the interleaving step size value and the interleaving start position index based on the structural significance score and the perturbation intensity score, and generating the structural component position set and the perturbation component position set accordingly; writing the components of the structural reconstructed latent variable vector into the corresponding positions of the reconstructed combined latent variable vector according to the structural component position set, and writing the components of the perturbation latent variable vector into the corresponding positions of the reconstructed combined latent variable vector according to the perturbation component position set, thus generating the reconstructed combined latent variable vector.

6. The data denoising and reconstruction method based on machine learning according to claim 1, characterized in that, The generation of the preliminary denoising and reconstruction results specifically includes: Read the set of reconstructed combination latent variables and organize the vector of reconstructed combination latent variables into a subset of fragment reconstructed combination latent variables based on the data fragment identifier; The fragment reconstruction combined latent variable subset is input into the inverse decoding path of the inverse mapping network, and inverse decoding is performed to generate fragment decoding data; Perform fragment splicing on the decoded fragment data to form preliminary spliced ​​data; Perform boundary continuity correction on the initial spliced ​​data, generate a correction transition window, and write back the correction transition window to obtain boundary continuity corrected data; The boundary continuity correction data is output according to the data channel identifier to form the preliminary denoising and reconstruction results.

7. The data denoising and reconstruction method based on machine learning according to claim 1, characterized in that, The generation of the observation residual set specifically includes: The preliminary denoising and reconstruction results are organized into a preliminary result fragment set according to the data fragment identifier; The preliminary result fragment set is input into the forward encoding path of the invertible mapping network to generate backtracking latent representation vectors, which are then summarized to form a backtracking latent representation set. Read the reconstructed set of combined latent variables and align it with the backtracked set of latent representations to obtain the fragment alignment set; Perform component-wise differencing on each backtracking latent representation vector in the fragment alignment set and the corresponding reconstructed combined latent variable vector to generate a backtracking bias vector, and bind the backtracking bias vector with the data fragment identifier to form a backtracking bias set; Aggregate the backtracking deviation set according to the data channel identifier. Calculate the sum of the absolute values ​​of the components of the backtracking deviation vector corresponding to the same data channel identifier to obtain the channel residual value. Then, summarize the channel residual values ​​according to the data segment identifier to form the observation residual set.

8. The data denoising and reconstruction method based on machine learning according to claim 1, characterized in that, The generation of the closed-loop modified structure latent variable set specifically includes: Read the set of observation residuals and establish an observation residual index based on the data segment identifier and data channel identifier to form a segment-channel residual matrix; Generate a set of combined latent variable correction vectors based on the segment channel residual matrix; Read the reconstructed set of combined latent variables and align it with the set of modified vectors of combined latent variables according to the data segment identifier. Perform modification and update on each reconstructed set of combined latent variables to generate a closed-loop modified set of combined latent variables. Structural component extraction is performed on the closed-loop modified combined latent variable set to determine the set of structural component locations, and the closed-loop modified structural latent variable vectors are extracted and summarized to form the closed-loop modified structural latent variable set.

9. The data denoising and reconstruction method based on machine learning according to claim 1, characterized in that, The generation of the latent variable set of the error correction structure specifically includes: Read the set of latent variables of the closed-loop modified structure, and organize the vector of latent variables of the closed-loop modified structure according to the data segment identifier to generate a subset of latent variables of the closed-loop modified structure of the segment. Perform check syndrome calculation on each closed-loop modified structure latent variable vector in the subset of fragment closed-loop modified structure latent variables to generate a check syndrome vector; The sum of the absolute values ​​of the components of the verification syndrome vector is used to obtain the syndrome intensity value, and the syndrome intensity value is compared with the preset syndrome threshold to generate a syndrome abnormality score; Based on the abnormal scores of the syndrome, error location is performed on the latent variable vector of the closed-loop modified structure to determine the set of error locations; Error correction decoding updates are performed on the components of the closed-loop correction structure latent variable vector corresponding to the set of error locations to generate an error correction update vector, and additive updates are performed on the closed-loop correction structure latent variable vector to generate an error correction structure latent variable vector. The error correction structure latent variable vector is bound to the data segment identifier and written into the error correction structure latent variable set.

10. The data denoising and reconstruction method based on machine learning according to claim 1, characterized in that, The generation of the final denoised and reconstructed data results specifically includes: Read the latent variable set of error correction structure and the latent variable set of disturbance, and establish a one-to-one segment pairing relationship based on the data segment identifier to form a segment error correction pairing set; Perform combined encoding on the fragment error correction pairing set to generate an error correction combined latent variable vector, and summarize it according to the data fragment identifier to form an error correction combined latent variable set; The set of error correction combined latent variables is input into the inverse decoding path of the invertible mapping network to generate closed-loop denoising and reconstruction data, and the fragment splicing is performed according to the data time index to generate closed-loop denoising and reconstruction results. Read the observation residual set corresponding to the current loop, sum the channel residual values ​​in the observation residual set according to the data segment identifier to obtain the segment potential function value, and form a potential function value set by combining the segment potential function values ​​corresponding to each data segment identifier; The potential function value is compared with the preset potential function convergence threshold to generate a potential function convergence determination result. The syndrome intensity value is obtained by calculating the sum of the absolute values ​​of the verification syndrome vector components, and the syndrome intensity value is compared with the preset syndrome threshold to generate the syndrome convergence determination result; A joint judgment is performed on the potential function convergence judgment result and the syndrome convergence judgment result to generate an output judgment result. When the output judgment result is satisfied, the closed-loop denoising and reconstruction result is used as the final denoising and reconstruction data result. When the output judgment result is not satisfied, the closed-loop update process is triggered and the observation residual set, the error correction structure latent variable set, the error correction combined latent variable set, the closed-loop denoising and reconstruction result, the potential function convergence judgment result and the syndrome convergence judgment result are regenerated until the output judgment result is satisfied or the preset maximum number of loops is reached, and then the final denoising and reconstruction data result is output. When the potential function value is not greater than the preset potential function convergence threshold, the potential function convergence determination result is satisfied; when the potential function value is greater than the preset potential function convergence threshold, the potential function convergence determination result is not satisfied. When the syndrome intensity value is not greater than the preset syndrome threshold, the syndrome convergence determination result is satisfied; when the syndrome intensity value is greater than the preset syndrome threshold, the syndrome convergence determination result is not satisfied. Only when both the potential function convergence determination result and the syndrome convergence determination result are satisfied, is the output determination result satisfied.

Images