Data quality evaluation and restoration method based on machine learning

Abstract

The invention discloses a data quality evaluation and restoration method and system based on machine learning. The method comprises the following steps: firstly, preprocessing data, and constructing a structured generator network containing a parameterized adjacent matrix; in the training stage, the network outputs preliminary reconstruction data, and manifold constraint projection is utilized to find a logic correction target matrix meeting domain constraints; and constructing a potential consistency loop through gradient blocking, and calculating a consistency loss to drive the network to approach a compliance manifold. Meanwhile, an overall target integrating reconstruction errors, logic consistency and acyclic constraints is constructed, and an augmented Lagrangian method and a double-layer circulation strategy are adopted to jointly optimize parameters. And after the double convergence judgment is met, final inference is executed based on mask synthesis. According to the method, the logic rule is internalized into the generation capability, and the sparse causal structure is mined synchronously, so that high-quality data recovery with logic compliance, distribution consistency and interpretability is realized.

Description

Technical Field

[0001] This invention relates to the fields of data processing and artificial intelligence technology, specifically to a data quality assessment and repair method based on machine learning. Background Technology

[0002] In big data applications such as industrial manufacturing, financial risk control, and healthcare, data integrity is fundamental to ensuring the accuracy of subsequent modeling, analysis, and decision-making. However, due to various factors such as sensor hardware failures, network packet loss, privacy protection policies, or human error in data entry, raw datasets commonly contain varying degrees of missing values. Currently, industry primarily relies on data imputation techniques to address missing data, including statistical imputation based on mean and median, as well as machine learning imputation methods such as chain equation multiple imputation, generative adversarial networks, or variational autoencoders. These techniques typically train models using existing observational data, predicting and filling in missing values ​​by capturing statistical correlations between data features, thereby attempting to restore the integrity and usability of the dataset.

[0003] While existing deep generative imputation models have made some progress in reducing data reconstruction errors, they still have fundamental limitations when dealing with high-dimensional data containing complex dependencies. Most existing techniques employ black-box fitting strategies, focusing only on making the imputed values ​​probabilistically close to the observed data, often neglecting the inherent causal mechanisms between feature variables and the strict logical constraints that must be followed in specific application domains. This neglect of causal structures and business rules makes the models prone to generating statistically high-fit data that actually violates physical laws or business logic. Furthermore, due to the lack of explicit modeling of the data generation process, existing imputation algorithms struggle to provide intuitive explanations of the dependencies between variables, failing to meet the stringent requirements of logical compliance and interpretability of data repair results in high-reliability scenarios. Summary of the Invention

[0004] The first aspect of this invention provides a data quality assessment and repair method based on machine learning. This method primarily addresses the problem of missing value imputation in multidimensional data, while simultaneously uncovering potential causal structures between data variables and ensuring that the repaired data conforms to the logical constraints of a specific domain.

[0005] The method includes: preprocessing the original input matrix containing missing values ​​and observations to generate an initialized input matrix and a corresponding binary mask matrix. Based on this, a structured generator network is constructed, which includes a parameterized adjacency matrix representing the causal relationships between variables.

[0006] During model training, the computer system executes iterative operations including inner-layer optimization. In each iteration, a structured generator network is used to perform forward propagation on the initial input matrix, outputting a preliminary reconstructed data matrix. Then, a manifold constraint projection step is performed. This step uses the preliminary reconstructed data matrix as initial values ​​and, while keeping the structured generator network parameters fixed, iteratively optimizes to find a logically corrected target matrix that satisfies the set of neighborhood constraints. This logically corrected target matrix numerically satisfies predefined logical rules and is adjacent to the preliminary reconstructed data matrix in Euclidean space.

[0007] To internalize logical constraints into the network's generative capabilities, a latent consistency learning loop is constructed and a latent consistency loss is calculated. This loss measures the consistency between the initially reconstructed data matrix and the logically corrected target matrix. During this process, a gradient blocking operation is performed on the logically corrected target matrix, setting it as a fixed constant tensor that does not participate in gradient backpropagation. By minimizing the latent consistency loss, the structured generator network directly approximates the manifold space satisfying the set of neighborhood constraints in a single forward propagation, thus transforming the non-differentiable logical constraint solution process into a differentiable distance approximation problem.

[0008] Furthermore, to synergistically optimize data reconstruction, logical compliance, and graph structure sparsity, an augmented Lagrange overall training objective is constructed. This overall training objective integrates reconstruction error loss, overall logical consistency loss, latent consistency loss, and acyclic constraint penalty term for the parameterized adjacency matrix. Based on this overall training objective, the weight parameters and parameterized adjacency matrix of the structured generator network are updated. When the preset convergence condition is met, the trained structured generator network performs the final inference on the data to be processed, outputting the repaired complete data.

[0009] In one specific implementation, the forward propagation process of the structured generator network employs an architecture combining linear causal aggregation and nonlinear feature mapping. First, a parameterized adjacency matrix is ​​used to perform linear transformation and feature aggregation on the initial input matrix; this step simulates the direct causal interactions between variables. Subsequently, the aggregated features are input to a nonlinear mapping layer to fit a complex data distribution. The parameterized adjacency matrix, as a learnable parameter jointly optimized with the network weights, explicitly models the directed weighted connectivity relationships between the dimensions of data features.

[0010] For the manifold-constrained projection step, a mask-protected projection gradient descent strategy is adopted. The computer system constructs a constrained optimization problem with the goal of minimizing the distance to the initially reconstructed data matrix while satisfying the neighborhood constraints. During the iteration process, the gradient of the logical loss relative to the current data matrix is ​​calculated, and this gradient is used to update only the values ​​at the missing value positions. Simultaneously, a Hadamard product operation is performed with the inverse of the binary mask matrix to forcibly reset the values ​​at the observation positions, ensuring they always remain as the original observation data. After the gradient update, analytical projection and truncation operations are performed to ensure that the updated values ​​strictly fall within a predefined numerical range.

[0011] To optimize the overall training objective, the augmented Lagrange multiplier method is employed to handle the equality constraints of the directed acyclic graph (DAG). The overall training objective includes a main task loss term weighted by reconstruction error loss, overall logical consistency loss, and potential consistency loss; a sparse regularization term used to induce the generation of a sparse causal graph structure; and an acyclic constraint penalty term consisting of Lagrange multiplier terms and quadratic penalty parameter terms. This method employs a two-layer loop strategy for parameter updates: in the inner loop, the Lagrange multipliers and quadratic penalty parameters are kept constant, and the network parameters are updated using the optimizer; in the outer loop, the quadratic penalty parameter and Lagrange multipliers are dynamically adjusted based on the decrease in the acyclic constraint function value. If the decrease in the acyclic constraint function value is insufficient, the quadratic penalty parameter is increased to enhance the penalty, thereby gradually tightening the constraints during the optimization process and forcing the graph structure to evolve towards an acyclic state.

[0012] A dual-criteria mechanism is employed for model convergence determination. The first criterion determines structural constraint convergence, i.e., the acyclic constraint function value calculated from the parameterized adjacency matrix is ​​lower than the numerical tolerance threshold. The second criterion determines task loss stability, i.e., the relative rate of change of the task's main loss term on the validation dataset is less than the convergence threshold. Training is considered complete only when both criteria are met simultaneously.

[0013] In the final inference stage, the trained model is used to perform efficient data repair and structure extraction. On one hand, a hard threshold truncation operation is performed on the parameterized adjacency matrix to construct a binary adjacency matrix as the output of the mined causal structure. On the other hand, the structure generator network is switched to inference mode to initialize and fill in missing data and perform forward propagation. Subsequently, a non-destructive data synthesis operation is performed, using a mask matrix to fuse the original observations with the network-generated imputation values, ensuring that the original records are not modified. Finally, numerical range truncation and discretization are performed on the fused data to output the final compliant imputation result.

[0014] A second aspect of the present invention provides a data quality assessment and repair system based on machine learning, the system comprising one or more processors and a memory. The memory is used to store one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the method described in the first aspect.

[0015] The technical solution of this invention effectively solves the problem of traditional generative models struggling to handle complex logical constraints by introducing a latent consistency learning loop. By internalizing the logical constraint solving process into the network's generative capability, the efficiency of the inference stage is significantly improved, avoiding the computational overhead of online iterative optimization. Simultaneously, by combining causal structure learning with the augmented Lagrange method, the underlying causal mechanisms of the data can be revealed while filling in the data, achieving a dual improvement in data repair accuracy and model interpretability.

[0016] This invention provides a data quality assessment and repair method based on machine learning. It has the following beneficial effects: 1. This invention constructs a structured generator network containing a parameterized adjacency matrix and combines it with the augmented Lagrange multiplier method to handle the constraints of directed acyclic graphs, achieving the effect of simultaneously mining the causal structure between variables in data imputation tasks. This method utilizes a two-layer optimization strategy of sparse regularization and dynamic penalty parameters to force the network parameters to converge while eliminating redundant spurious associations, so that the final output binary adjacency matrix can accurately reflect the essential causal dependency topology between feature dimensions. This effectively overcomes the shortcomings of traditional imputation methods that only focus on statistical fitting and ignore the data generation mechanism, and significantly improves the interpretability of the model.

[0017] 2. This invention introduces a latent consistency learning loop and a manifold constraint projection mechanism to internalize non-differentiable domain logic rules into the neural network's generation capability. During the training phase, this method uses gradient blocking technology to construct a logic correction target matrix as a supervision signal. By minimizing the latent consistency loss, it drives the network distribution to approximate a compliant manifold, thereby endowing the model with the ability to follow complex logical constraints without changing the network structure. This allows the model to directly output high-quality data that meets business rules during the inference phase, avoiding the computational overhead caused by the time-consuming iterative post-processing correction in traditional methods during inference.

[0018] 3. This invention constructs an overall training objective that integrates reconstruction error, logical consistency, and causal structure, and combines it with a mask-protected data synthesis strategy. This achieves the effect of optimizing the accuracy of missing value filling while ensuring the integrity of the original observation data. The method uses a dual convergence criterion to collaboratively monitor the structural evolution and task fitting status, ensuring the stability of the model in the multi-objective optimization process. At the same time, by utilizing the mask fusion operation in the inference stage, it strictly limits the prediction and filling of missing positions without tampering with any real observation records, thereby ensuring the rationality of the statistical distribution of the repaired data and its authenticity and credibility in business applications. Attached Figure Description

[0019] Figure 1 This is a flowchart of the data repair method based on causal structure learning of the present invention. Detailed Implementation

[0020] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Example: Please see the appendix Figure 1 The embodiments of the present invention provide a hardware structure for an electronic device, which may be an independent physical server, a graphics workstation, a personal computer, or a virtual computing node or a distributed computing cluster in a cloud computing platform.

[0022] Electronic devices mainly include processing components, storage components, communication components, and bus structures.

[0023] The processing component, as the computing core of the electronic device, is configured to execute computer program instructions stored in the storage component to implement various computational steps in the data repair process. Specifically, the processing component includes at least one central processing unit and at least one parallel computing acceleration unit. The parallel computing acceleration unit is selected from one or more combinations of a graphics processing unit, a tensor processing unit, a neural network processor, or a field-programmable gate array.

[0024] In practice, the CPU is configured to perform logic control, data flow scheduling, and pre-compilation processing of domain constraint rules. Pre-compilation processing includes processing the text-based set of domain constraints. This is resolved to the logical loss node in the computation graph. The parallel computing acceleration unit is configured to execute the generator network in the structural causal model. and encoder Forward and backward propagation calculations.

[0025] For the dynamic structure learning algorithm involved in this invention, the parallel computing acceleration unit is equipped with a dedicated matrix operation core for performing the following operations in parallel: For parameterized adjacency matrix Perform Hadamard product operation ( ); For matrix exponents Taylor series expansion or eigenvalue decomposition is performed to evaluate the acyclic constraint terms. ; In the manifold-constrained projection step, the constrained optimization problem is solved using a parallelized gradient flow iterative algorithm. The initially generated data is mapped to the logically feasible manifold.

[0026] The storage components include non-volatile and volatile storage media. The non-volatile storage media is used for persistent storage of the raw observation dataset to be processed. Predefined set of domain constraints And the trained model parameters. Specifically, the model parameters include the adjacency matrix of the finally converged causal graph. and generator network weights Volatile storage media is used to temporarily store intermediate calculation results during runtime. These intermediate calculation results include the mask matrix. Latent space representation vector and the gradient tensor generated during automatic differentiation. .

[0027] The bus architecture interconnects the processing, storage, and communication components. In embodiments involving collaborative computing by multiple parallel computing acceleration units, the bus architecture is configured to support direct memory access, enabling parallel computing acceleration units to directly read observation datasets from volatile storage media. This reduces the CPU's data copying overhead.

[0028] Communication components enable communication connections between electronic devices and other external devices or network systems. The electronic device receives data streams to be repaired from upstream databases or sensors via the communication component and transmits the repaired, high-quality data. Transmitted to downstream business systems.

[0029] At the software level, electronic devices run an operating system and a deep learning computing framework. The deep learning computing framework is configured to build a dynamic computation graph in volatile storage media and provides automatic differentiation functionality. This automatic differentiation functionality is used to calculate the logistic loss function according to the chain rule. Relative to the adjacency matrix The gradient is used to support gradient-based causal structure search. The kernel scheduling, memory paging management, and underlying operator implementation of the computing framework are well-known technologies to those skilled in the art and will not be elaborated upon here.

[0030] The data quality assessment and repair system based on dynamic causal evolution and counterfactual potential calibration includes a data preprocessing module, a dynamic causal structure learning module, a structured generative inference module, a logically constrained differentiable compilation module, a manifold projection and potential calibration module, and a joint optimization and update module. These modules work collaboratively under the scheduling of the processing components to achieve end-to-end repair from dirty data to high-quality, logically consistent data.

[0031] The data preprocessing module is configured to receive the raw observation dataset. and domain constraint set The data preprocessing module scans the observation dataset, identifies missing values ​​(NaN) or values ​​marked as outliers, and constructs a mask matrix accordingly. Mask matrix With observation dataset Having the same dimensions, where the mask matrix elements in Identifier The first sample Each attribute represents a valid observation, and the mask matrix is ​​used for this purpose. elements in Identifier The first sample These attributes are values ​​to be repaired.

[0032] The dynamic causal structure learning module is configured to initialize and maintain a learnable parameterized adjacency matrix. Parameterized adjacency matrix It is a real-valued matrix used to characterize the strength of the potential causal relationship between data variables. The Dynamic Causal Structure Learning module is also configured to compute acyclic constraint terms. Acyclic constraint terms Trace operation based on matrix exponent Construct a topological constraint for applying directed acyclic graphs (DAGs) during the optimization process.

[0033] The structured generative inference module is configured to build a neural network-based structural causal model generator. The structured generation inference module receives masked observation data. Exogenous noise variables sampled from a standard normal distribution and the adjacency matrix provided by the dynamic causal structure learning module. As input, the structured generation inference module performs a structure masking operation during the generation process. Specifically, the structure masking operation involves modifying the parameterized adjacency moments... As a mask, it is multiplied by Hadamard (element-wise) with the weight matrix of the first fully connected layer of the generator network. This operation forces the connection weights of the neural network that do not conform to the causal graph structure to zero, thus causing the generator network to use only the adjacency matrix. The parent node variable information corresponding to the non-zero element is used to generate the estimated value of the child node variable, and the output is preliminary repair data that reflects the causal dependency. .

[0034] The logic constraint differentiable compiler module is configured to take a set of domain constraints in textual or symbolic form. This is transformed into a differentiable loss function computation graph. The logic constraint differentiable compiler module has a built-in T-norm logic operation definition. The logic constraint differentiable compiler module will convert the domain constraint set... Boolean logic operations are mapped to algebraic operations over continuous fields. Specifically, the logic constraint differentiable compilation module maps the logical "AND" operation to a product operation or a minimum value operation, the logical "OR" operation to a maximum value operation or a probability summation operation, and the logical "NOT" operation to a subtraction of the probability value from 1. Based on these operational rules, the logic constraint differentiable compilation module performs preliminary data repair. Perform logic checks and calculate the loss due to logic violations. The logic constraint differentiable compiler module calculates the logic violation penalty through an automatic differentiation mechanism. Relative to the adjacency matrix and gradient Passed to the dynamic causal structure learning module for updating the adjacency matrix Causal weights in the equation.

[0035] The manifold projection and latent calibration module is configured to perform geometric correction of the data and logical alignment of the feature space. The manifold projection and latent calibration module includes constrained optimization solution logic and consistency coding logic. The constrained optimization solution logic is configured to build upon the initial data repair. As the initial point, with the set of domain constraints This is an optimization problem with hard constraints. The constrained optimization solution logic utilizes the projected gradient descent algorithm or the augmented Lagrange algorithm, iteratively updating data points during the inference process to calculate the projected data that satisfies all constraints. The consistency coding logic is configured to utilize the encoder network. The preliminary repair data will be respectively and projection data Mapping to the latent space yields the original latent representation. and calibration potential characterization The manifold projection and latent calibration module calculates the original latent characterization. and calibration potential characterization The Euclidean distance between them serves as a potential calibration loss. It is used to supervise the generator network to learn the potential distribution that conforms to the logical manifold.

[0036] The joint optimization update module is configured to construct the overall objective function. The overall objective function includes observation data reconstruction error, logic violation loss, acyclic constraint term, sparse regularization term, and potential calibration loss. The joint optimization update module uses the stochastic gradient descent algorithm to synchronously update the generator network parameters. Encoder network parameters and adjacency matrix This process continues until the system converges. After system convergence, the joint optimization and update module outputs the final causal adjacency matrix. And high-quality restored data after projection correction. .

[0037] The data quality assessment and repair method is executed by a computer system and includes the following steps S100 to S700.

[0038] Step S100: Receive the original observation dataset and the set of domain constraints, and perform preprocessing and differentiable compilation.

[0039] The computer system receives the raw observation dataset to be repaired. and predefined set of domain constraints The computer system scans the raw observation dataset. Use null or exception identifiers in the data to construct a mask matrix. Mask matrix The mask matrix is ​​used to distinguish between observed values ​​and missing values ​​in subsequent calculations. Compared with the original observation dataset Consistent dimensions. The computer system uses the original observation dataset. Missing positions in the input data are filled with zero or random noise to obtain the initial input data. Simultaneously, the computer system utilizes T-norm fuzzy logic rules to define the domain constraint set. The discrete logic rules in the code are transformed into continuously differentiable logic loss functions. Logical loss function Used to measure the degree to which generated data violates business rules.

[0040] Step S200: Initialize the dynamic causal structure parameters.

[0041] The computer system initializes a learnable parameterized adjacency matrix. Parameterized adjacency matrix For continuous real-valued matrices, parameterized adjacency matrix The numerical values ​​of the elements in the expression represent the strength of the causal relationship between variables. The computer system also defines acyclic constraint functions. Acyclic constraint function By calculating the trace of the matrix exponent To quantize the parameterized adjacency matrix The existence of cycles in the corresponding graph structure is determined to ensure that the final learned structure satisfies the property of a directed acyclic graph (DAG).

[0042] Step S300: Perform generative inference based on the structural causal model.

[0043] Computer systems utilize generator networks in structural causal models Generate preliminary repair data During the generation process, the computer system performs a structure masking operation. The structure masking operation transforms the parameterized adjacency matrix of continuous real values... Directly affects the generator network The weight parameters or input vector of the generator network make the generator network The output of the parameterized adjacency matrix Differentiable generator networks Preliminary repair data is calculated using the following formula. :

[0044] in Represents element-wise product. This refers to exogenous noise variables sampled from a standard normal distribution. Preliminary data repair is performed using structure masking operations. The generation path is forcibly constrained to the parameterized adjacency matrix. On the defined causal path.

[0045] Step S400: Calculate the logical violation gradient and reverse-guide the structural evolution.

[0046] The computer system compiles the logical loss function in step S100. Calculate preliminary repair data The degree of logical violation is then determined. Subsequently, the computer system uses an automatic differentiation mechanism to calculate the logical loss function. Relative to parameterized adjacency matrix gradient Due to initial data repair It is a parameterized adjacency matrix The function, gradient Able to indicate parameterized adjacency matrix The direction of numerical adjustment of the weights of each element in the algorithm. Specifically, the gradient. This reduces the weight of erroneous causal connections that lead to logical violations in subsequent updates.

[0047] Step S500: Perform manifold constraint projection and latent space calibration.

[0048] The computer system has begun preliminary data repair. Starting from the domain constraint set Given hard constraints, solve the constrained optimization problem to obtain projected data that satisfies all constraints. Subsequently, the computer system utilizes the encoder network. Initial data repair and projection data Mapping back to the latent space respectively, we obtain the original latent representation. and calibration potential characterization Computer systems compute the original latent representation. and calibration potential characterization The Euclidean distance between them serves as a potential calibration loss. :

[0049] in This indicates that the gradient operation should be stopped. Stopping the gradient operation instructs the computer system to stop calculating the potential calibration loss. Relative to generator network parameters When calculating the gradient, the underlying representation will be calibrated. Treated as a constant target, no calculation is performed using the projected data. The gradient being returned.

[0050] Step S600: Jointly optimize model parameters and structural parameters.

[0051] Overall objective function for computer system construction Overall objective function Includes observation data reconstruction error Logical loss function Potential calibration loss Acyclic constraint functions and for parameterized adjacency matrices The sparse regularization term. The computer system uses stochastic gradient descent or Adam optimization algorithms, based on the overall objective function. Synchronously update generator network parameters Encoder network parameters and parameterized adjacency matrix .

[0052] Step S700: Determine convergence and output the repair result.

[0053] Computer system determines overall objective function The system checks if the training has converged or reached the preset number of training rounds. If it has not converged, the computer system returns to step S300 to continue iterating. If it has converged, the computer system outputs the finally learned causal adjacency matrix. As a result of the data causal relationship analysis, and utilizing the trained generator network The manifold-constrained projection process outputs the final high-quality repair dataset. .

[0054] Step S110: Construct the observation data matrix and the state mask matrix.

[0055] The computer system receives the raw observation dataset. Original observation dataset Stored as a A two-dimensional real-valued matrix, where Indicates the number of samples. This indicates the number of feature dimensions. The computer system processes the original observation dataset. Perform a full scan to identify the original observation dataset. Missing value identifiers. Missing value identifiers include NaN (Not a Number), null, or preset abnormal placeholder values ​​as defined in the computer floating-point standard. The computer system constructs a mask matrix based on the scan results. Mask matrix It is a dimension A binary matrix. The computer system sets the mask matrix according to the following rules. Element values ​​in:

[0056] Mask matrix It acts as a gradient gate in subsequent calculations. Mask matrix Ensure that the computer system only accumulates the error of valid observations when calculating the reconstruction error loss, and mask invalid gradients generated at the location of missing values.

[0057] Step S120: Generate the neural network initialization input matrix.

[0058] Since deep neural networks cannot directly process input data containing non-numeric (NaN) values, the computer system constructs an initial input matrix without null values. The computer system first calculates the raw observation dataset. Statistical mean of each feature dimension in the column Computer systems calculate statistical means. When only the mask matrix is ​​selected Valid observation samples with a value of 1 at the corresponding position are used in the calculation to eliminate the interference of missing values. Subsequently, the computer system uses a noisy mean-filling strategy to generate the initial input matrix. Initialize the input matrix. The elements in the calculation are as follows:

[0059] in, This indicates a value with a mean of 0 and a variance of 0. Random noise sampled from a normal distribution. Introducing random noise. The purpose is to break the initialization input matrix Numerical constancy in missing regions prevents the generator network from getting stuck in local minima or producing deterministic output collapse in the early stages of training due to a single input.

[0060] Step S130: Synchronization and standardization of data and constraints.

[0061] To accelerate the convergence of the gradient descent algorithm and to unify the weight contributions of features with different physical dimensions in the logistic loss function, the computer system initializes the input matrix. Perform Z-score normalization. The computer system calculates each feature dimension. Standard deviation (Based on valid observation samples only). The computer system uses the following formula to map the data to a distribution space with zero mean and unit variance, thus obtaining standardized data. :

[0062] in, It is a very small constant used to prevent division by zero errors.

[0063] Importantly, in order to maintain the consistency of domain knowledge before and after the feature space transformation, the computer system synchronously modulates the domain constraint set. The numerical constants in the domain are standardized. Involving the first Feature variables With constant Comparison relationships (e.g.) The computer system will use constants Convert to standardized constants :

[0064] In the subsequent logical loss calculation step, the computer system uses standardized data. With standardization constant The calculations are performed to ensure that the mathematical expression of the logical rules matches the numerical distribution of the input data.

[0065] Step S210: Parsing of domain constraint rules and standardization of atomic predicates. The computer system reads the predefined set of domain constraints. Domain constraint set This includes business rules, functional dependencies (FDs), and conditional functional dependencies (CFDs). The computer system uses this set of domain constraints. Each rule in the algorithm is converted into a first-order logical expression. A first-order logical expression is composed of atomic predicates and logical connectives. To accommodate subsequent numerical calculations, the computer system uniformly converts all atomic predicates into standard inequality forms. Or standard equation form For example, for the rule "If the age is greater than 18, then the income must be greater than 0", the computer system will parse it into a logical implication: In this step, the computer system extracts two standardized atomic predicate functions: as well as .

[0066] Step S220: Continuous relaxation mapping of atomic predicates.

[0067] Because Boolean logic values ​​are discrete and non-differentiable, they cannot be directly used for gradient-based parameter optimization. Computer systems require specific steps... The atomic predicates extracted in S210 are subjected to a continuous relaxation operation. The computer system then applies this relaxation operation to the currently generated preliminary repair data X. Each sample vector in Calculate the continuous truth values ​​of atomic predicates. Continuous truth values ​​are mapped to real numbers in the interval [0,1]. The value, a continuous truth value, represents the confidence level that the predicate is satisfied.

[0068] For the form Inequality constraints are used by computer systems to construct continuous truth functions using the Sigmoid function. :

[0069] in, This refers to temperature parameters. Used to adjust the steepness of the gradient, larger ones The value makes the function approximate the step function, the smaller the value. The value allows the function to have a wider range of gradient responses near zero.

[0070] For the form Based on equality constraints, the computer system uses radial basis functions (RBF) to construct continuous truth functions. :

[0071] in, For bandwidth parameters, bandwidth parameters Numerical tolerance range used to control equality constraints. Step S230: Compilation of composite operations based on T-modular fuzzy logic. The computer system, based on the Product T-norm logic system, uses algebraic operations to compose the continuous truth values ​​of atomic predicates into the overall satisfaction of complex logical rules. For two logical propositions... and The computer system calculates logical propositions respectively. and logical propositions Continuous truth values and And calculate the truth value of the composite logic according to the following algebraic rules: 1. Logical AND operation: Computer systems calculate the product as the truth value of the logical AND operation.

[0072] 2. Logical OR operation: Computer systems calculate the truth value of logical OR based on probability and formulas.

[0073] 3. Logical NOT operation: Computer systems calculate the complement of a number as the truth value of logical NOT.

[0074] 4. Logical Implication Operation: Computer systems calculate the truth value of logical implication based on Reichenbach's implied meaning formula.

[0075] The gradient property of the Reichenbach implied formula is that the overall truth value approaches 0 only when the truth value of the antecedent A approaches 1 and the truth value of the consequent B approaches 0, thereby generating an effective gradient signal to reduce v(A) or increase v(B).

[0076] Step S240: Aggregation calculation of batch logic violation loss function. The computer system will aggregate the domain constraint set. In Applying the above calculation logic to each rule, we obtain the first rule. Rule No. Rule satisfaction on a sample ,in , N represents the batch size of the data. To drive the generative model to satisfy all constraints, the computer system calculates the overall logic violation loss. Overall logical violation losses Defined as the average or weighted sum of the negative log-likelihoods of all rules across all samples:

[0077] in, For the first The weight coefficient of each rule, To prevent logarithmic overflow at the minimum value, the overall logic violation penalty is minimized. The computer system uses the backpropagation algorithm to synchronously update the generator network parameters and causal structure parameters.

[0078] Step S310: Construct and initialize a differentiable parameterized adjacency matrix. To transform the discrete causal graph structure search into a continuous optimization problem, the computer system constructs a real-valued parameterized adjacency matrix. ,in The number of dimensions for data features. Parameterized adjacency matrix. elements in Indicates the first The characteristic variable for the th... The strength of the direct causal influence of each feature variable. Parameterized adjacency matrix. The adjacency matrix can take any real value; positive values ​​represent a positive promoting effect, and negative values ​​represent a negative inhibiting effect. To prevent gradient instability in the early stages of optimization and to start the search from a sparse structure, the computer system parameterizes the adjacency matrix. Initialize as a matrix of all zeros or a matrix with a mean of 0 and minimal variance (e.g., 10). Furthermore, to eliminate the self-loop effects of variables, the computer system enforces the parameterized adjacency matrix. The diagonal elements are fixed at zero. After each parameter update step, the computer system performs a diagonal masking operation:

[0079] in, is the identity matrix, 1 is an all-one matrix, and © denotes element-wise product.

[0080] Step S320: Construct a differentiable directed acyclic graph (DAG) constraint function. The causal structure must satisfy the acyclicity assumption. The computer system constructs an acyclic constraint function based on the trace of the matrix exponential. Used for quantizing parameterized adjacency matrices The degree of existence of cycles in the represented graph structure. Due to the parameterized adjacency matrix It may contain negative values; the computer system first calculates the parameterized adjacency matrix. Hadamard square (i.e., element) The matrix is ​​composed of these elements to ensure that the weights of all edges in the graph are non-negative. This process guarantees the matrix exponentiation operation. The first in Each element is strictly equal to the element from the node. Departure process Step back to node The sum of path weights. Acyclic constraint function. The specific calculation formula is as follows:

[0081] Where tr(·) represents the trace of the matrix, The matrix exponent. The parameterized adjacency matrix. When the corresponding graph does not contain any cyclic paths, the acyclic constraint function The value is equal to 0; otherwise, the acyclic constraint function... The value is strictly greater than 0. Acyclic constraint function. For parameterized adjacency matrix It is differentiable everywhere and supports gradient calculation via backpropagation algorithm.

[0082] Step S330: Series approximation and numerical calculation of constraint functions.

[0083] Computer systems calculate matrix exponents using Taylor series expansion. To balance computational efficiency and loop detection capability, the computer system uses the first k terms of the series for truncation approximation:

[0084] Here, the value of k is set to an integer not less than the feature dimension d. This approximate calculation method ensures that the computer system can detect all potential loops with a length not exceeding k. In a practical implementation, the computer system can also call the scaling and squaring algorithm built into the deep learning framework to calculate the matrix exponent for higher numerical accuracy.

[0085] Step S340: Definition of sparsity regularization.

[0086] To obtain a concise and interpretable causal graph structure, the computer system defines sparse regularization terms. Sparse regularization terms use Norm calculation:

[0087] Sparse regularization terms It is added to the overall objective function as a penalty term. Because The norm has the cusp property of being non-differentiable at zero, and the sparse regularization term... The optimization process can promote the parameterization of the adjacency matrix. Elements with smaller absolute values ​​converge to strict zero values, thus achieving automatic pruning of the causal structure.

[0088] Step S350: Construct the Augmented Lagrangian objective function. To strictly enforce directed acyclic graph constraints during neural network training, the computer system uses the Augmented Lagrangian method to transform the acyclic constraint function... This is transformed into a penalty term in an unconstrained optimization problem. The computer system constructs an augmented Lagrangian objective function. Augmented Lagrangian objective function Combined with the overall loss function (Including reconstruction error, logic violation loss, and sparse regularization term) as well as linear Lagrange multipliers and quadratic penalty terms for acyclic constraints. Augmented Lagrange objective function. The calculation formula is as follows:

[0089] in, Represents the Lagrange multipliers. This represents the penalty parameter. To balance the initial structural exploration and data fitting in optimization, the computer system uses Lagrange multipliers. Initialize to 0, and set the penalty parameter Initialize to a small positive number (e.g.) Lagrange multipliers Cost and penalty parameter used for linear approximation constraint terms Used to control the weight of the secondary penalty term.

[0090] Step S360: Perform inner subproblem optimization (PrimalStep). The computer system employs a two-level alternating optimization strategy. In the inner loop, the computer system fixes the Lagrange multipliers. and penalty parameters With the value remaining unchanged, the Lagrange objective function will be augmented. Consider it as a parameterized adjacency matrix Generator network parameters and encoder network parameters The computer system uses the Adam optimizer or stochastic gradient descent (SGD) optimizer to minimize the augmented Lagrangian objective function. The computer system sets the termination condition for the inner loop as: reaching a preset number of iterations (e.g., 100 times) or an augmented Lagrange objective function. The decrease in magnitude over several consecutive iterations is less than a preset threshold. During backpropagation, the computer system uses an automatic differentiation mechanism to calculate the augmented Lagrangian objective function. Relative to parameterized adjacency matrix gradient:

[0091] In the gradient formula The terms are obtained by calculating the Fréchet derivative of the matrix exponent or by an automatic differentiation engine, and the gradient is obtained by... It indicates the direction of parameter updates to reduce the number of loops in the graph structure.

[0092] Step S370: Dynamic Update of Lagrange Multipliers and Penalty Parameters (DualStep). Once the inner loop meets the termination condition, the computer system enters the outer loop and updates the Lagrange multipliers based on the current degree of violation of the acyclic constraint. and penalty parameters The computer system first calculates the current iteration round. acyclic constraint function value The computer system updates the Lagrange multipliers using the dual ascent rule.

[0093]

[0094] Subsequently, the computer system updates the penalty parameters based on the improvement of the constraints.

[0095]

[0096] in, For growth factors (e.g.) , For tolerance factors (e.g.) To prevent penalty parameters Excessive values ​​can lead to numerical overflow or gradient explosion. Therefore, computer systems set an upper limit for the penalty parameter. (e.g. 10) When calculated Exceeding the upper limit of the penalty parameter At that time, the computer system will Cut off as .

[0097] Step S380: Discretization and post-processing of the cause-effect graph. When the global convergence condition (i.e., the acyclic constraint function value) is satisfied... Less than tolerance And the overall loss function After stabilization, the computer system outputs a parameterized adjacency matrix with continuous values. To obtain the final discrete causal graph, the computer system performs a threshold truncation operation. The computer system sets a binarization threshold. (For example For parameterized adjacency matrices Each element in If element The absolute value is less than the binarization threshold. The computer system will use elements Set to 0; otherwise, the computer system reserves the element. The numerical value and elements The corresponding directed edges are marked as existing. The threshold truncation operation is used to eliminate the small numerical noise generated during continuous optimization, ensuring that the causal structure of the final output meets the sparsity requirement.

[0098] Step S410: Construct a graph-topology-based structured generator network architecture. The computer system constructs a structured generator network. Structured generator networks Configured to be based on parameterized adjacency matrix The defined directed acyclic graph structure maps noise-filled initial data to complete reconstructed data. (Computer system configuration structured generator network) It consists of an input layer, a structure aggregation layer, several nonlinear hidden layers, and an output layer. (Structured Generator Network) Receive initialization input matrix (or a sampled subset thereof) is used as input data, and a parameterized adjacency matrix is ​​received. As a structural control parameter. To accommodate the nonlinear relationship learning of high-dimensional features, the computer system adjusts the dimension of the hidden layer. Set as feature dimension Multiples of (e.g.) This allows features to be mapped to a high-dimensional latent space for interaction.

[0099] Step S420: Perform structure-guided linear feature aggregation. The computer system in the structured generator network... The first-level execution structure guides the feature aggregation operation, which aims to enforce the establishment of dependency paths between variables. The computer system calculates the aggregated feature matrix through matrix multiplication.

[0100]

[0101] in, for The matrix, for The matrix. The calculation steps implement the weighted summation logic: for the aggregated feature matrix... The first in Column vectors are generated from the initial input matrix. All column vectors in the parameterized adjacency matrix A are based on the parameterized adjacency matrix A. The column weight coefficients are obtained by linear combination. Since the diagonal of the parameterized adjacency matrix A has been set to zero, the aggregated characteristic matrix... It only contains information from the parent node, not information about the node itself.

[0102] Step S430: Multi-layer nonlinear transformation and residual reconstruction. The computer system will aggregate the feature matrix. Input to include parameters In a multilayer sensor network (MLP), the multilayer sensor network consists of... It consists of 1 fully connected layer. For the 1st Hidden layers ( The computer system performs the following transformation:

[0103] in, and The first Layer weight matrix and bias vector, This is a non-linear activation function (e.g., Sigmoid or LeakyReLU). In the output layer of the multilayer perceptron sub-network, the computer system does not use a non-linear activation function and directly outputs the transformed feature increments. The computer system connects the transformed feature increments to the initialized input matrix through residual connections. Adding them together yields the final reconstructed data matrix. :

[0104] The residual connection structure enables structured generator networks The focus is on learning the amount of missing values ​​to fill in and the amount of correction to noisy initial values, rather than refitting the overall distribution benchmark of the data.

[0105] Step S440: Calculate the masked reconstruction error loss. This is necessary for training the structured generator network. To accurately recover observation data, the computer system calculates the reconstruction error loss. The computer system uses a mask matrix M to construct an element-selection filter, ensuring that only valid observations participate in the loss calculation. The computer system calculates the reconstruction error loss based on the mean squared error (MSE) criterion.

[0106]

[0107] Where o represents the Hadamard product, This represents the Frobenius norm. The mask matrix M blocks the gradient signal generated at missing value locations (i.e., initialization noise locations), preventing errors from invalid data from misleading the network parameters. And the optimization direction of the parameterized adjacency matrix A.

[0108] Step S450: Optimization and update of generator network parameters. During the inner loop phase of the training process, the computer system calculates the reconstruction error loss based on the Automatic Differentiation algorithm. Relative to network parameters partial derivative gradient The computer system utilizes the Adam optimizer or the RMSProp optimizer, based on the partial derivative gradient. Update network parameters In this step, the parameterized adjacency matrix A is treated as a constant and does not participate in the update; the parameterized adjacency matrix A is updated only in a separate step through the gradient of the augmented Lagrangian objective function, thus achieving decoupled alternation between structure learning and parameter learning.

[0109] Step S510: Construct a tuple-level unary logistic loss function. The computer system first processes the set of domain constraints. Unary constraint rules in computer systems. Unary constraint rules are validation rules that apply only to a single row of data records. They include numerical range constraints, enumerated value constraints, and format constraints. The computer system identifies the set of unary constraints. For a set of univariate constraints Each rule in The continuous mapping function defined in step S220 of the computer system application, and the calculation rules. The system calculates the true value for each row of samples. The computer system then calculates the sum of the negative log-likelihoods of all samples violating the univariate constraints, yielding the univariate logistic loss. :

[0110] in, Represents the reconstruction of the data matrix The row vectors This represents the number of samples in the current training batch. To prevent the numerical stability constant of the logarithmic function from being zero (e.g., Univariate logic loss Used to penalize the reconstruction of the data matrix Individual attribute values ​​that exceed the predefined range or violate format requirements.

[0111] Step S520: Construct a dataset-level binary logistic loss function. Computer systems process sets of binary constraints involving multi-row data dependencies. This mainly includes functional dependencies (FDs) and conditional functional dependencies (CFDs). This is necessary to compute the reconstructed data matrix. Does it satisfy the form "attribute set A determines attribute set B"? To meet the dependency rules, the computer system needs to construct sample pairs for comparison. The computer system generates a randomly indexed vector p and uses it to construct a shuffled reconstructed data matrix. The computer system will reconstruct the original data matrix. The OK Reconstructing data matrices by shuffling the order The OK Yes. For each dependency rule Computer system calculation antecedent attribute set and consequent attribute set The similarity between sample pairs is compared. To eliminate the influence of different attribute value ranges on distance calculation, the computer system pre-normalizes all attribute columns (mapping them to the [0,1] interval), or performs normalization processing on each attribute beforehand. Configure independent bandwidth parameters The computer system uses the Gaussian kernel function to calculate similarity;

[0112]

[0113] For conditional functional dependencies (CFDs), the computer system treats them as functional dependencies with an additional conditional antecedent C', that is, by expanding the antecedent attribute set to... Subsequently, the computer system uses the Reichenbach fuzzy implication operator to compute the satisfaction probability of the dependency rule f. :

[0114] According to the properties of the Reichenbach operator, the comparison sample pair is only considered if they are highly similar on the antecedent attribute set A. However, the differences are significant on the consequent attribute set B. At that time, the probability of satisfaction Approaching 0 results in a large gradient penalty value; otherwise, the satisfaction probability... Approaching 1.

[0115] Computer system calculates binary logic loss

[0116]

[0117] Step S530: Calculate the overall logical consistency loss. The computer system aggregates the unary logical loss using a weighted summation method. and binary logic loss The overall logical consistency loss is obtained.

[0118]

[0119] in, and These are preset weighting coefficients. Overall logical consistency loss. As a differentiable regularization term added to the objective function of model training, it minimizes the overall logistic consistency loss. Computer system driver generator network Output reconstructed data that conforms to domain knowledge constraints, and correct logical inconsistencies that may arise from filling data based solely on statistical patterns.

[0120] Step S610: Construct the computation graph and backpropagation path for the structural parameters. This is to update the parameterized adjacency matrix. To optimize the causal structure, the computer system constructs a parameterized adjacency matrix during the backpropagation phase of the neural network. The gradient calculation path. Due to the parameterized adjacency matrix. Participated in Structured Generator Network First-level aggregated feature matrix The calculation (i.e.) Any structured generator network The loss function calculated at the output is applied to the parameterized adjacency matrix. Differentiable. The computer system is configured with an automatic differentiation engine, enabling it to track the process from the overall loss function to the parameterized adjacency matrix. The gradient flow. The overall loss function to the parameterized adjacency matrix. The gradient flow must pass through the reconstruction error loss. Loss of overall logical consistency After reconstructing the data matrix The data is returned via a structured generator network. The nonlinear hidden layer eventually reaches the parameterized adjacency matrix. .

[0121] Step S620: Calculate the gradient components of each objective function with respect to the structural parameters. The computer system calculates the gradient components of the three objective functions with respect to the parameterized adjacency matrix. The gradient components. First, the computer system calculates the overall logical consistency loss. The gradient. The computer system first calculates the overall logical consistency loss. Compared to reconstructing the data matrix partial derivative matrix Subsequently, the computer system uses the chain rule to convert the partial derivative matrix... Backpropagation to the first layer aggregated feature matrix , thus obtaining the intermediate gradient matrix Finally, the computer system calculates the adjacency matrix relative to the parameterized adjacency matrix according to the differential rules of matrix multiplication. Logistic gradient components

[0122]

[0123] The formula shows that the logical gradient components It is determined by the initialization input matrix With the returned logic error signal The inner product determines how the connection weights between features are adjusted to reduce logical violations in the output data. Similarly, the computer system calculates the reconstruction error loss. Relative to parameterized adjacency matrix Reconstructed gradient components Furthermore, the computer system directly calculates the acyclic constraint gradient components based on the acyclic constraint term in the augmented Lagrangian objective function.

[0124]

[0125] Step S630: Weighted aggregation of gradients and parameter update execution.

[0126] The computer system aggregates the above gradient components and adds the gradient of the sparse regularization term to obtain the overall update gradient G_total of the parameterized adjacency matrix A:

[0127] in, and These are the preset weight hyperparameters. For the L_1 sparse regularization term ||A||_1, since the L_1 norm is not differentiable at zero, the computer system adopts a sub-gradient strategy, using the sign function sign(A) to calculate the gradient direction; when the element value is 0, the computer system sets the corresponding gradient component to 0 or takes a value in the interval [-1,1].

[0128] The computer system uses momentum-based optimization algorithms (such as Adam or SGD-Momentum) to update the parameterized adjacency matrix A according to the overall update gradient G_total.

[0129] in, is the learning rate. Through multiple iterations, the values ​​of the elements in the parameterized adjacency matrix A will be adjusted so that the generated graph structure simultaneously satisfies the data fitting accuracy, logical rule consistency, and topological constraints of a directed acyclic graph.

[0130] Step S710: Construct the constrained optimization problem for the inference phase. In the structured generator network... Complete training and output the initial reconstructed data matrix. Next, the computer system performs a manifold-constrained projection step. The computer system models the final stage of data repair as a constrained optimization problem, aiming to find a set of neighborhood constraints while maintaining fidelity to the original observation data. The optimal corrected data matrix The computer system defines the optimization objective as minimizing the optimal corrected data matrix. The Euclidean distance between them must satisfy both logical constraints. The mathematical expression of the optimization problem is as follows:

[0131] st

[0132] And M

[0133] in, To optimize the variable matrix, The allowed logical tolerance threshold, This is a binary mask matrix (1 for observed values ​​and 0 for missing values). The input matrix contains the observations, and © denotes the Hadamard product (element-by-element product). Constraints. This ensures that all original observed valid data values ​​remain constant during the optimization process, with only the filling data at missing value locations being allowed to be numerically adjusted.

[0134] Step S720: Perform gradient descent iterations under mask protection. Due to the logistic loss function Typically non-convex, the computer system uses the Projected Gradient Descent algorithm to solve for the optimal corrected data matrix. .

[0135] Computer system freezes structured generator network All parameters will be used to initially reconstruct the data matrix. As the initial value of the optimization variable Z .

[0136] The computer system executes an iterative update cycle. In the... In this iteration, the computer system calculates the overall logical consistency loss. Relative to the current optimization variable gradient matrix The computer system uses the gradient matrix... Update the values ​​at the locations of missing values, and force a reset of the values ​​at the locations of observed values. The update formula is as follows:

[0137] in, This represents the learning rate during the inference phase, where 1 represents an all-one matrix. Multiplying by... In this step, the computer system forces the gradient component corresponding to the observation location to zero, thus mathematically ensuring that the observed data is not affected by the backpropagation of the logistic loss. The computer system stops iterating when either of the following termination conditions is met: the number of iterations reaches a preset upper limit (e.g., 50 times), or the overall logistic consistency loss is reached. It is less than the preset convergence threshold.

[0138] Step S730: Boundary truncation and discretization post-processing.

[0139] After the gradient descent iterations are completed, the computer system optimizes the variables. Perform numerical truncation and discretization operations to obtain the final repair result.

[0140] For continuous variable attributes If a predefined range of values ​​exists The computer system performs a clipping operation:

[0141] The truncation operation ensures that the final repaired data strictly falls within the legal numerical range, eliminating the tiny numerical drift that gradient updates may introduce.

[0142] For discrete categorical variables (typically represented in a network as a One-Hot code or probability distribution vector), the computer system performs a maximum probability selection operation (Argmax):

[0143] in, Indicates the first The sample in the classification attribute of the _th The computer system assigns the probability value or Logits value to each category. The final determined repair category value is used to map the continuous optimization results back to the discrete symbol space, thus completing the final data repair process.

[0144] Step S810: Construct the logical correction target matrix and gradient blocking for the training phase.

[0145] To encode the logical constraint solving capability of the inference phase into the structured generator network In the weight parameters, the computer system constructs the logically corrected target matrix in the inner loop of model training.

[0146] When processing the current training batch, the computer system first utilizes a structured generator network. Generate a preliminary reconstructed data matrix Subsequently, the computer system reconstructed the data matrix. Starting from the initial point, perform the manifold constraint projection operation described in steps S710 to S730 to calculate the set of neighborhood constraints. The optimal corrected data matrix During this process, the computer system optimizes the data matrix. Perform a gradient-stopping operation. Gradient-stopping refers to cutting off the optimal correction data matrix in the computational graph. Relative to network parameters The gradient backpropagation path. The computer system will optimize the data matrix. Treating it as a fixed target constant tensor, its value is only used for subsequent loss calculations, and gradients are not allowed to flow through the manifold constraint projection step.

[0147] Step S820: Calculate the potential uniform distillation loss.

[0148] Computer system construction potential consistency loss Used to measure structured generator networks Consistency between the direct output distribution and the compliant data distribution after logical correction.

[0149] The computer system calculates the initial reconstruction of the data matrix. With logically corrected target matrix Mean squared error (MSE) between:

[0150] in, For structured generator networks The output of the first row vectors. To logically correct the target matrix The first in Row vectors.

[0151] By introducing potential consistency loss The computer system implements an online knowledge distillation: logically correcting the target matrix. Acting as a "teacher signal" to guide the structured generator network (Student model) Adjust parameters This allows the network to directly output close to [value] in a single forward propagation. The result is that this effectively transforms non-differentiable discrete logic constraints into differentiable Euclidean distance monitoring signals.

[0152] Step S830: Perform joint backpropagation of multi-objective gradients and parameter updates. The computer system will then apply the potential consistency loss. This is integrated into the overall training objective function to construct a closed-loop optimization mechanism. The computer system calculates the total loss function.

[0153]

[0154] in, and These are preset hyperparameter weights. The computer system calculates the total loss function based on an automatic differentiation algorithm. Relative to network parameters The gradient of the mixed gradient. In this mixed gradient: reconstruction error loss. The driving network fits the statistical regularity of the data; overall logical consistency loss. Provides soft penalties for obvious violations; potential loss of consistency. It provides precise correction directions to the projection points of the compliant manifold. The computer system uses an optimizer to update the structured generator network based on gradients. The parameters. As training iterations proceed, the structured generator network... The generated preliminary reconstructed data matrix It will gradually approach the target matrix of logical correction. This allows the network to directly generate high-quality data that conforms to domain knowledge constraints when the model converges, reducing the reliance on post-processing during the inference phase.

[0155] Step S910: Construct the augmented Lagrange overall training objective.

[0156] The computer system uses the Augmented Lagrangian Method to transform the equality constraints of a directed acyclic graph into... It is transformed into a penalty term in the unconstrained optimization objective.

[0157] Computer system defines overall objective function objective function It consists of the main task loss term, the sparse regularization term, and the acyclic constraint penalty term:

[0158] in, For parameterized adjacency matrix, For structured generator networks The weight parameters, To induce sparsity Norm, For Lagrange multipliers, This is a secondary penalty parameter. The computer system will use the main task loss term. Defined as:

[0159] In this formula, To compensate for reconstruction error loss, For the loss of overall logical consistency, This represents potential consistency loss. It is achieved by minimizing the overall objective function. During the optimization process, the computer system simultaneously seeks solutions that satisfy the following conditions: the data reconstruction error is minimized, the degree of violation of logical rules is minimized, and the graph structure corresponding to the parameterized adjacency matrix A satisfies the property of a directed acyclic graph.

[0160] Step S920: Perform hierarchical optimization and dynamic parameter adjustment.

[0161] The computer system employs a two-level loop optimization strategy to train the model.

[0162] In the inner loop, the computer system keeps the Lagrange multiplier α and the quadratic penalty parameter ρ constant. The system uses a gradient-based optimizer (e.g., Adam) based on the overall objective function. The parameterized adjacency matrix A and network weights θ are iteratively updated relative to the gradients of parameters A and θ. The inner loop continues to execute until a complete training epoch is completed.

[0163] In the outer loop, the computer system evaluates the current value of the acyclic constraint function h(A) at the end of each training round.

[0164] The computer system sets a convergence threshold γ (e.g., γ=0.25). The computer system then determines the current value of the acyclic constraint function. Does the descent condition meet?

[0165] If the condition is met, it indicates that the descent rate of the acyclic constraint has not reached the expected level. The computer system performs a penalty enhancement operation, increasing the quadratic penalty parameter ρ:

[0166] Where μ is the growth rate (e.g., μ=10). Increasing the quadratic penalty parameter ρ will lead to changes in the overall objective function. The gradient components that violate the DAG constraints increase sharply, thus forcing the parameterized adjacency matrix A to evolve into an acyclic structure more quickly in the next inner loop.

[0167] If the condition is not met, it indicates that the acyclic constraint is converging effectively. The computer system keeps the quadratic penalty parameter ρ constant and updates the Lagrange multiplier α:

[0168] Step S930: Hyperparameter Configuration and Cold Start Protection Strategy. The computer system initializes its hyperparameter configuration. For sparse regularized weights... Computer systems based on feature dimensions Set a value (e.g.) This is to suppress noisy edges while preserving key causal edges.

[0169] To prevent instability during the initial training phase, the computer system implements a cold start protection strategy. Before training begins... Rounds (e.g.) Internally, the computer system forces the logical loss weights to be adjusted. And consistency loss weight Set to 0. At this stage, the computer system relies solely on the reconstruction error loss. Optimize the network to make the structured generator network First, learn the basic statistical manifolds of the observed data.

[0170] When the number of training rounds exceeds Then, the computer system gradually increases the logical loss weights using either a linear or step-by-step growth method. And consistency loss weight The target value is raised to the preset value. This strategy avoids optimization divergence or mode collapse caused by the conflict between the directions of the logical constraint gradient and the reconstruction gradient during the random initialization phase of network parameters.

[0171] Step S940: Perform convergence determination based on dual criteria.

[0172] The computer system simultaneously monitors the degree to which structural constraints are satisfied and the accuracy of data fitting during the training loop. The system employs a dual convergence criterion to determine whether to terminate training.

[0173] The first criterion concerns the satisfaction of constraints in a directed acyclic graph. The computer system calculates the current acyclic constraint function value. If there are no cyclic constraints, the function value... Below the preset numerical tolerance threshold (For example The computer system determines that the parameterized adjacency matrix A has converged to an acyclic state.

[0174] The second criterion concerns the stability of task losses. The computer system calculates the main task loss term. The moving average on the validation set. If the main task loss term on the validation set... In continuous training rounds (e.g.) The relative rate of change within is less than the preset convergence threshold. (e.g. 10) Computer systems determine structured generator networks The parameters have reached a stable state.

[0175] The computer system stops updating parameters and saves the final parameterized adjacency matrix A and the structured generator network only if both of the above criteria are met simultaneously. The weight parameters. If the training epochs reach the preset upper limit and the acyclic constraint function value is... The training has not yet converged, and the computer system outputs an error signal indicating that the training is diverging.

[0176] Step S950: Discretization extraction of causal structure. Parameterized adjacency matrix at the end of training. It contains continuous floating-point values, where tiny non-zero values ​​are usually caused by computational noise and do not represent true causal relationships. To obtain a sparse and deterministic causal graph structure, the computer system performs a hard thresholding operation. The computer system sets an edge detection threshold. (For example The computer system constructs the final binary adjacency matrix according to the following rules.

[0177]

[0178] in, This is an indicator function that takes a value of 1 when the condition is met and 0 otherwise. The final binary adjacency matrix... The directed edge connections between feature variables are directly defined. The computer system binarizes the adjacency matrix. The output is the mined causal structure, which reveals the variable dependency paths in the data generation process.

[0179] Step S960: Fast inference and data synthesis based on forward propagation.

[0180] Upon receiving a data matrix containing missing values ​​to be processed At that time, the computer system loads the trained structured generator network. And structured generator network Switch to Inference Mode, where all Dropout layers are disabled and the Batch Normalization layer uses moving average statistics. The computer system first processes the data matrix to be processed. Perform initialization filling. The computer system will then process the data matrix. The elements at the positions of missing values ​​are set to 0 or the mean of the corresponding dimension to obtain the initial input matrix. This step ensures the structured generator network. It can perform valid matrix multiplication operations.

[0181] The computer system will initialize the input matrix. Input to structured generator network After forward propagation calculation, the reconstructed query matrix is ​​output. .

[0182] The computer system performs the final data synthesis operation to generate the complete data completion result. The computer system utilizes the matrix of data to be processed. The corresponding binary mask matrix (Observations are 1, missing values ​​are 0), the original observations are fused with the reconstructed values ​​generated by the network:

[0183] Through the synthesis formula, the computer system ensures that the original observation records in the data are completely unmodified, with only missing locations being modified by the structured generator network. The inferred values ​​are used to fill in the gaps. Finally, the computer system processes the data completion result X. Perform the numerical range truncation and discretization processing described in 5730 to output the final compliant repair data.

[0184] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A data quality assessment and repair method based on machine learning, characterized in that, The method includes: The original input matrix containing missing values ​​and observations is preprocessed to generate an initial input matrix and a corresponding binary mask matrix. Construct a structured generator network, which includes a parameterized adjacency matrix for characterizing causal relationships between variables; Execute the model training loop, performing the following steps in each iteration of the training loop: The structured generator network is used to perform forward propagation on the initial input matrix to output a preliminary reconstructed data matrix; Perform the manifold constraint projection step, using the preliminary reconstructed data matrix as the initial value, and while keeping the structured generator network parameters fixed, find the logically corrected target matrix that satisfies the domain constraint set through iterative optimization; Construct a potential consistency learning loop and calculate the potential consistency loss, which is used to measure the consistency between the initial reconstructed data matrix and the logically corrected target matrix. An augmented Lagrange overall training objective is constructed, which integrates the reconstruction error loss, the overall logical consistency loss, the potential consistency loss, and the acyclic constraint penalty term for the parameterized adjacency matrix; The weight parameters of the structured generator network and the parameterized adjacency matrix are updated based on the overall training objective. When the preset convergence condition is met, the trained structured generator network performs the final inference on the data to be processed and outputs the repaired complete data.

2. The data quality assessment and repair method based on machine learning according to claim 1, characterized in that, When performing forward propagation, the structured generator network first uses the parameterized adjacency matrix to perform linear transformation and feature aggregation on the initial input matrix to simulate the direct causal effect between variables; then the aggregated features are input to the nonlinear mapping layer to fit the complex data distribution. The parameterized adjacency matrix is ​​a learnable parameter matrix jointly optimized with the weight parameters of the structured generator network, used to explicitly model the directed weighted connectivity between data feature dimensions.

3. The data quality assessment and repair method based on machine learning according to claim 1, characterized in that, The step of performing manifold constraint projection specifically includes: Construct a constrained optimization problem for the inference phase, which aims to minimize the Euclidean distance to the initially reconstructed data matrix and is constrained by the set of domain constraints. Perform mask-protected projection gradient descent iteration: calculate the gradient of the logical loss relative to the current data matrix, use this gradient to update only the values ​​at the missing value positions, and at the same time, perform a Hadamard product operation with the inverse of the binary mask matrix to forcibly reset the values ​​at the observation positions, so that they remain as the original observation data. After gradient update, perform analytical projection and truncation operations to ensure that the updated values ​​fall within a predefined range, thus obtaining the logically corrected target matrix.

4. The data quality assessment and repair method based on machine learning according to claim 1, characterized in that, The construction of the potential consistency learning loop specifically includes: A gradient blocking operation is performed on the logically modified target matrix, setting it as a fixed constant tensor that does not participate in gradient backpropagation; The mean square error between the initial reconstructed data matrix directly output by the structured generator network and the logically corrected target matrix after gradient blocking is calculated and used as the potential consistency loss. By minimizing the potential consistency loss, the structured generator network is driven to directly approximate the manifold space that satisfies the set of domain constraints in a single forward propagation, transforming the non-differentiable logical constraint solving process into a differentiable supervised learning process.

5. The data quality assessment and repair method based on machine learning according to claim 1, characterized in that, The construction of the augmented Lagrange overall training objective specifically employs the augmented Lagrange multiplier method, and the overall training objective includes: The main task loss term is composed of the weighted sum of the reconstruction error loss, the overall logical consistency loss, and the potential consistency loss. The sparse regularization term is the L1 norm of the parameterized adjacency matrix, used to induce the generation of a sparse causal graph structure. The acyclic constraint penalty term, consisting of Lagrange multiplier terms and quadratic penalty parameter terms, is used to transform the equality constraints of the directed acyclic graph into an unconstrained optimization objective. The method minimizes the overall training objective while finding a solution that minimizes data reconstruction error, minimizes the degree of violation of logical rules, and satisfies the property of a directed acyclic graph.

6. The data quality assessment and repair method based on machine learning according to claim 5, characterized in that, The step of updating the weight parameters of the structured generator network and the parameterized adjacency matrix based on the overall training objective specifically involves executing a two-layer loop optimization strategy: In the inner loop, the Lagrange multipliers and the second penalty parameters are kept constant, and the optimizer is used to iteratively update the parameters based on the gradient of the overall training objective with respect to the network parameters. In the outer loop, the decrease in the value of the acyclic constraint function is evaluated: if the decrease in the value of the acyclic constraint function does not reach the preset threshold, the quadratic penalty parameter is increased to enhance the penalty for violating the acyclic constraint; if the value of the acyclic constraint function decreases significantly, the quadratic penalty parameter is kept unchanged and the Lagrange multiplier is updated.

7. The data quality assessment and repair method based on machine learning according to claim 1, characterized in that, When the preset convergence condition is met, the specific steps include performing a convergence determination based on a dual criterion: The first criterion is structural constraint convergence: determining whether the acyclic constraint function value calculated by the parameterized adjacency matrix is ​​lower than a preset numerical tolerance threshold; The second criterion is task loss stability: determining whether the relative rate of change of the task's main loss term on the validation dataset is less than a preset convergence threshold; The model training is considered complete only if both of the above criteria are met simultaneously, and the current parameterized adjacency matrix and structured generator network parameters are saved.

8. The data quality assessment and repair method based on machine learning according to claim 1, characterized in that, Before performing final inference on the data to be processed based on the trained structured generator network, the method further includes: A hard threshold truncation operation is performed on the parameterized adjacency matrix at the end of training, setting elements whose absolute value is less than a preset edge determination threshold to zero, and constructing the final binary adjacency matrix. The binarized adjacency matrix is ​​output as the causal structure among the mined feature variables.

9. The data quality assessment and repair method based on machine learning according to claim 1, characterized in that, The structured generator network, after training, performs the final inference on the data to be processed, specifically including: Switch the structured generator network to inference mode and freeze the parameters; The missing locations in the data to be processed are initialized and filled, and the input network is used to perform forward propagation to obtain the reconstructed query matrix; Perform data synthesis operation, using the mask matrix corresponding to the data to be processed to fuse the original observation values ​​with the fill values ​​in the reconstruction query matrix generated by the network, to ensure that the original observation records are not modified; The fused data is truncated and discretized to output the final compliant data completion result.

10. A data quality assessment and repair system based on machine learning, characterized in that, The system includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 9.

Images