Training method of archive data reconstruction model, archive data anomaly detection method, device, equipment and medium

Abstract

The invention provides a training method of an archive data reconstruction model, an archive data anomaly detection method and device, equipment and a medium. The method comprises the steps of obtaining an archive record to be detected; vectorizing the archive record to be detected to obtain a corresponding input vector; inputting the input vector into an archive data reconstruction model obtained based on normal archive record unsupervised training to obtain a corresponding reconstruction vector; calculating a reconstruction error between the input vector and the reconstruction vector; and when the reconstruction error is greater than a preset error threshold value, determining that the to-be-detected archive record is abnormal data. Through the pre-trained archive data reconstruction model, the internal association and non-linear relationship of each field in the archive record can be identified, novel anomalies of unknown types and undefined rules are found, the limitation of traditional hard coding rules is broken through, and accurate detection of deep and hidden defects in archive data is realized.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to a training method for an archival data reconstruction model, an archival data anomaly detection method, apparatus, equipment, and medium. Background Technology

[0002] In the daily operations of enterprises and organizations, massive amounts of archival data are generated, such as financial vouchers, personnel files, purchase orders, and customer information. This data is typically entered manually or imported from various sources using automated tools like OCR, resulting in inconsistent data quality. Various errors are common, such as: typos during manual entry (e.g., adding or omitting a zero in the amount), OCR recognition errors, inconsistent unit entries (e.g., mixing "yuan" and "ten thousand yuan"), illogical outliers (e.g., an employee's age being 200 years old), and logical contradictions between fields (e.g., reimbursement amounts being much larger than invoice amounts). These erroneous data severely impact the accuracy of subsequent data analysis, report generation, and intelligent decision-making.

[0003] In existing technologies, a series of hard-coded rules are typically predefined to validate data, such as "the amount field must be a number," "age must be between 18 and 65," and "phone number must be 11 digits." While such rules can effectively intercept some explicit formatting errors or boundary violations, they are essentially static validations based on explicit business rules, which have significant limitations: on the one hand, the rules struggle to cover complex, dynamic, or context-dependent anomaly patterns; on the other hand, traditional rules cannot model and identify non-linear logical contradictions hidden in the joint distribution of multiple fields (such as "accommodation expenses for mid-level employees in first-tier cities typically do not exceed 12,000 yuan"). Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a training method for archival data reconstruction model, an archival data anomaly detection method, device, equipment and medium, so as to achieve accurate detection of deep-seated and hidden defects in archival data.

[0005] Firstly, a training method for an archival data reconstruction model is provided, including: Retrieve several verified normal archival records from the historical archive database; Each field in each normal record is vectorized to obtain an input vector, and all input vectors are used to form a training sample set. Based on the pre-built archival data reconstruction model, unsupervised iterative training is performed on the training sample set until the preset iteration stopping condition is reached; In each iteration of training, the non-linear dependencies between fields in normal archival records are learned and their joint distribution characteristics are implicitly modeled to output a reconstruction vector. The parameters of the archival data reconstruction model are then updated based on the reconstruction loss between the reconstruction vector and the input vector.

[0006] Optionally, the field types should include at least category fields and numeric fields; each field in each normal record is vectorized to obtain the input vector, which includes: Perform one-hot encoding on the categorical fields to obtain a categorical field vector; the dimension of the categorical field vector is the number of all categorical fields. Normalize the numeric fields to map the original values ​​to a preset range and obtain a numeric field vector; The input vector is obtained by concatenating all categorical field vectors with all numeric field vectors in a preset order.

[0007] Optionally, the archival data reconstruction model includes an encoder and a decoder. The encoder includes an input layer, a first hidden layer, and a second hidden layer, while the decoder includes a third hidden layer and an output layer. The input layer is used to receive normal archival records that have undergone vectorization as input vectors; The first hidden layer is used to perform preliminary compression and feature abstraction on the input vector through a non-linear activation function in order to capture local correlation patterns between fields; The second hidden layer serves as the bottleneck layer, used to generate a low-dimensional latent representation that implicitly encodes the global nonlinear dependencies and joint distribution characteristics between fields in a normal archival record. The third hidden layer is used to symmetrically expand the low-dimensional latent representation and gradually recover the high-dimensional feature structure through a non-linear activation function. The output layer is used to output a reconstruction vector with the same dimension as the input vector, which represents the optimal reconstruction result for the normal archival data distribution under the current model parameters.

[0008] Optionally, updating the parameters of the archival data reconstruction model based on the reconstruction loss between the reconstructed vector and the input vector includes: Calculate the reconstruction error between the input vector and the reconstructed vector in each field's corresponding dimension; The mean squared error between the input vector and the reconstructed vector is determined based on the reconstruction error in all dimensions, and the mean squared error is used to characterize the reconstruction loss. Based on the reconstruction loss, the network parameters of the archival data reconstruction model are adjusted through backpropagation to minimize the reconstruction loss.

[0009] Secondly, a method for detecting anomalies in archival data is provided, including: Obtain the archive records to be inspected; The archive records to be detected are vectorized to obtain the corresponding input vector; The input vector is fed into the archive data reconstruction model trained by any of the methods in the first aspect to obtain the corresponding reconstruction vector; Calculate the reconstruction error between the input vector and the reconstructed vector; If the reconstruction error exceeds the preset error threshold, the archive record to be inspected is determined to be abnormal data.

[0010] Optionally, the preset error threshold is determined in the following way: Obtain the reconstruction error distribution generated from normal archival records; Based on the reconstruction error distribution, the statistical characteristics of the reconstruction error distribution are determined; Determine the preset error threshold based on statistical characteristics; The statistical characteristics include any of the following: The error value corresponding to the preset high percentile is used in the reconstructed error distribution; Reconstruct the mean and standard deviation of the error distribution.

[0011] Optionally, after determining that the archive record to be detected is anomalous data, the method further includes: Calculate the reconstruction error between the input vector and the reconstructed vector in the corresponding dimensions of each original field, and identify at least one field whose reconstruction error is greater than a preset error threshold as an abnormal field position; Based on the location of the abnormal field, the vector of the corresponding field in the reconstructed vector is normalized inversely to generate a repair suggestion value with the same data format as the original field. In the data quality audit interface, the archive records corresponding to abnormal data are highlighted, and the original field values ​​and suggested repair values ​​are presented in a comparison chart.

[0012] Thirdly, a device for detecting anomalies in archival data is provided, comprising: The acquisition unit is used to acquire the archive records to be inspected; The vectorization processing unit is used to perform vectorization processing on the archive records to be detected to obtain the corresponding input vector; The reconstruction unit is used to input the input vector into the archive data reconstruction model trained by any of the methods in the first aspect, and obtain the corresponding reconstruction vector; The computation unit is used to calculate the reconstruction error between the input vector and the reconstructed vector. The determination unit is used to determine that the archive record to be detected is abnormal data when the reconstruction error is greater than a preset error threshold.

[0013] Fourthly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements either the first aspect or the second aspect of the method.

[0014] Fifthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the method of either the first aspect or the second aspect.

[0015] This invention provides a training method for an archival data reconstruction model, an archival data anomaly detection method, apparatus, equipment, and medium. The method involves acquiring the archival record to be detected; vectorizing the record to obtain a corresponding input vector; inputting the input vector into an archival data reconstruction model trained unsupervised on normal archival records to obtain a corresponding reconstruction vector; calculating the reconstruction error between the input vector and the reconstruction vector; and determining the archival record to be detected as anomalous data when the reconstruction error exceeds a preset error threshold. This invention, through a pre-trained archival data reconstruction model, can identify the inherent relationships and non-linear dependencies among fields in archival records, discover novel anomalies of unknown types and undefined rules, and overcome the limitations of traditional hard-coded rules, achieving accurate detection of deep-seated and hidden defects in archival data.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a training method for an archival data reconstruction model provided in an embodiment of the present invention is shown; Figure 2 A flowchart of another method for detecting archival data anomalies provided by an embodiment of the present invention is shown; Figure 3 This diagram illustrates the structure of an archival data anomaly detection device provided in an embodiment of the present invention. Figure 4A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] Considering the obvious limitations of static validation based on explicit business rules: on the one hand, the rules are difficult to cover complex, dynamic or context-dependent abnormal patterns; on the other hand, traditional rules cannot model and identify nonlinear logical contradictions hidden in the joint distribution of multiple fields (such as "the accommodation expenses of mid-level employees in first-tier cities usually do not exceed 12,000 yuan").

[0021] Based on this, embodiments of the present invention provide a method and apparatus for detecting anomalies in archival data, which are described below through embodiments.

[0022] This invention provides a training method for an archival data reconstruction model, such as... Figure 1 As shown, it includes the following steps: Step S101: Obtain several verified normal archive records from the historical archive database.

[0023] In this embodiment of the invention, the historical archive database refers to a collection of structured business data stored within an enterprise or organization, such as expense reports in a financial system, purchase orders in an ERP system, and employee files in an HR system.

[0024] Verified normal records refer to a subset of historical data that has passed manual review, business rule verification, or has been running for a long time without controversy, and its data quality is high and there are no obvious errors or anomalies.

[0025] For example, reimbursement forms with a status of "reimbursed" from the past three years can be selected as normal samples.

[0026] It should be noted that the quality of the training data directly determines the model's accuracy in learning normal paradigms. Therefore, it is important to ensure that the selected records are authentic, compliant, and representative as much as possible.

[0027] Step S102: Vectorize each field in each normal record to obtain input vectors, and use all input vectors to form a training sample set.

[0028] Continuing from the previous example, a reimbursement record may contain the following fields: {Department (Category), Employee Level (Category), Reimbursement Type (Category), Reimbursement Amount (Value), Number of Invoices (Value)}.

[0029] Vectorization aims to convert fields with different semantics and dimensions into computable numerical vectors, thereby constructing a high-quality training sample set.

[0030] Step S103: Based on the pre-built archive data reconstruction model, perform unsupervised iterative training on the training sample set until the preset iteration stopping condition is reached.

[0031] In each iteration of training, the non-linear dependencies between fields in normal archival records are learned and their joint distribution characteristics are implicitly modeled to output a reconstruction vector. The parameters of the archival data reconstruction model are then updated based on the reconstruction loss between the reconstruction vector and the input vector.

[0032] It can be explained that "pre-built" here refers to the pre-defined network structure of the model (such as the number of layers and activation functions), rather than the model having already been trained.

[0033] Unsupervised iterative training means that the entire process requires no label guidance and relies solely on the input data for self-supervised learning. Preset stopping conditions for iteration can be either that the reconstruction loss decreases by less than a tolerance threshold for several consecutive iterations, or that the maximum number of training iterations is reached.

[0034] The core of this step lies in the model's ability to gradually capture the business logic hidden within normal data by repeatedly reconstructing the input, and to remember the main patterns of normal data archives. For example, the accommodation fee for mid-level R&D personnel is typically between 8,000 and 12,000 yuan, thereby establishing an effective representation of the distribution of normal data.

[0035] After sufficient training, the model parameters converge, and the model can be saved for subsequent anomaly detection tasks. This model has a strong ability to generalize to normal patterns, laying the foundation for achieving high-precision, low-false-positive data quality control.

[0036] This embodiment employs unsupervised training using only verified, clean, and normal archival records. This allows the model to focus on learning real-world data patterns and reasonable relationships between fields within the business scenario, avoiding contamination of the normal paradigm by anomalous samples. Compared to training methods that introduce noise or erroneous samples, this approach significantly improves the model's accuracy in depicting the normal data manifold. Consequently, it exhibits higher sensitivity and discriminative power against anomalous data deviating from this manifold during the inference phase, fundamentally solving the problem that traditional rule-based methods cannot cover implicit and complex anomalies.

[0037] Based on the above implementation, the field types include at least category fields and numeric fields; each field in each normal record is vectorized to obtain the input vector, which includes: Step S102A: Perform one-hot encoding on the categorical field to obtain the categorical field vector.

[0038] The dimension of the categorical field vector is the number of all categorical fields. For example, if the department field has 5 possible values ​​(such as R&D Department, Marketing Department, Finance Department, Human Resources Department, Administration Department), then R&D Department will be encoded as a 5-dimensional vector [1,0,0,0,0].

[0039] The advantage of one-hot encoding is that it can eliminate spurious ordinal relationships between categories (such as avoiding the model from mistakenly identifying the marketing department as > the R&D department), while assigning independent feature dimensions to each category, making it easier for the model to learn its unique attributes.

[0040] Step S102B: Normalize the numeric fields to map the original values ​​to a preset range and obtain a numeric field vector.

[0041] In one feasible implementation, min-max normalization is used to scale the values ​​to the [0,1] range. For example, if the historical reimbursement amount ranges from 0 to 50,000 yuan, then 10,000 yuan will be mapped to 0.2.

[0042] This operation effectively eliminates the differences in units and orders of magnitude between different numerical fields. For example, the amount is in "yuan" while the quantity is only a single digit. This avoids the convergence difficulties caused by gradient imbalance during model training and significantly improves the training stability and efficiency of the neural network.

[0043] Step S102C: Concatenate all categorical field vectors and all numeric field vectors in a preset order to obtain the input vector.

[0044] Specifically, all the sub-vectors generated in steps S102A and S102B are concatenated end to end according to the field order (e.g., category first, then value) to form a high-dimensional numerical vector with a fixed dimension.

[0045] For example, the aforementioned expense reimbursement record can ultimately be represented as an 11-dimensional vector. This unified vector retains all the information from the original fields while also meeting the input format requirements of deep learning models, serving as a crucial bridge connecting business data and AI models.

[0046] It should be noted that, in addition to categorical and numerical fields, actual archive data may also include text fields (such as explanations of the matter and approval opinions) and time fields (such as creation time and reimbursement date).

[0047] For such fields, this invention also supports vectorization: for text fields, pre-trained language models (such as BERT, Sentence-BERT) can be used to extract fixed-length semantic vectors, or TF-IDF combined with PCA can be used for dimensionality reduction; for time fields, they can be decomposed into components such as year, month, day, and day of the week, and sine / cosine encoding can be used for periodic components (such as day of the week and month) to preserve their cyclic characteristics.

[0048] All of the above vectors can be concatenated with other field vectors and incorporated into a unified input vector, thereby expanding the applicability of this method to multimodal archival data.

[0049] This embodiment transforms heterogeneous archival data into a structured numerical representation through a standardized vectorization process. This not only meets the input requirements of deep learning models but also preserves the semantic and statistical characteristics of the original fields. More importantly, all preprocessing rules are set based on the distribution characteristics of clean and normal data, ensuring consistency between training and inference, avoiding the introduction of false anomalies due to preprocessing biases, and further enhancing the model's ability to learn real business patterns.

[0050] Based on the above embodiments, the archival data reconstruction model includes an encoder and a decoder. The encoder includes an input layer, a first hidden layer, and a second hidden layer, and the decoder includes a third hidden layer and an output layer; wherein: The input layer is used to receive normal archival records that have been vectorized as input vectors.

[0051] The number of neurons in this layer is equal to the dimension of the input vector, such as... It is responsible for importing data into the network.

[0052] The first hidden layer is used to perform preliminary compression and feature abstraction on the input vector through a non-linear activation function in order to capture local correlation patterns between fields.

[0053] In this layer, the number of neurons gradually decreases. In a specific example, the first hidden layer may contain 32 neurons and use the ReLU activation function to effectively extract local features such as the impact of the department + job level combination on the reimbursement type.

[0054] The second hidden layer, acting as a bottleneck layer, is used to generate a low-dimensional latent representation that implicitly encodes the global nonlinear dependencies and joint distribution characteristics between fields in a normal archival record.

[0055] This layer has the fewest neurons. For example, the bottleneck layer dimension can be set to 16, which is much smaller than the input dimension.

[0056] This information bottleneck forces the model to discard noise and redundant information, retaining only the most essential statistical regularities. It is this layer that enables the model to understand complex business rules such as "intermediate + accommodation fee ≈ 10,000".

[0057] The third hidden layer is used to symmetrically expand the low-dimensional latent representation and gradually recover the high-dimensional feature structure through a nonlinear activation function.

[0058] Its structure is usually symmetrical to the first hidden layer (e.g., also 32 neurons), and the original feature space is gradually reconstructed through reverse mapping.

[0059] The output layer is used to output a reconstruction vector with the same dimension as the input vector, which represents the optimal reconstruction result for the normal archival data distribution under the current model parameters.

[0060] In one feasible implementation, the output layer employs the Sigmoid activation function to match input data normalized to the [0,1] interval. The overall hourglass architecture design ensures that the model can both efficiently compress information and accurately reconstruct it, thus providing a reliable basis for anomaly detection.

[0061] This embodiment employs an hourglass-shaped autoencoder structure to construct an archival data reconstruction model. A bottleneck layer forces the model to learn a low-dimensional manifold of normal data, which is entirely defined by clean, normal samples. Since no anomalous patterns are encountered during training, the model does not consider errors as valid. Therefore, when faced with new data, it can reconstruct normal records with extremely high fidelity, while exhibiting significant reconstruction bias towards anomalous records. This characteristic of recognizing only normal data and not anomalies is the fundamental guarantee of the high detection rate and low false alarm rate of this method.

[0062] Based on the above embodiments, the parameters of the archive data reconstruction model are updated based on the reconstruction loss between the reconstruction vector and the input vector, including: Step S103A: Calculate the reconstruction error between the input vector and the reconstructed vector in the corresponding dimensions of each field.

[0063] That is, for each element in the vector (representing an original field or its encoded component), the square of the difference between its input value and the reconstructed value is calculated.

[0064] Step S103B: Determine the mean squared error between the input vector and the reconstructed vector based on the reconstruction error in all dimensions, and use the mean squared error to characterize the reconstruction loss.

[0065] Specifically, the mean squared error is obtained by summing the squared errors of all dimensions and then averaging them. This loss function can be expressed as:

[0066] in, The dimension of the input vector. and These are the first and second digits of the input vector and the reconstructed vector, respectively. Each field.

[0067] Step S103C: Based on the reconstruction loss, adjust the network parameters of the archive data reconstruction model through backpropagation to minimize the reconstruction loss.

[0068] In this embodiment of the invention, a gradient descent-type optimization algorithm (such as Adam) is used to update the weights of each layer along the negative gradient direction of the loss function. After multiple iterations, the model's ability to reconstruct normal data is continuously enhanced, and its sensitivity to abnormal data also increases, eventually reaching a stable convergence state.

[0069] This embodiment uses mean squared error as the loss function, which can reflect the reconstruction quality of each field in a balanced way, and is especially sensitive to small deviations in numerical fields.

[0070] Since the loss calculation is based entirely on clean, normal data, the optimization objective is explicitly aimed at minimizing the reconstruction error of normal records, thereby ensuring that the model parameters converge to the state that best represents normal business logic. This training mechanism, which uses normality as the sole optimization objective, is the essential feature that distinguishes this method from supervised or semi-supervised anomaly detection, and it is also the key to its high generalization ability.

[0071] In another embodiment of the present invention, a method for detecting anomalies in archival data is provided, such as... Figure 2 As shown, it includes the following steps: Step S201: Obtain the file record to be detected.

[0072] This record could be a newly entered expense report, newly imported customer information, etc., and has not yet undergone quality verification.

[0073] Step S202: Vectorize the archive record to be detected to obtain the corresponding input vector.

[0074] The processing method is completely consistent with step S102, ensuring that the data preprocessing process is consistent between the training and inference stages, and avoiding misjudgment due to processing deviations.

[0075] Step S203: Input the input vector into the pre-trained archive data reconstruction model to obtain the corresponding reconstruction vector.

[0076] The archival data reconstruction model is pre-trained using the methods described in the above embodiments. Based on the normal paradigms it has learned, the model outputs a reconstruction version that best conforms to the business logic.

[0077] Step S204: Calculate the reconstruction error between the input vector and the reconstructed vector.

[0078] This reconstruction error can be expressed as the square of the Euclidean distance between vectors:

[0079] in, For the input vector, This is the reconstructed vector. This metric comprehensively measures the degree to which the entire record deviates from the normal pattern.

[0080] Step S205: When the reconstruction error is greater than the preset error threshold, the archive record to be detected is determined to be abnormal data.

[0081] In one feasible implementation, the preset error threshold is determined in the following way: Step A: Obtain the reconstruction error distribution generated from normal archival records; Step B: Based on the reconstruction error distribution, determine the statistical characteristics of the reconstruction error distribution.

[0082] The statistical characteristics include any of the following: The error value corresponding to the preset high percentile is used in the reconstructed error distribution; Reconstruct the mean and standard deviation of the error distribution.

[0083] Step C: Determine the preset error threshold based on statistical characteristics.

[0084] When the statistical feature is the error value corresponding to the preset high percentile in the reconstructed error distribution, the error value corresponding to the preset high percentile is directly used as the preset error threshold.

[0085] For example, the error value corresponding to the highest percentile is preset to the 99th percentile, which is used as the preset error threshold. .

[0086] When the statistical characteristics are the mean and standard deviation of the reconstructed error distribution, the sum of preset multiples of the mean and standard deviation is used as the preset error threshold. For example, .

[0087] The threshold set in this way has a solid statistical basis and can effectively balance the detection rate and false alarm rate. It can not only detect obvious errors (such as an amount of 1 million), but also identify subtle but illogical deviations (such as an amount of 9,995 yuan vs. a reasonable value of 10,000 yuan), which greatly improves the sensitivity and robustness of anomaly detection.

[0088] The anomaly detection in this embodiment relies entirely on the model's learning results from normal data distribution. Since the model has never seen anomalous samples, any data that deviates from its reconstruction capabilities will be marked as suspicious. This mechanism of using normality as a yardstick to measure anomalies allows the model to identify the inherent relationships and non-linear dependencies between fields in archival records, discover novel anomalies of unknown types and undefined rules, break through the limitations of traditional hard-coded rules, and achieve accurate detection of deep-seated, hidden defects in archival data.

[0089] Based on the above embodiments, after determining that the archive record to be detected is abnormal data, the method further includes: Step S206: Calculate the reconstruction error between the input vector and the reconstructed vector in the corresponding dimensions of each original field, and determine at least one field whose reconstruction error is greater than a preset error threshold as an abnormal field position.

[0090] Specifically, by performing field-by-field error analysis, the root cause of the problem can be located. For example, if the error in the reimbursement amount field is much higher than that in other fields, then that field is determined to be an anomaly.

[0091] This refined positioning capability solves the pain point of traditional methods that only know there is an error, but do not know where the error lies.

[0092] Step S207: Based on the location of the abnormal field, perform normalization inverse processing on the vector of the corresponding field in the reconstructed vector to generate a repair suggestion value with the same data format as the original field.

[0093] Continuing the previous example, if the normalized value corresponding to the amount in the reconstructed vector is 0.2, it is restored to 10,000 yuan using the inverse Min-Max formula; if the category field reconstruction result is [1,0,0], it is decoded as the R&D department. The suggested values ​​generated in this way can be directly understood and adopted by business personnel, greatly improving the efficiency of the repair process.

[0094] Step S208: In the data quality audit interface, the archive records corresponding to the abnormal data are highlighted, and the original field values ​​and repair suggestion values ​​are presented in a comparison chart.

[0095] In a specific example, the review interface highlights the entire abnormal record in red; after the user clicks, a pop-up window displays the original value (abnormal fields are highlighted in red) on the left and the system's suggested value and confidence level on the right (e.g., suggested to be corrected to 10,000 yuan, confidence level 95%). This allows users to intuitively see where the abnormality is.

[0096] The repair suggestions in this embodiment are directly derived from the model's reconstruction results of normal data, essentially correcting abnormal input using normal patterns. Since the model is trained on a large amount of clean data, its reconstructed values ​​naturally conform to business common sense and logical constraints between fields, thus the suggestions are highly reasonable and credible. Combined with visual interaction, this significantly improves the automation level and business efficiency of archival data governance.

[0097] Based on the same inventive concept, embodiments of the present invention provide a device for detecting anomalies in archival data, such as... Figure 3 As shown, it includes: Acquisition unit 301 is used to acquire the file records to be detected.

[0098] The vectorization processing unit 302 is used to perform vectorization processing on the archive record to be detected to obtain the corresponding input vector.

[0099] The reconstruction unit 303 is used to input the input vector into the pre-trained archive data reconstruction model to obtain the corresponding reconstruction vector.

[0100] The calculation unit 304 is used to calculate the reconstruction error between the input vector and the reconstructed vector.

[0101] The determination unit 305 is used to determine the archive record to be detected as abnormal data when the reconstruction error is greater than a preset error threshold.

[0102] Based on the same technical concept, embodiments of the present invention also provide an electronic device, such as... Figure 4 As shown, it includes a processor 401, a communication interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0103] Memory 403 is used to store computer programs; When the processor 401 executes the program stored in the memory 403, it implements the steps in the above method embodiments.

[0104] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0105] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0106] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0107] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0108] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps in the above-described method embodiments. Specific implementation details can be found in the method embodiments and will not be repeated here.

[0109] The device for detecting anomalies in archival data provided in this embodiment of the invention can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this embodiment of the invention are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiments can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0110] In the embodiments provided by this invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0111] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] In addition, the functional units in the embodiments provided by the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0113] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0115] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. All should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A training method for an archival data reconstruction model, characterized in that, include: Retrieve several verified normal archival records from the historical archive database; Each field in each of the aforementioned normal file records is vectorized to obtain an input vector, and all the input vectors are used to form a training sample set; Based on the pre-built archive data reconstruction model, the training sample set is subjected to unsupervised iterative training until the preset iteration stopping condition is reached. In each iteration of training, the non-linear dependencies between fields in the normal archival records are learned, and their joint distribution characteristics are implicitly modeled. Output a reconstructed vector and update the parameters of the archive data reconstruction model based on the reconstruction loss between the reconstructed vector and the input vector.

2. The method according to claim 1, characterized in that, The types of the fields include at least category fields and numeric fields; the vectorization process of each field in each normal record to obtain the input vector includes: One-hot encoding is performed on the categorical fields to obtain a categorical field vector; wherein the dimension of the categorical field vector is the number of all categorical fields. The numerical fields are normalized to map the original values ​​to a preset range, resulting in a vector of numerical fields. The input vector is obtained by concatenating all categorical field vectors with all numeric field vectors in a preset order.

3. The method according to claim 1, characterized in that, The archival data reconstruction model includes an encoder and a decoder. The encoder includes an input layer, a first hidden layer, and a second hidden layer. The decoder includes a third hidden layer and an output layer. The input layer is used to receive normal archival records that have undergone vectorization as input vectors; The first hidden layer is used to perform preliminary compression and feature abstraction on the input vector through a non-linear activation function in order to capture local correlation patterns between fields; The second hidden layer serves as a bottleneck layer, used to generate a low-dimensional latent representation that implicitly encodes the global nonlinear dependencies and joint distribution characteristics between fields in a normal archival record. The third hidden layer is used to symmetrically expand the low-dimensional latent representation and gradually recover the high-dimensional feature structure through a nonlinear activation function; The output layer is used to output a reconstruction vector with the same dimension as the input vector, which represents the optimal reconstruction result for the normal archival data distribution under the current model parameters.

4. The method according to claim 1, characterized in that, The parameters of the archive data reconstruction model are updated based on the reconstruction loss between the reconstructed vector and the input vector, including: Calculate the reconstruction error between the input vector and the reconstructed vector in each field's corresponding dimension; The mean squared error between the input vector and the reconstructed vector is determined based on the reconstruction error in all dimensions, and the mean squared error is used to characterize the reconstruction loss. Based on the reconstruction loss, the network parameters of the archive data reconstruction model are adjusted through backpropagation to minimize the reconstruction loss.

5. A method for detecting anomalies in archival data, characterized in that, include: Obtain the archive records to be inspected; The file record to be detected is vectorized to obtain the corresponding input vector; The input vector is input into the archive data reconstruction model trained by any one of claims 1 to 4 to obtain the corresponding reconstruction vector; Calculate the reconstruction error between the input vector and the reconstructed vector; When the reconstruction error is greater than a preset error threshold, the file record to be detected is determined to be abnormal data.

6. The method according to claim 5, characterized in that, The preset error threshold is determined in the following way: Obtain the reconstruction error distribution generated from normal archival records; Based on the reconstruction error distribution, determine the statistical characteristics of the reconstruction error distribution; The preset error threshold is determined based on the statistical characteristics; The statistical characteristics include any of the following: The error value corresponding to the preset high percentile in the reconstructed error distribution; The mean and standard deviation of the reconstruction error distribution.

7. The method according to claim 5, characterized in that, After determining that the archive record to be detected is abnormal data, the method further includes: Calculate the reconstruction error between the input vector and the reconstructed vector in the corresponding dimensions of each original field, and determine at least one field whose reconstruction error is greater than a preset error threshold as an abnormal field position; Based on the location of the abnormal field, the vector of the corresponding field in the reconstructed vector is subjected to normalization inverse processing to generate a repair suggestion value with the same data format as the original field; In the data quality audit interface, the archive records corresponding to the abnormal data are highlighted, and the original field values ​​and the suggested repair values ​​are presented in a comparison chart.

8. A device for detecting anomalies in archival data, characterized in that, include: The acquisition unit is used to acquire the archive records to be inspected; The vectorization processing unit is used to perform vectorization processing on the archive record to be detected to obtain the corresponding input vector; A reconstruction unit is used to input the input vector into the archive data reconstruction model trained by the method of any one of claims 1 to 4, and obtain the corresponding reconstruction vector; A calculation unit is used to calculate the reconstruction error between the input vector and the reconstructed vector; The determining unit is used to determine that the archive record to be detected is abnormal data when the reconstruction error is greater than a preset error threshold.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images