Data anomaly identification method and system based on machine learning

Abstract

The invention discloses a data exception identification method and system based on machine learning, and the method comprises the steps: obtaining reference document data, carrying out the document analysis and document enhancement, obtaining standard document data, carrying out the multi-modal feature layering extraction according to the standard document data, and obtaining multi-modal document features, performing feature edge processing according to the multi-modal document features to obtain combined document features, obtaining document density and document statistical data according to the multi-modal document features, calculating document anomaly coefficients, and constructing a self-adaptive data anomaly recognition model, and inputting to-be-identified document data into the adaptive data exception identification model to obtain a document data exception identification result. The method not only can improve the accuracy and comprehensiveness of document data exception recognition, but also has better interpretability, and can be directly applied to a data exception recognition system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data recognition, in particular to a data anomaly recognition method and system based on machine learning. BACKGROUND

[0002] With the acceleration of enterprise digitization process, document data has become the core information carrier, covering structured reports, semi-structured logs and unstructured texts and other forms. The data anomaly recognition technology based on machine learning is crucial for ensuring document security, improving data quality and optimizing knowledge management. Therefore, an efficient and accurate document anomaly detection system has become a key infrastructure for enterprise data governance.

[0003] The current traditional document data anomaly recognition technology still has many limitations: first, most traditional technologies focus on anomaly detection of structured data, and lack feature extraction capability for unstructured texts and semi-structured data, resulting in a large number of potential anomalies that cannot be identified; second, traditional technologies are often based on fixed rules or simple statistical models, and once the generation mode or business rules of document data change, the model performance will decrease sharply, increasing the model training cost and rule maintenance cost; third, traditional technologies lack effective graph neural network architecture, making it difficult to use semantic connections such as document references and associations to assist anomaly detection, resulting in a lack of global perspective in the detection results and a large number of false positives and false negatives. The present application proposes a data anomaly recognition method and system based on machine learning, which constructs a document relationship graph through a context enhancer, captures cross-document association patterns, designs a hierarchical multi-modal feature extraction framework, and specifically processes the heterogeneous features of three types of documents. The innovative dynamic threshold feedback system effectively balances precision and recall rate, overcoming the defects of traditional technologies. This method can improve the accuracy and comprehensiveness of document data anomaly recognition, providing strong technical support for ensuring data quality and driving the stable development of digital business. SUMMARY

[0004] The purpose of the present application is to provide a data anomaly recognition method and system based on machine learning.

[0005] To achieve the above purpose, the present application is implemented according to the following technical solutions: The present application comprises the following steps: Obtaining reference document data, performing document parsing and document enhancement on the reference document data to obtain standard document data; the reference document data includes document data and anomaly labels; Performing multi-modal feature hierarchical extraction on the standard document data to obtain multi-modal document features, and performing feature edge processing on the multi-modal document features to obtain combined document features; the combined document features include document cross features and document splicing features; According to the multi-modal document feature, document density and document statistical data are obtained, and a document anomaly coefficient is calculated according to the document density and the document statistical data; According to the document anomaly coefficient and the combined document feature, an adaptive data anomaly recognition model is constructed, and document data anomaly recognition results are obtained by inputting to-be-recognized document data into the adaptive data anomaly recognition model.

[0006] Further, the method for obtaining standard document data comprises: Reference document data is obtained, and the reference document data is divided into structured documents, semi-structured documents and unstructured documents by using a BERT-based text classifier; the reference document data comprises document data and anomaly labels; the document data comprises text content, metadata and structural features; Type document analysis is performed according to the document data division results to obtain document analysis data; the document analysis data comprises field value relationships, field table relationships, hierarchical paths and document location parameters; The document analysis data is input into a context enhancer to perform document enhancement to obtain document enhancement data; the context enhancer comprises an association mining enhancement module, a knowledge graph enhancement module and a behavior log enhancement module; the association mining enhancement module constructs a document topological relationship through explicit links and implicit associations of semi-structured documents; the knowledge graph enhancement module constructs an enterprise graph through structured documents, and injects entity links into the enterprise graph to determine document context labels; the behavior log enhancement module generates a behavior vector according to dynamic quantitative access, editing and sharing behaviors; the document enhancement data comprises a document topological relationship, an enterprise graph and a behavior vector; The standard document data is composed of the document enhancement data and the reference document data.

[0007] Further, the method for obtaining combined document features comprises: The standard document data is input into a multi-modal feature extraction model to output multi-modal document features; the multi-modal feature extraction model comprises a text feature extraction layer, a metadata encoding layer, a structural feature conversion layer, a topological relationship feature extraction layer and a behavior vector straight-through layer; The text feature extraction layer processes text content through a three-layer BERT model and a topic-sentiment double-channel to obtain a text feature vector; the text feature vector comprises word-level anomalies, syntax feature document semantic embeddings, topic distribution vectors, sentiment polarity scores and emotion intensity values; The metadata encoding layer edits metadata to obtain a metadata feature vector; the metadata feature vector comprises periodic features, version change frequencies and embedded features; The structural feature conversion layer processes structural features and structural feature analysis results to obtain a structural feature vector; The topology relationship feature extraction layer processes the document topology relationship to obtain a topology feature vector, and includes a graph structure feature calculation module, a neighbor feature aggregation module, a multi-scale graph neural network module and a feature fusion module. The graph structure feature calculation module calculates multi-relation weighted centrality, clustering coefficient and degree distribution according to the document topology relationship, and obtains graph structure features by combination. , wherein is the weight between the relationship types and , , is the embedding representation of the relationship type , , is a temperature parameter for controlling the "sharpness" degree of the exponential function. The neighbor feature aggregation module determines neighbor features by calculating the semantic similarity and the topological proximity between documents, and the expression is as follows: , , wherein is a vector aggregated by the neighbor node features of the node , is a neighbor node set of the node , denotes the node weight between the nodes , , is an activation function, W is a weight matrix for linear transformation of the neighbor node features, is the feature of the node , is a feature bias term, is an activation function, is the transpose of a vector attention query vector , is the semantic similarity between the nodes , , is the topological proximity between the nodes , , which is calculated by using the Adamic-Adar index. The multi-scale graph neural network module balances the local and global features of the document topology relationship by using a gating mechanism to obtain graph neural network features, and the expression is as follows: , , , in For nodes The graph neural network features of is the layer normalization operation, For nodes exist The hidden state of the layer, For nodes The global characteristics of is a gated recurrent unit, is the attention weight, which is used to measure the node For Node The degree of impact, For the The weight matrix of the layer, is the sigmoid activation function, is the weight matrix of global feature gating, For nodes In the hidden state of the initial layer, is a pooling operation used to aggregate the features of neighboring nodes. For nodes 's K-hop neighbor set; The feature fusion module adaptively fuses graph structure features, neighbor features, and graph neural network features to obtain a topological feature vector, which is expressed as: , , in For nodes The fused topological feature vector, is the adaptive weight coefficient, For nodes The graph structure characteristics of is a multi-layer perceptron, For nodes The degree, For nodes The entropy of the neighbor label distribution, For nodes The initial eigenvector of For nodes Place feature weight function; The behavior vector pass-through layer directly receives the behavior vector, performs Z-score normalization, and outputs the standard behavior vector; The text feature vector, metadata feature vector, structural feature vector, topological feature vector and standard behavior vector are respectively subjected to feature concatenation and polynomial feature crossover to obtain the combined document feature.

[0008] Further, the method for calculating the document density and the document statistical data comprises: statistically analyzing the document splicing features to obtain the document statistical data; extracting a text feature vector in the document splicing features, marking corresponding fields of the text content according to the text feature vector, taking the marked fields as a first document set, dividing fields containing numerical fields and numerical values outside a preset numerical value range of a corresponding category in the first document set into a first mark, and taking the remaining marked fields as a second mark; extracting a metadata feature vector and a standard behavior vector in the document splicing features, taking numerical features in the metadata feature vector and the standard behavior vector as a second document set, and taking numerical features with values greater than a preset threshold value of a corresponding category in the second document set as a third mark; determining a first dimension according to a category number of the first mark, determining a first document density according to a ratio of the first mark number and the first dimension, determining a second dimension according to a category number of the second mark, and determining a second document density according to a ratio of the second mark number and the second dimension; the first document density is related to abnormal text content; and the second document density is related to normal text content; calculating a mean value of a ratio of a numerical feature of the third mark to a preset threshold value of a corresponding category as a first offset degree, determining a third dimension according to a ratio of a data length of the third mark and a category number of the third mark, and taking a product of the first offset degree and the third dimension as a third document density; the third document density is related to abnormal text information or operation; the document density is composed of the first document density, the second document density, and the third document density.

[0009] Further, the method for calculating the document anomaly coefficient comprises: calculating the document anomaly coefficient according to the document density and the document statistical data, and the expression is: , wherein is the document anomaly coefficient, is the first document density, is the second document density, is the third document density, N is the number of abnormal data, is the document skewness of the first abnormal data corresponding to the document splicing feature .

[0010] Further, the method for obtaining the document data anomaly recognition result comprises: ​The document density, document anomaly coefficient, combined document features, and anomaly annotations form a comprehensive document set, which is then divided into a training set and a test set at a ratio of 6:4. The training set is used to train the adaptive data anomaly recognition model, and the test set is used to evaluate the performance of the adaptive data anomaly recognition model. The adaptive data anomaly recognition model includes an anomaly detection layer, an adaptive learning layer, and an output decision layer; the anomaly detection layer performs anomaly detection on document data to obtain anomaly scores and corresponding anomaly labels, including a reconstruction model module, a density model module, and a graph model module; the anomaly scores include a reconstruction anomaly score, a density anomaly score, and a graph anomaly score; The reconstruction model module uses a conditional variational autoencoder to detect feature-level anomalies and identify feature combinations that deviate from the normal pattern to obtain a reconstruction anomaly score. and exception labels; The density model module extracts density features, fuses Gaussian mixture model and kernel density estimation to perform mixed density estimation to identify outliers on the edge of data distribution, and outputs density anomaly scores. and exception labels; The graph model module processes the document topology relationship according to the heterogeneous graph neural network and performs abnormal state enhancement propagation to obtain the graph anomaly score. and exception labels, the expression is: , , , in is the sigmoid activation function, is the transpose of the weight vector, For nodes The final hidden state of For nodes The degree, is a multi-layer perceptron, For nodes In the hidden state of the last layer, For nodes The abnormal state coefficient, For nodes Neighboring nodes The maximum abnormal state coefficient in , is the ReLU activation function, For nodes 、 The node weights between For the The weight matrix of the layer, For nodes exist The hidden state of the layer; The adaptive learning layer comprises a concept drift detection unit, a dynamic threshold adjustment unit and a model weight updating unit; The concept drift detection unit identifies the input data and triggers anomaly detection and model updating; The dynamic threshold adjustment unit calculates a three-dimensional feedback factor according to the concept drift detection result, the anomaly score output by the anomaly detection layer and the corresponding anomaly label, optimizes the anomaly judgment boundary and updates the dynamic threshold according to the three-dimensional feedback factor, and the expression is: , Wherein is the dynamic threshold at time step t, is the standard dynamic threshold, , , is a three-dimensional weight coefficient, , is a precision feedback factor, is a precision sensitivity coefficient, is the current model precision, is the target precision, is a recall rate feedback factor, is a recall rate sensitivity coefficient, is the target recall rate, is the current model recall rate, is a concept drift feedback factor, is a drift sensitivity coefficient, is a Sigmoid function, is a drift response intensity, is the KL divergence at time t, is a drift detection threshold; The model weight updating unit calculates a weighted integrated anomaly score according to the dynamic threshold updating result, the anomaly score output by the anomaly detection layer and the corresponding anomaly label, and outputs a three-model integrated weight through sparse regularization updating of a weight loss function, and the expression is: , , Wherein is the update amount of weight , comprises reconstruction weight, density weight and graph weight, is a learning rate for controlling the step size of weight updating, is a sparse regularization strength parameter, is a sign function of weight , is a weight loss function, is a true label, a weighted ensemble of the three model anomaly scores; The output decision layer makes an abnormality decision according to the document topological relationship, the spliced document features, the anomaly score output by the anomaly detection layer, the weighted ensemble anomaly score and the updated dynamic threshold to output a document data anomaly identification result, which includes an integrated decision unit and an interpretable analysis unit; the document data anomaly identification result includes an abnormality decision, an abnormality confidence, an abnormality type and an abnormality reason.

[0011] In a second aspect, a machine learning-based data anomaly identification system includes: A document processing module: acquires reference document data, performs document parsing and document enhancement on the reference document data to obtain standard document data; A feature extraction module: configured to perform multi-modal feature hierarchical extraction on the standard document data to obtain multi-modal document features, and perform feature edge processing on the multi-modal document features to obtain combined document features; An anomaly coefficient module: configured to obtain document density and document statistical data according to the multi-modal document features, and calculate a document anomaly coefficient according to the document density and the document statistical data; An anomaly identification module: configured to construct an adaptive data anomaly identification model according to the document anomaly coefficient and the combined document features, input the document data to be identified into the adaptive data anomaly identification model to obtain a document data anomaly identification result; An intelligent management module: configured to store, view and manage the document data anomaly identification result, and perform document repair and management according to the document data anomaly identification result.

[0012] The present application has the following beneficial effects: Compared with the prior art, the present application has the following technical effects: The present application can improve the data preprocessing capability and enhance the model adaptability in document data anomaly identification through the steps of document parsing and enhancement, multi-modal feature hierarchical extraction, document density calculation, document anomaly coefficient calculation and model construction, thereby improving the efficiency and accuracy of document data anomaly identification, optimizing the document data anomaly identification technology, greatly saving resources, improving work efficiency, realizing the identification of document data anomaly, providing solid technical support for guaranteeing data quality and driving the stable development of digital business, and being suitable for different document data anomaly identification systems and different users' machine learning-based data anomaly identification needs, and having certain universality. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 The present application is a step flowchart of a machine learning-based data anomaly identification method. DETAILED DESCRIPTION

[0014] The present invention will be further described below through specific examples. The illustrative examples and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0015] The present invention provides a data anomaly identification method and system based on machine learning, comprising the following steps: like Figure 1 As shown, in this embodiment, the following steps are included: Acquire reference document data, and perform document parsing and document enhancement on the reference document data to obtain standard document data; the reference document data includes document data and anomaly annotations; Performing multimodal feature hierarchical extraction on the standard document data to obtain multimodal document features, and performing feature edge processing on the multimodal document features to obtain combined document features; the combined document features include document intersection features and document splicing features; Obtaining document density and document statistical data according to the multimodal document features, and calculating a document anomaly coefficient according to the document density and the document statistical data; An adaptive data anomaly recognition model is constructed according to the document anomaly coefficient and the combined document feature, and the document data to be recognized is input into the adaptive data anomaly recognition model to obtain a document data anomaly recognition result.

[0016] In this embodiment, the method for obtaining standard document data includes: Obtain reference document data, and classify the reference document data into structured documents, semi-structured documents, and unstructured documents using a BERT-based text classifier; the reference document data includes document data and anomaly annotations; the document data includes text content, metadata, and structural features; Performing typed document parsing according to the document data division result to obtain document parsing data; the document parsing data includes field value relationships, field table relationships, hierarchical paths, and document location parameters; The document parsing data is input into a context enhancer for document enhancement to obtain document enhancement data; the context enhancer includes an association mining enhancement module, a knowledge graph enhancement module, and a behavior log enhancement module; the association mining enhancement module constructs document topology relationships through explicit links and implicit associations of semi-structured documents; the knowledge graph enhancement module constructs an enterprise graph through structured documents, and injects entity links into the enterprise graph to determine document context labels; the behavior log enhancement module generates behavior vectors based on dynamically quantified access, editing, and sharing behaviors; the document enhancement data includes document topology relationships, enterprise graphs, and behavior vectors; The standard document data is composed of document enhancement data and reference document data; In actual evaluation, taking the data anomaly identification of a company's document as an example, the text content is divided into structured documents (regular data composed of pre-defined fields), semi-structured documents (free content mixed with labels), and unstructured documents (continuous free text stream), the structure features are divided into structured documents (explicit table structure), semi-structured documents (hierarchical label structure), and unstructured documents (implicit logical structure), and the metadata is divided into structured documents (table name, data source IP, and update timestamp), semi-structured documents (file path, template version, and author department), and unstructured documents (creator mailbox, file size, and access permission list); The document data division result is subjected to typed document analysis to obtain document analysis data, and the specific steps include: The structured document output field value and table relationship are extracted by using a pandas table analysis library, the hierarchical path (content, link list, and format label) is obtained by extracting the semi-structured document hierarchical label by using JSONPath, and the NLP pipeline and LayoutLM layout analysis model are respectively used to perform word segmentation and sentence segmentation on the unstructured document and identify the position to output the document position parameters (sentence list, paragraph boundary, and OCR text coordinates); In the correlation mining enhancement module: the explicit correlation specifically refers to the analysis of hyperlinks and reference marks in the document, and the implicit correlation is determined by calculating the semantic similarity (BERT embedding cosine distance) or co-occurring entities (such as the same customer name appearing in multiple contracts) between documents; in the knowledge graph enhancement module: the enterprise knowledge graph API is connected, the company name / project ID in the document is mapped to the graph node through entity linking, and the graph attributes of the document author's department, the project's life cycle stage, and the customer's risk level are updated; in the behavior log enhancement module: the document audit log is queried, the access frequency in the past 30 days, the editing time sequence, and the sharing range change record (such as expanding from the department group to the entire staff) are extracted, and the behavior vector [daily access volume, editing frequency variance, and sharing group quantity] is generated; The standard document data is composed of the document enhancement data and the reference document data.

[0017] In this embodiment, the method for obtaining the combined document features includes: The standard document data is input into a multi-modal feature extraction model to output multi-modal document features; the multi-modal feature extraction model includes a text feature extraction layer, a metadata encoding layer, a structure feature conversion layer, a topological relationship feature extraction layer, and a behavior vector straight-through layer; The text feature extraction layer processes the text content through a three-layer BERT model and a topic-sentiment double-channel to obtain a text feature vector; the text feature vector includes a word-level anomaly, a syntax feature document semantic embedding, a topic distribution vector, a sentiment polarity score, and an emotion intensity value; The metadata encoding layer obtains a metadata feature vector by metadata editing; the metadata feature vector includes periodicity features, version change frequency, and embedding features; The structural feature conversion layer obtains a structural feature vector by processing the structural features and the structural feature analysis results; The topology relationship feature extraction layer processes document topology relationships to obtain a topology feature vector, including a graph structure feature calculation module, a neighbor feature aggregation module, a multi-scale graph neural network module, and a feature fusion module; The graph structure feature calculation module calculates multi-relation weighted centrality, clustering coefficient, and degree distribution according to the document topology relationships, and combines to obtain graph structure features, and a multi-relation weighted weight calculation expression is as follows: , wherein is a weight between relation types and , , is an embedding representation of the relation type , , is a temperature parameter for controlling the "sharpness" degree of the exponential function; The neighbor feature aggregation module determines neighbor features by calculating semantic similarity and topological proximity between documents, and an expression is as follows: , , wherein is a vector aggregated by neighbor node features of a node , is a neighbor node set of the node , denotes a node weight between nodes , , is an activation function, W is a weight matrix for linear transformation of neighbor node features, is a feature of the node , is a feature bias term, is an activation function, is a transpose of a vector attention query vector , is semantic similarity between nodes , , is topological proximity between nodes , , and is calculated by using an Adamic-Adar index. The multi-scale graph neural network module balances the local and global features of the topology relationship through a gating mechanism to obtain a graph neural network feature, and the expression is: , , , wherein is a graph neural network feature of a node , is a layer normalization operation, is a hidden state of a node at a layer, is a global feature of a node , is a gated recurrent unit, is an attention weight, used to measure the influence degree of a node on a node , is a weight matrix of an i-th layer, is a sigmoid activation function, is a weight matrix of a global feature gating, is a hidden state of a node at an initial layer, is a pooling operation, used to aggregate the features of neighbor nodes, is a K-hop neighbor set of a node ; The feature fusion module adaptively fuses the graph structure feature, the neighbor feature and the graph neural network feature to obtain a topology feature vector, and the expression is: , , , wherein is a fused topology feature vector of a node , is an adaptive weight coefficient, is a graph structure feature of a node , is a multi-layer perceptron, is a degree of a node , is an entropy of a neighbor label distribution of a node , is an initial feature vector of a node , is an i-th feature weight function of a node ; ​​The behavior vector direct-through layer directly receives the behavior vector, performs Z-score standardization, and outputs a standard behavior vector; The text feature vector, the metadata feature vector, the structure feature vector, the topology feature vector, and the standard behavior vector are respectively subjected to feature splicing and polynomial feature crossing to obtain a combined document feature; In actual evaluation, taking the data anomaly identification of a company document as an example, in the text feature extraction layer: the bottom-layer BERT model captures word-level anomalies such as unusual terms and sensitive words, the middle-layer BERT model extracts syntax features such as dependency tree depth and passive voice proportion, the top-layer BERT model generates a 512-dimensional semantic embedding vector, and the BERTopic model outputs a 128-dimensional topic distribution vector; the output text feature vector is [word-level anomaly label, syntax feature, semantic embedding, topic vector, sentiment polarity, and emotion intensity]; In the metadata encoding layer: the creation / modification time is converted into periodic features (workday / weekend, quarterly peak label), the version change frequency is calculated according to the number of modifications in a unit time, the author / departments are graph-embedded to obtain author embedding and department embedding, and the type embedding is obtained by scientific system embedding of the document type; the output metadata feature vector is [time period encoding, version frequency, author embedding, department embedding, and type embedding]; In the structure feature conversion layer: the table association complexity (foreign key reference depth x associated table number) is calculated according to the structured document (explicit table structure), the label level depth is extracted and the cross-chapter link density (number of inter-chapter hyperlinks / total number of chapters) is calculated according to the semi-structured document (hierarchical label structure), and the layout structure embedding vector is generated according to the structure feature analysis result (document position parameter) and the paragraph length dispersion (standard deviation / mean) is calculated; The temperature parameter is taken in the topology relationship feature extraction layer = 0.5, the feature bias term = 0.1, the document topology relationship is input into the topology relationship feature extraction layer to obtain a topology feature vector, and the behavior vector direct-through layer directly receives the behavior vector, performs Z-score standardization, and outputs a standard behavior vector; The text feature vector, the metadata feature vector, the structure feature vector, the topology feature vector, and the standard behavior vector are spliced to obtain document splicing features, the text feature vector, the metadata feature vector, the structure feature vector, the topology feature vector, and the standard behavior vector are subjected to polynomial feature crossing to obtain document crossing features, and the combined document feature is composed of the document splicing features and the document crossing features.

[0018] In the embodiment, the method for calculating the document density and the document statistical data comprises: The document splicing features are subjected to statistical analysis to obtain the document statistical data; extracting the text feature vector in the document splicing feature, marking the corresponding field of the text content according to the text feature vector, taking the marked field as a first document set, dividing the field containing the numerical field and the numerical value outside the corresponding category preset numerical value range in the first document set into a first mark, and taking the remaining marked field as a second mark; extracting the metadata feature vector and the standard behavior vector in the document splicing feature, taking the numerical features in the metadata feature vector and the standard behavior vector as a second document set, and taking the numerical features with values greater than the corresponding category preset threshold value in the second document set as a third mark; determining a first dimension according to the number of categories of the first mark, determining a first document density according to the ratio of the first mark quantity and the first dimension, determining a second dimension according to the number of categories of the second mark, and determining a second document density according to the ratio of the second mark quantity and the second dimension; the first document density is related to abnormal text content; the second document density is related to normal text content; calculating the average of the ratio of the numerical feature value of the third mark to the corresponding category preset threshold value as a first offset, determining a third dimension according to the ratio of the data length of the third mark to the number of categories of the third mark, and taking the product of the first offset and the third dimension as a third document density; the third document density is related to abnormal text information or operation; The document density is composed of the first document density, the second document density and the third document density; In actual evaluation, taking the data anomaly identification of a company document as an example, the numerical features of the document splicing features to be identified are compared with the average of the historical document feature values to obtain the document skewness, and the numerical features of the document splicing features exceeding 0.2 times the average of the corresponding historical document feature values are defined as abnormal hidden danger data (15); the document skewness and the abnormal hidden danger data are used to form the document statistical data; Taking part of the text feature vector in the document splicing feature as an example, locating the marked text content A (containing the special word "high risk" and the numerical value "100"), B (containing the symbol "" and the reference relationship "obtained from X"), C (containing the numerical value "1500"), D (containing the special word "risk" and the numerical value "80"), and E (containing the numerical value "800"), the text content A-E is taken as a first document set, the text content E and E are divided into a first mark according to the risk threshold range 0-85 and the numerical threshold 1000, and the remaining text content A-C is divided into a second mark, and the first document set corresponding to the text feature vector is divided to obtain a first mark quantity of 6, a first dimension of 5, a second mark quantity of 25, and a second dimension of 5. The first document density is 1.2, and the second document density is 5; The second document set is composed of some metadata feature vectors (version change frequency 8 times / cycle) and standard behavior vectors [average daily visits 25, edit variance 0.4, number of shared groups 5] in the document splicing features. The ratios to the corresponding category preset thresholds [6, 30, 0.5, 3] are 1.33, 0.83, 0.8, and 1.66, respectively. The average ratio of all numerical features of the second document set to the corresponding category preset thresholds is 1.15. The third dimension is determined to be 1.25 based on the ratio of the third tag data length 15 and the number of third tag categories 12, and the third document density is 1.44.

[0019] In this embodiment, the method for calculating the document anomaly coefficient includes: The document anomaly coefficient is calculated based on the document density and document statistics. The expression is: , in is the document anomaly coefficient, is the first document density, is the second document density, is the third document density, N is the number of abnormal data, For the Abnormal data corresponding to document splicing features Document skewness of In the actual evaluation, taking the anomaly identification of a company's document data as an example, the document anomaly coefficient was calculated to be 1.245 based on the document skewness (the maximum document skewness was 1.8) and document density of 15 abnormal hidden danger data.

[0020] In this embodiment, the method for obtaining document data anomaly recognition results includes: The document density, document anomaly coefficient, combined document features, and anomaly annotations form a comprehensive document set, which is then divided into a training set and a test set at a ratio of 6:4. The training set is used to train the adaptive data anomaly recognition model, and the test set is used to evaluate the performance of the adaptive data anomaly recognition model. The adaptive data anomaly recognition model includes an anomaly detection layer, an adaptive learning layer, and an output decision layer; the anomaly detection layer performs anomaly detection on document data to obtain anomaly scores and corresponding anomaly labels, including a reconstruction model module, a density model module, and a graph model module; the anomaly scores include a reconstruction anomaly score, a density anomaly score, and a graph anomaly score; The reconstruction model module uses a conditional variational autoencoder to detect feature-level anomalies and identify feature combinations that deviate from the normal pattern to obtain a reconstruction anomaly score. and exception labels; The density model module identifies outliers at the edge of data distribution by extracting density features, fusing Gaussian mixture model and kernel density estimation for mixed density estimation, and outputs density anomaly scores and anomaly labels; The graph model module processes document topological relations and performs abnormal state enhancement propagation according to a heterogeneous graph neural network to obtain graph anomaly scores and anomaly labels, and the expression is: , , , wherein is a sigmoid activation function, is the transpose of a weight vector, is the final hidden state of a node , is the degree of a node , is a multi-layer perceptron, is the hidden state of a node at the last layer, is an abnormal state coefficient of a node , is the maximum abnormal state coefficient in neighbor nodes of a node , is a ReLU activation function, is a node weight between a node , is a weight matrix of an i-th layer, is the hidden state of a node at the i-th layer; ; The adaptive learning layer comprises a concept drift detection unit, a dynamic threshold adjustment unit and a model weight updating unit; The concept drift detection unit identifies input data and triggers abnormality detection and model updating; The dynamic threshold adjustment unit calculates a three-dimensional feedback factor according to the concept drift detection result, the abnormality score and the corresponding anomaly label output by the abnormality detection layer, optimizes the abnormality judgment boundary according to the three-dimensional feedback factor and updates the dynamic threshold, and the expression is: , wherein is a dynamic threshold at a time step t, is a standard dynamic threshold, ​​​​​is the three-dimensional weight coefficient, , is the precision feedback factor, is the precision sensitivity coefficient, is the current model accuracy, is the target accuracy, is the recall rate feedback factor, is the recall sensitivity coefficient, is the target recall rate, is the recall rate of the current model, is the concept drift feedback factor, is the drift sensitivity coefficient, is the Sigmoid function, is the drift response strength, is the KL divergence at time t, is the drift detection threshold; The model weight update unit calculates the weighted integrated anomaly score based on the dynamic threshold update result, the anomaly score output by the anomaly detection layer, and the corresponding anomaly label, and performs sparse regularization update through the weight loss function to output the integrated weight of the three models. The expression is: , , in Weight The update amount, Including reconstruction weight, density weight and graph weight, is the learning rate, which is used to control the step size of weight update. is the sparse regularization strength parameter, Weight The symbolic function of is the weight loss function, is the true label, A weighted ensemble of the anomaly scores of the three models; The output decision layer makes an anomaly decision based on the document topology, the spliced ​​document features, the anomaly score output by the anomaly detection layer, the weighted integrated anomaly score, and the updated dynamic threshold, and outputs the document data anomaly recognition result, including an integrated decision unit and an interpretable analysis unit; the document data anomaly recognition result includes an anomaly decision, anomaly confidence, anomaly type, and anomaly cause; In the actual evaluation, taking the anomaly recognition of a company's document data as an example, drift detection is performed by calculating the KL divergence in the concept drift detection unit. The expression is: , in is the KL divergence at time t, To sum the probabilities of all categories, the probability distribution of category b at time t; is calculated is greater than a preset threshold , triggering abnormal detection and model updating; In the reconstruction model module, the encoder concatenates the input document stitching features and the document anomaly coefficient to the latent space to obtain the distribution of latent variables z, and the decoder generates the reconstructed document stitching features according to the latent variables z and the document anomaly coefficient , and calculates the reconstructed anomaly score, the expression is: , , wherein is the Euclidean distance between the input features and the reconstructed features , is the scale factor of the document anomaly coefficient , is the loss function of the conditional variational autoencoder, is the distribution of the encoder, is the encoder parameter, is the distribution of the decoder, is the decoder parameter, is the Kullback-Leibler divergence, is the reconstruction error parameter, used to balance the weight of reconstruction error and KL divergence; In the density model module, the density features are extracted according to the document density, the document anomaly coefficient and the cross-document features, and the adaptive mixing weight is calculated according to the document anomaly coefficient , the mixed density estimation is performed according to the adaptive mixing weight to output the density anomaly score , the expression is: , , , wherein is the density feature, is the probability density of the mixed density estimation, is the probability density function of the Gaussian mixture model, is the probability density function of the kernel density estimation, is the control parameter of the document anomaly coefficient , used to adjust the influence degree of the anomaly state coefficient, is a multilayer perceptron, is the first document density, is the third document density, is the top K=10 features selected from the fused features, is the Sigmoid activation function, =0.8 is the activation weight, =0.2 is the activation bias; The weight vector in the graph model module is [0.2, -0.1, 0.3], the node neighbor node has the maximum abnormal state coefficient ; The document density, document anomaly coefficient, and combined document features of the to-be-identified document data are input into the adaptive data anomaly identification model, and the anomaly score and corresponding anomaly label are output by the anomaly detection layer: =0.75 / financial numerical value is too large / access frequency is high, =0.68 / normal data, =0.783 / data correlation is too strong; In the dynamic threshold adjustment unit, the current model accuracy =0.85, the current model recall rate =0.92, the accuracy sensitivity coefficient =0.5, the target accuracy =0.9, the recall rate sensitivity coefficient =0.5, the target recall rate =0.95, the drift sensitivity coefficient =0.3, the drift response intensity =1, and the three-dimensional feedback factor (accuracy factor 0.975, recall rate factor 1.015, and drift factor 1.1536) is calculated; The standard dynamic threshold =0.6, the three-dimensional weight coefficient is 0.4 / 0.4 / 0.2, the dynamic threshold =0.616 is updated according to the three-dimensional feedback factor; The model weights are initially set to 0.3, 0.3, and 0.4, the weighted integrated anomaly score is calculated as 0.3*0.75+0.3*0.68+0.4*0.783=0.7482, the weight loss function =0.387 is calculated, the learning rate =0.01, the sparse regularization intensity parameter =0.001, and the three-model integrated weight module is output by sparse regularization update through the weight loss function as 0.28, 0.29, and 0.43; In the output decision layer, the decision unit obtains an abnormal decision (determined as abnormal) according to the weighted integrated abnormal score and the dynamic threshold The abnormal confidence 0.533 is obtained by using a Sigmoid function to calculate The explainable analysis unit determines the abnormal type (financial data abnormality) and locates the document splicing feature according to the abnormal score and the corresponding abnormal label output by the abnormal detection layer, locates the abnormal reason (1. Strong association of high-risk project documents; 2. Income value / 1500 million exceeds the threshold range / 100-1000 million; 3. Daily average visits / 25 times exceed the base / 15 times, there is a risk of data tampering) through the abnormal document splicing feature and the document topology relationship.

[0021] In a second aspect, a data anomaly identification system based on machine learning includes: A document processing module: obtains reference document data, performs document parsing and document enhancement on the reference document data to obtain standard document data; A feature extraction module: used for multi-modal feature hierarchical extraction on the standard document data to obtain multi-modal document features, and feature edge processing on the multi-modal document features to obtain combined document features; An abnormal coefficient module: used for obtaining document density and document statistical data according to the multi-modal document features, and calculating document abnormal coefficients according to the document density and the document statistical data; An abnormal identification module: used for constructing an adaptive data anomaly identification model according to the document abnormal coefficients and the combined document features, inputting to-be-identified document data into the adaptive data anomaly identification model to obtain a document data anomaly identification result; An intelligent management module: used for storing, viewing and managing the document data anomaly identification result, and performing document repair and management according to the document data anomaly identification result.

[0022] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.​​

Claims

1. A data anomaly identification method based on machine learning, characterized in that: The following steps are involved: S1. Obtain reference document data, perform document parsing and document enhancement on the reference document data to obtain standard document data; the reference document data includes document data and anomaly annotations; S2. Performing multimodal feature hierarchical extraction on the standard document data to obtain multimodal document features, and performing feature edge processing on the multimodal document features to obtain combined document features; the combined document features include document intersection features and document splicing features; S3. Obtaining document density and document statistical data based on the multimodal document features, and calculating a document anomaly coefficient based on the document density and the document statistical data; S4. Construct an adaptive data anomaly recognition model based on the document anomaly coefficient and the combined document feature, and input the document data to be recognized into the adaptive data anomaly recognition model to obtain a document data anomaly recognition result.

2. The data anomaly identification method based on machine learning according to claim 1 is characterized in that: The method for obtaining standard document data includes: Obtain reference document data, and classify the reference document data into structured documents, semi-structured documents, and unstructured documents using a BERT-based text classifier; the reference document data includes document data and anomaly annotations; the document data includes text content, metadata, and structural features; Performing typed document parsing according to the document data division result to obtain document parsing data; the document parsing data includes field value relationships, field table relationships, hierarchical paths, and document location parameters; The document parsing data is input into a context enhancer for document enhancement to obtain document enhancement data; the context enhancer includes an association mining enhancement module, a knowledge graph enhancement module, and a behavior log enhancement module; the association mining enhancement module constructs document topology relationships through explicit links and implicit associations of semi-structured documents; the knowledge graph enhancement module constructs an enterprise graph through structured documents, and injects entity links into the enterprise graph to determine document context labels; the behavior log enhancement module generates behavior vectors based on dynamically quantified access, editing, and sharing behaviors; the document enhancement data includes document topology relationships, enterprise graphs, and behavior vectors; Standard document data consists of document enhancement data and reference document data.

3. The data anomaly identification method based on machine learning according to claim 1 is characterized in that: The method for obtaining the combined document features includes: Inputting standard document data into a multimodal feature extraction model to output multimodal document features; the multimodal feature extraction model includes a text feature extraction layer, a metadata encoding layer, a structural feature conversion layer, a topological relationship feature extraction layer and a behavior vector pass-through layer; The text feature extraction layer processes the text content through a three-layer BERT model and a topic-sentiment dual channel to obtain a text feature vector; the text feature vector includes word-level anomalies, syntactic feature document semantic embedding, topic distribution vector, sentiment polarity score, and sentiment intensity value; The metadata encoding layer edits the metadata to obtain a metadata feature vector; the metadata feature vector includes a periodic feature, a version change frequency, and an embedding feature; The structural feature conversion layer processes the structural features and the structural feature analysis results to obtain the structural feature vector; The topological relationship feature extraction layer processes the document topological relationship to obtain the topological feature vector, including the graph structure feature calculation module, the neighbor feature aggregation module, the multi-scale graph neural network module and the feature fusion module; The graph structure feature calculation module calculates the multi-relationship weighted centrality, clustering coefficient and degree distribution according to the document topological relationship, and combines them to obtain the graph structure feature. The multi-relationship weight calculation expression is: ， in For relationship type and The weight between 、 For relationship type 、 The embedding representation of is the temperature parameter, which is used to control the "sharpness" of the exponential function; The neighbor feature aggregation module determines the neighbor features by calculating the semantic similarity and topological proximity between documents. The expression is: ， ， in For nodes The vector formed by the aggregation of neighbor node features, For nodes The set of neighbor nodes of Representation node 、 The node weights between is the activation function, W is a weight matrix used to perform linear transformation on the features of neighbor nodes. For nodes Features, is the feature bias term, is the activation function, Query vector for vector attention The transpose of For nodes 、 The semantic similarity between For nodes 、 The topological proximity between them is calculated using the Adamic-Adar index; The multi-scale graph neural network module obtains the graph neural network features by balancing the local and global features of the topological relationship through a gating mechanism. The expression is: ， ， ， in For nodes The graph neural network features of is the layer normalization operation, For nodes exist The hidden state of the layer, For nodes The global characteristics of is a gated recurrent unit, is the attention weight, which is used to measure the node For Node The degree of impact, For the The weight matrix of the layer, is the sigmoid activation function, is the weight matrix of global feature gating, For nodes In the hidden state of the initial layer, is a pooling operation used to aggregate the features of neighboring nodes. For nodes 's K-hop neighbor set; The feature fusion module adaptively fuses graph structure features, neighbor features, and graph neural network features to obtain a topological feature vector, which is expressed as: ， ， in For nodes The fused topological feature vector, is the adaptive weight coefficient, For nodes The graph structure characteristics of is a multi-layer perceptron, For nodes The degree, For nodes The entropy of the neighbor label distribution, For nodes The initial eigenvector of For nodes Place feature weight function; The behavior vector pass-through layer directly receives the behavior vector, performs Z-score normalization, and outputs the standard behavior vector; The text feature vector, metadata feature vector, structural feature vector, topological feature vector and standard behavior vector are respectively subjected to feature concatenation and polynomial feature crossover to obtain the combined document feature.

4. The data anomaly identification method based on machine learning according to claim 1 is characterized in that: The method for calculating document density and document statistics includes: Perform statistical analysis on document splicing features to obtain document statistical data; Extracting text feature vectors from the document splicing features, marking corresponding fields of the document content according to the text feature vectors, and using the marked fields as the first document set. Fields in the first document set that contain numerical fields and whose values ​​are outside the preset numerical range of the corresponding category are classified as the first mark, and the remaining marked fields are used as the second mark; Extracting metadata feature vectors and standard behavior vectors from the document splicing features, using the numerical features in the metadata feature vectors and the standard behavior vectors as a second document set, and using the numerical features in the second document set whose values ​​are greater than a preset threshold of the corresponding category as a third marker; Determine a first dimension based on the number of categories of the first marker, determine a first document density based on the ratio of the number of the first marker to the first dimension, determine a second dimension based on the number of categories of the second marker, and determine a second document density based on the ratio of the number of the second marker to the second dimension; the first document density is associated with abnormal text content; the second document density is associated with normal text content; Calculating the mean of the ratio of the third marker numerical feature to the corresponding category preset threshold as the first offset, determining the third dimension based on the ratio of the third marker data length to the number of third marker categories, and taking the product of the first offset and the third dimension as the third document density; the third document density is related to the abnormal text information or operation; The document density is composed of the first document density, the second document density, and the third document density.

5. The method for identifying data anomalies based on machine learning according to claim 1, characterized in that: The method for calculating the document anomaly coefficient includes: The document anomaly coefficient is calculated based on the document density and document statistics. The expression is: ， in is the document anomaly coefficient, is the first document density, is the second document density, is the third document density, N is the number of abnormal data, For the Abnormal data corresponding to document splicing features The document skewness of .

6. The method for identifying data anomalies based on machine learning according to claim 1, characterized in that: The method for obtaining a document data anomaly recognition result includes: The document density, document anomaly coefficient, combined document features, and anomaly annotations form a comprehensive document set, which is then divided into a training set and a test set at a ratio of 6:

4. The training set is used to train the adaptive data anomaly recognition model, and the test set is used to evaluate the performance of the adaptive data anomaly recognition model. The adaptive data anomaly recognition model includes an anomaly detection layer, an adaptive learning layer, and an output decision layer; the anomaly detection layer performs anomaly detection on document data to obtain anomaly scores and corresponding anomaly labels, including a reconstruction model module, a density model module, and a graph model module; the anomaly scores include a reconstruction anomaly score, a density anomaly score, and a graph anomaly score; The reconstruction model module uses a conditional variational autoencoder to detect feature-level anomalies and identify feature combinations that deviate from the normal pattern to obtain a reconstruction anomaly score. and exception labels; The density model module extracts density features, fuses Gaussian mixture model and kernel density estimation to perform mixed density estimation to identify outliers on the edge of data distribution, and outputs density anomaly scores. and exception labels; The graph model module processes the document topology relationship according to the heterogeneous graph neural network and performs abnormal state enhancement propagation to obtain the graph anomaly score. and exception labels, the expression is: ， ， ， in is the sigmoid activation function, is the transpose of the weight vector, For nodes The final hidden state of For nodes The degree, is a multi-layer perceptron, For nodes In the hidden state of the last layer, For nodes The abnormal state coefficient, For nodes Neighboring nodes The maximum abnormal state coefficient in , is the ReLU activation function, For nodes 、 The node weights between For the The weight matrix of the layer, For nodes exist The hidden state of the layer; The adaptive learning layer includes a concept drift detection unit, a dynamic threshold adjustment unit and a model weight update unit; The concept drift detection unit identifies the input data and triggers anomaly detection and model update; The dynamic threshold adjustment unit calculates a three-dimensional feedback factor based on the concept drift detection results, the anomaly score output by the anomaly detection layer, and the corresponding anomaly label. The three-dimensional feedback factor is used to optimize the anomaly decision boundary and update the dynamic threshold. The expression is: ， in is the dynamic threshold at time step t, is the standard dynamic threshold, 、 、 is the three-dimensional weight coefficient, , is the precision feedback factor, is the precision sensitivity coefficient, is the current model accuracy, is the target accuracy, is the recall rate feedback factor, is the recall sensitivity coefficient, is the target recall rate, is the recall rate of the current model, is the concept drift feedback factor, is the drift sensitivity coefficient, is the Sigmoid function, is the drift response strength, is the KL divergence at time t, is the drift detection threshold; The model weight update unit calculates the weighted integrated anomaly score based on the dynamic threshold update result, the anomaly score output by the anomaly detection layer, and the corresponding anomaly label, and performs sparse regularization update through the weight loss function to output the integrated weight of the three models. The expression is: ， ， in Weight The update amount, Including reconstruction weight, density weight and graph weight, is the learning rate, which is used to control the step size of weight update. is the sparse regularization strength parameter, Weight The symbolic function of is the weight loss function, is the true label, A weighted ensemble of the anomaly scores of the three models; The output decision layer makes an anomaly decision and outputs a document data anomaly recognition result based on the document topology relationship, the spliced ​​document features, the anomaly score output by the anomaly detection layer, the weighted integrated anomaly score and the updated dynamic threshold, including an integrated decision unit and an explainable analysis unit; the document data anomaly recognition result includes an anomaly decision, anomaly confidence, anomaly type and anomaly cause.

7. A data anomaly identification system based on machine learning, used to execute the method according to any one of claims 1 to 6, characterized in that: include: Document processing module: obtains reference document data, performs document parsing and document enhancement on the reference document data to obtain standard document data; Feature extraction module: used for performing multimodal feature hierarchical extraction on the standard document data to obtain multimodal document features, and performing feature edge processing on the multimodal document features to obtain combined document features; Anomaly coefficient module: used for obtaining document density and document statistics according to the multimodal document features, and calculating document anomaly coefficient according to the document density and the document statistics; Anomaly recognition module: used to construct an adaptive data anomaly recognition model based on the document anomaly coefficient and the combined document features, and input the document data to be recognized into the adaptive data anomaly recognition model to obtain a document data anomaly recognition result; Intelligent management module: used to store, view and manage the document data anomaly identification results, and perform document repair and management based on the document data anomaly identification results.

Images