A multi-link service data tracing technology

Abstract

The application discloses a kind of multi-link service data tracing technology, comprising: data acquisition module is used to collect metadata and service table number;Natural language processing module is used to convert the natural language text that machine cannot recognize into numerical data;Number source standardization module is used to the number source unit of data acquisition module and natural language processing module is unified standardization, and generates standard number source unit achievement library and standard service scene achievement library;Deep learning algorithm module is configured to be based on number source unit clustering result and scene clustering result, service table data is trained, and field-level data tracing work is completed;The data storage module is used to store the data generated by number source standardization module and deep learning algorithm module.The application only needs to complete the collection of table structure, without developing blood relationship analysis module for each engine and tool, can complete the feature construction of model and number source modeling work.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-link service data tracing, and particularly relates to a multi-link service data tracing technology. BACKGROUND

[0002] With the rapid development of the Internet, various application services have accumulated a large amount of data, and we have entered an era of information explosion. When a business and multiple services call each other, a service call link is formed. When the same data is added, used, and modified in different business systems, multi-link business data is generated. This will inevitably exacerbate data growth, increasing database storage pressure and operation and maintenance costs, and also leading to problems such as inconsistent data in different tables and inaccurate tracing of data sources.

[0003] Some data fields with the same name and synonym record the same kind of data, but because they are recorded in different business systems, differences will occur. At this time, it is necessary to trace the same source business data under multi-link. The popular solution is to use metadata management (Metadata management), data lineage (Datalineage) and other technical means to analyze the blood relationship of data fields from the database to the current table. For a well-structured and well-built data warehouse, this type of technology can complete the full-link data lineage analysis. However, in reality, different business systems belong to different business units and are subject to different business architectures. Therefore, business data is not simply a database call relationship, and many data replication calls are completed through means other than databases, such as data ferry or new library table, etc. Therefore, the database table blood relationship under different business systems is chaotic, and it is difficult to complete multi-link data tracing through traditional methods.

[0004] There are already many data lineage and data tracing products on the market. Well-known products include ApacheAtlas, SQLFlow, sparklogicplan, MaxComputeInformationSchema, etc. These are currently popular data lineage analysis and data tracing tools. These products basically provide metadata management, data lineage, data tracing, UI query and other functions. However, these products usually make some sacrifices in terms of depth to meet as many use cases as possible, and cannot well solve the data tracing problem after data collection in different business systems. The current commercial products have the defects of high development cost of blood relationship analysis module, the need to develop blood relationship analysis modules for each engine and tool, and the need to confirm blood relationship interfaces, not all ETL toolboxes will provide blood relationship interfaces, poor flexibility, and inability to adapt to multi-business scenario data collection data tracing tasks. SUMMARY

[0005] The present application aims at the above-mentioned problems, and provides a multi-link service data tracing technology, which solves the defects of the existing data tracing technology, such as high development cost of blood relationship analysis module, the need to develop blood relationship analysis module for each engine and tool, the need to confirm blood relationship interface, not all ETL toolboxes can provide blood relationship interface, poor flexibility, and the inability to adapt to the data tracing task after multi-service scene data collection.

[0006] The technical scheme adopted by the present application is as follows:

[0007] A multi-link service data tracing technology, comprising: a data acquisition module, configured to acquire metadata and service table data;

[0008] A natural language processing module, configured to convert natural language text that cannot be recognized by a machine into numerical data;

[0009] A number source standardization module, configured to uniformly standardize the number source units of the data acquisition module and the natural language processing module, and generate a standard number source unit achievement library and a standard service scene achievement library;

[0010] A deep learning algorithm module, configured to learn and train the service table data based on the number source unit clustering result and the scene clustering result, and complete the field-level data tracing work; and

[0011] A data storage module, configured to store the data generated by the number source standardization module and the deep learning algorithm module.

[0012] According to the present application, only the acquisition of table structure is needed, and the blood relationship analysis module does not need to be developed for each engine and tool, so that the feature construction and number source modeling of the model can be completed. Meanwhile, the adaptive deep learning model of the present application can complete the field-level blood relationship analysis of the service data generated by different engines and tools, without the need to develop a blood relationship analysis module for each engine and tool. The present application does not need to confirm the data blood relationship interface. Because the essence of the data tracing work in the present service scene is not the source of the database blood relationship, the data blood relationship interface does not need to be confirmed. The present application can adapt to the data tracing task after multi-service scene data collection, and has high flexibility. The present application also solves the defects of "data acquisition has vulnerabilities. When replaying SQL, the metadata may change, such as temporary tables may be dropped, there is no temporary user-defined function UDF, or SQL parsing fails".

[0013] Optionally, the operation method of the natural language processing module is as follows:

[0014] Preprocessing of the text;

[0015] Using open source library Jieba library, Gensim library, loading pre-training corpus, completing the word segmentation and vectorization of the text;

[0016] Using LDA algorithm to generate text keywords, using these keywords to complete the keyword association calculation with Apriori, and using this association to complete the clustering of the original unnecessary text corpus with DBSCAN algorithm, and finally generating the result corpus.

[0017] Optionally, the preprocessing includes but is not limited to deduplication, special symbol removal, and simplified Chinese to traditional Chinese conversion.

[0018] Optionally, the working method of the number source standardization module includes:

[0019] Connecting the data of each business system;

[0020] After completing the metadata collection, performing basic data cleaning and standardization on the data;

[0021] Importing the data into the natural language processing module;

[0022] Finally, two achievement libraries are generated, which are standard number source unit achievement library and standard business scenario achievement library.

[0023] Optionally, the deep learning algorithm module includes sub-modules of natural language processing, convolutional neural network, word vector similarity and word extension algorithm, semi-supervised neural network label training, gradient neural network and multi-classification algorithm.

[0024] Optionally, the working method of the deep learning algorithm module includes:

[0025] Generating word vectors of fields using Gensim;

[0026] Using convolutional neural network to pool the word vectors to form a tensor database;

[0027] Importing the gradient neural network of semi-supervised learning together with the standard number source unit achievement library and the standard business scenario achievement library, and the externally imported label library, based on the loss of classification error, performing semi-supervised training;

[0028] Finally, using the Softmax layer to complete the classification of the field data number source unit.

[0029] Optionally, the convolutional neural network uses convolutional neural network for pooling work, and the pooling work is to divide the input tensor matrix into several rectangular regions, and output the maximum value of each sub-region, so as to achieve the effect of feature compression.

[0030] Optionally, the semi-supervised gradient neural network adopts a pseudo label algorithm, and an operation method of the pseudo label algorithm is as follows:

[0031] Step 1: A pseudo label is given to unlabeled data by using the trained model; and a training method is that the model in training is used to predict the unlabeled data, and a class with the highest probability is taken as a pseudo label of the unlabeled data;

[0032] Step 2: An entropy regularization idea is used to convert the unsupervised data into a regularization term of an objective function; the unlabeled data with the pseudo label is regarded as labeled data, and then cross entropy is used to evaluate the error size.

[0033] Optionally, the semi-supervised gradient neural network is an optimization and modification based on a deep auto-encoding network and a ladder network; the ladder network is connected to a decoding layer in a transverse line connection mode for each layer of coding based on the auto-encoding network; noise is introduced in each layer of the ladder network, and a loss function of the ladder network is a total of each layer loss function C.

[0034] Optionally, the semi-supervised gradient neural network is configured as:

[0035] The highest layer of the coding layer accesses a supervised loss function, and in the training process, the unsupervised and supervised loss functions are added to be total loss, and gradient descent training is performed.

[0036] (Three) beneficial effects

[0037] 1. The application provides a brand-new data traceability solution, makes the data traceability work more convenient, and effectively solves the business pain points in the multi-link data traceability field. The system has low early development cost, does not need to develop corresponding blood relationship analysis interfaces, only needs to import metadata and field data, and can complete the data traceability work, and better stability and reliability can be brought.

[0038] 2. The application uses a pre-training corpus based on metadata training, and has stronger adaptability to metadata management scenes, and can quickly adapt to different business system metadata.

[0039] 3. The application uses a semi-supervised learning network, and only a small amount of label values can solve the training work of hundreds of times of the label amount of an unsupervised data set. The technology greatly reduces manual labeling work and saves social labor cost. DETAILED DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a logic structure diagram of the multi-link service data traceability technology of embodiment 1 of the application;

[0041] Figure 2Is the internal logical structure diagram of the natural language processing module of the multi-link service data tracing technology of embodiment 2 of the application;

[0042] Figure 3 Is the internal logical structure diagram of the number source standardization module of the multi-link service data tracing technology of embodiment 2 of the application;

[0043] Figure 4 Is the internal logical structure diagram of the deep learning algorithm module of the multi-link service data tracing technology of embodiment 2 of the application;

[0044] Figure 5 Is the structure diagram of the pooling layer of the multi-link service data tracing technology of embodiment 2 of the application;

[0045] Figure 6 Is the function schematic diagram of the forward conduction formula of the single-layer auto-encoding network of the multi-link service data tracing technology of embodiment 2 of the application;

[0046] Figure 7 Is the flow schematic diagram of the ladder network of the multi-link service data tracing technology of embodiment 2 of the application;

[0047] Figure 8 Is the logical structure diagram of the ladder network of the multi-link service data tracing technology of embodiment 2 of the application. DETAILED DESCRIPTION

[0048] The application will be described in detail below with reference to the accompanying drawings.

[0049] LDA in the application refers to implicit Dirichlet distribution.

[0050] Gensim of the application is an open source third-party Python toolkit for learning the topic vector expression of the text hidden layer from the original unstructured text in an unsupervised manner.

[0051] Embodiment 1

[0052] As shown in the figure, the technical scheme adopted by the application is as follows: Figure 1

[0053] A multi-link service data tracing technology, comprising: a data acquisition module, the data acquisition module is used for collecting metadata and service table data;

[0054] A natural language processing module, the natural language processing module is used for converting natural language text that cannot be recognized by a machine into numerical data;

[0055] A number source standardization module, the number source standardization module is used for uniformly standardizing the number source unit of the data acquisition module and the natural language processing module; ​

[0056] a deep learning algorithm module configured to learn and train the business table data based on the data source unit clustering result and the scenario clustering result, to complete the field-level data tracing work; and

[0057] a data storage module for storing the data generated by the data source standardization module and the deep learning algorithm module.

[0058] It should be noted that the data acquisition module in the embodiment mainly acquires two types of data, i.e., metadata and business table data. The metadata provides business services for the data source standardization module, and based on the calculation and analysis of the business system metadata, the clustering logic of the data source unit is generated; the business table data serves the deep learning module, and provides data support for neural network training and field classification matching.

[0059] It should be noted that the main function of the natural language processing module is to convert natural language text that cannot be recognized by a machine into numerical data. It covers regular natural language processing work such as word segmentation, word vector, part-of-speech tagging, LDA; at the same time, according to the actual scene demand, statistical learning algorithms such as association calculation Apriori and text clustering DBSCAN are added to output the standardized results.

[0060] It should be noted that the data source standardization module refers to the process work of uniformly standardizing all data source units of the business system tables covered by the multi-link business system. The module mainly includes six sub-modules of data collection, data cleaning, data standardization, natural language processing (invocation), data source clustering, and scenario clustering. The main functions are: 1) data preprocessing of metadata text, 2) completion of data source clustering with standardized data source units as index, and 3) completion of scenario clustering with standardized data source units as index.

[0061] It should be noted that the deep learning module refers to the deep learning network module designed for multi-link business data tracing. The module mainly includes six sub-modules of natural language processing, convolutional neural network (CNN), word vector similarity and word expansion algorithm (Gensim), semi-supervised neural network label training (Pseudo label), gradient neural network (Ladder Networks), and multi-classification algorithm (Softmax). The main function of the deep learning module is to learn and train the business table data based on the data source unit clustering result and the scenario clustering result, to complete the field-level data tracing work. The data source unit clustering result and the scenario clustering result of the application are respectively stored in the standardized data source unit achievement library and the standardized business scenario achievement library.

[0062] It should be noted that the data storage module refers to a data storage module supporting a multi-link service data source system, including a mainstream relational database MySQL, Redis for signal queue, a big data platform ODPS, etc. Users can select different data storage systems according to actual data volume and usage.

[0063] Embodiment 2

[0064] A multi-link service data traceability technology includes a data acquisition module for acquiring metadata and service table data;

[0065] A natural language processing module for converting natural language text that cannot be recognized by a machine into numerical data;

[0066] A data source standardization module for uniformly standardizing data sources of the data acquisition module and the natural language processing module;

[0067] A deep learning algorithm module configured to learn and train service table data based on data source unit clustering results and scene clustering results, and complete field-level data traceability work; and

[0068] A data storage module for storing data generated by the data source standardization module and the deep learning algorithm module.

[0069] The operation method of the natural language processing module is:

[0070] Pretreatment of the text;

[0071] Using open source libraries Jieba library and Gensim library, loading pre-trained corpus, completing text segmentation and vectorization work;

[0072] Using LDA algorithm to generate text keywords, using these keywords to complete keyword correlation calculation with Apriori, and using this correlation to complete clustering of original unnecessary text corpus with DBSCAN algorithm, and finally generating result corpus.

[0073] Its preprocessing includes but is not limited to deduplication, special symbol removal, and simplified Chinese to traditional Chinese conversion.

[0074] The working method of the data source standardization module includes:

[0075] Connecting data of each business system;

[0076] After completing metadata collection, performing basic data cleaning and standardization on the data;

[0077] Importing the data into the natural language processing module;

[0078] Finally, two achievement libraries are generated, namely, a standard data source unit achievement library and a standard business scenario achievement library.

[0079] The deep learning algorithm module includes sub-modules of natural language processing, convolutional neural network, word vector similarity and word extension algorithm, semi-supervised neural network label training, gradient neural network, and multi-classification algorithm.

[0080] The working method of the deep learning algorithm module includes:

[0081] The word vector of the field is generated by Gensim;

[0082] The convolutional neural network is used to pool the word vector to form a tensor database;

[0083] The gradient neural network of semi-supervised learning is imported together with the standard data source unit achievement library and the standard business scenario achievement library, and the external imported label library, based on the loss of classification error, for semi-supervised training;

[0084] Finally, the Softmax layer is used to complete the classification of the field data source unit.

[0085] The convolutional neural network uses the convolutional neural network for pooling work, which divides the input tensor matrix into several rectangular regions, outputs the maximum value of each sub-region, and achieves the effect of feature compression.

[0086] The gradient neural network of semi-supervised learning adopts a pseudo-label algorithm, and the operation method of the pseudo-label algorithm is:

[0087] Step 1: Use the trained model to give a pseudo-label to the unlabeled data; the training method is to use the model in training to predict the unlabeled data, and the class with the highest probability is used as the pseudo-label of the unlabeled data;

[0088] Step 2: Use the Entropy Regularization idea to convert the unsupervised data into a regularization term of the objective function; the unlabeled data with pseudo-labels is regarded as labeled data, and then the cross-entropy is used to evaluate the error size.

[0089] The semi-supervised gradient neural network is an optimized transformation based on the deep auto-encoding network and the ladder network; the ladder network uses horizontal connection to connect each layer of encoding to the decoding layer on the basis of the auto-encoding network; the ladder network also introduces noise at each layer, and the loss function of the ladder network is the sum of each layer loss function C.

[0090] And the semi-supervised gradient neural network is configured to:

[0091] The highest layer of the encoding layer is connected to a supervised loss function. During training, the unsupervised and supervised loss functions are added together to obtain the total loss, and gradient descent training is performed.

[0092] The natural language processing module runs throughout the entire data processing module, and its internal logic is shown in the appendix. Figure 2 As shown, before processing metadata or field data, calling this module code can automate the processing of natural language, including text preprocessing (duplicate removal, removal of special symbols, and conversion between simplified and traditional Chinese characters); using the open-source libraries Jieba and Gensim, loading a pre-trained corpus, and performing word segmentation and vectorization of the text; using the LDA algorithm to generate text topic terms, using these topic terms to perform topic term association calculations using Apriori; using this association relationship, using the DBSCAN algorithm to perform clustering of the originally unnecessary text corpus, and finally generating the result corpus.

[0093] The entire module's algorithmic structure is built upon a thorough understanding of the metadata, with clear algorithmic relationships, ensuring that the results of previous calculations can be fully utilized in subsequent algorithms. Furthermore, only the DBSCAN algorithm in the entire module utilizes unsupervised learning and requires hyperparameter tuning; the algorithmic logic of the remaining parts is clear, and the results are unique and stable.

[0094] The LDA algorithm in this embodiment, short for Latent Dirichlet Allocation, is a classic bag-of-words model in natural language processing. Its algorithm principle can be briefly summarized as follows: A document is assumed to be a set of words, with no inherent order or sequence between them. A document can contain multiple topics, and each word in the document is generated from one of these topics. The Dirichlet distribution is the conjugate prior probability distribution of the multinomial distribution. In the model, a document is generated as follows:

[0095] 1) Sample from the Dirichlet distribution α to generate the topic distribution θ of document i. i ;

[0096] 2) From the multinomial distribution θ of the topic i The topic z of the j-th word in document i is generated by sampling. i，j ;

[0097] 3) Generate topic x by sampling from the Dirichlet distribution β i，j Word distribution

[0098] 4) From the polynomial distribution of words Mid-sampling ultimately generates the word ω i，j ;

[0099] Thus the joint distribution of all visible variables and hidden variables in the model is

[0100]

[0101] The maximum likelihood estimate of the word distribution of a document is then obtained by integrating over and summing over

[0102]

[0103] The topic words for a document are then obtained as the maximum likelihood estimate of p(ω i |α,β).

[0104] Apriori in this embodiment is a kind of association rule learning algorithm, breadth-limited search and hash coefficient structure are used to effectively calculate candidate item set. Apriori algorithm uses support as the standard to judge frequent item set, the goal of the algorithm is to find the maximum K-item frequent set, that is, to find the frequent set meeting the support standard, or to find the maximum number of frequent sets.

[0105] Apriori algorithm uses iterative method, first searches candidate 1-item set and corresponding support, prunes 1-item set lower than support to get frequent 1-item set. Then connect the remaining frequent 1-item set to get candidate frequent 2-item set, filter out candidate frequent 2-item set lower than support to get real frequent 2-item set, and so on, iterate down, until no frequent k+1-item set can be found, the corresponding frequent k-item set is the output result of the algorithm.

[0106] The algorithm flow is as follows:

[0107] Input: data set D, support threshold α;

[0108] Output: the maximum frequent k-item set;

[0109] 1) Scan the entire data set to get all the data that appears, as candidate frequent 1-item set.

[0110] k=1, frequent 0-item set is empty set;

[0111] 2) Mine frequent k-item set:

[0112] a) Scan the data to calculate the support of candidate frequent k-item set;

[0113] b) remove the data set whose support is lower than the threshold value in the candidate frequent k-item set, and obtain the frequent k-item set. If the obtained frequent k-item set is empty, directly return the set of frequent k-1-item sets as the algorithm result, and the algorithm ends. If the obtained frequent k-item set has only one item, directly return the set of frequent k-item sets as the algorithm result, and the algorithm ends;

[0114] c) based on the frequent k-item set, connect to generate a candidate frequent k+1-item set;

[0115] 3) let k = k + 1, and go to step 2).

[0116] The DBSCAN algorithm in the embodiment is a density-based spatial clustering algorithm of applications with noise. The algorithm principle is to divide the points to be clustered in the sample space into core points, density reachable points and outliers, and perform clustering on the different three types of points.

[0117] 1) The DBSCAN model needs two basic parameters: maximum neighborhood ε (eps) and minimum number of points minPts required to form a high-density area;

[0118] 2) If a point p has minPts points within the distance range ε, the point p is called a core point;

[0119] 3) The points within the ε (eps) range of the core point are called directly reachable by p, that is, density reachable points;

[0120] 4) If there is a road p1, p2, p3,... p n , p1 = p and p n = q, and each p i+1 is directly reachable by p i , then q is called reachable by p;

[0121] 5) All points not reachable by any point are called outliers;

[0122] If p is a core point, it forms a cluster with all the points reachable by it (including core points and non-core points), each cluster has at least one core point, and non-core points can also be part of the cluster, but it is in the "edge" position of the cluster because it cannot reach more points.

[0123] Each cluster in the DBSCAN algorithm model meets two properties:

[0124] 6) Each two points in the same cluster are connected to each other;

[0125] 7) If point p is reachable by point q in a cluster, then p is in the cluster to which q belongs.

[0126] The number source standardization module in the embodiment is different from the traditional data traceability system. The multi-link business data traceability needs to clearly define the standard number source to ensure that the number source unit of the same business attribute is unique and standardized. Therefore, before data traceability, the number source needs to be standardized by using the number source standardization module. As shown in the accompanying Figure 3 The number source standardization module first interfaces the data of each business system, performs basic data cleaning and standardization on the data after completing metadata collection, and then imports the data into the natural language processing module. Finally, two achievement libraries are generated, namely the standard number source unit achievement library and the standard business scenario achievement library.

[0127] The deep learning network structure in the embodiment is shown in Figure 4 The field data of each business system, i.e., the field data that needs to be traced, is used. Like the number source standardization, preprocessing and natural language processing are needed, but the word vector processing of this module is different from that of the number source standardization module. After the Gensim generates the word vector of the field, since the field word vector is a high-dimensional sparse tensor, a convolutional neural network is designed to pool this part of the word vector to form a tensor database. The standard number source unit achievement library and the standard business scenario achievement library, and the externally imported label library (Pseudo label) are imported into the gradient neural network of semi-supervised learning together, based on the loss of classification error, semi-supervised training is performed. Finally, a Softmax layer is used to complete the classification of the field data number source unit.

[0128] The text mainly uses the convolutional neural network for pooling work, so here the important role of the pooling layer in the high-dimensional sparse tensor is specifically introduced. Pooling is actually a kind of nonlinear form of down-sampling, and the text selects "max pooling" as the down-sampling scheme. The specific algorithm is to divide the input tensor matrix into several rectangular regions, and output the maximum value of each sub-region, so as to achieve the effect of feature compression.

[0129] The pooling operation provides another form of translation invariance. Because the convolution kernel is a feature finder, local features in the tensor matrix can be easily found through the convolution layer, but the features found by the convolution layer are often too accurate, and through the pooling layer we can reduce the sensitivity of the convolution layer to local features while reducing the dimension.

[0130] The pooling layer calculates the output on one pooling window at a time, and then moves the pooling window according to the stride. Figure 5is a commonly used pooling layer, stride is 2, and the pooling window is a two-dimensional maximum pooling layer with (2X2). Every 2 elements are divided into a (2X2) block from the matrix, and then the maximum value of the 4 numbers in each block is taken.

[0131] Pesudo Label, which directly translates to pseudo-label algorithm, is a simple and effective semi-supervised learning method for deep neural networks. The network uses a supervised method of training with labeled and unlabeled data. For unlabeled data, the class with the highest prediction probability is selected as the pseudo label; L signature, like real labels, is used.

[0132] The steps of Pseudo-Labels are:

[0133] Step 1: Use the trained model to give the unlabeled data a pseudo label. The method is very direct: use the model in training to predict the unlabeled data, and take the class with the highest probability as the pseudo label of the unlabeled data;

[0134] Step 2: Use the idea of entropy regularization to convert unsupervised data into a regularization term of the objective function. In practice, the unlabeled data with pseudo labels are treated as labeled data, and then the cross-entropy is used to evaluate the error size.

[0135] The semi-supervised gradient neural network (Semi-supervised ladder networks) in this embodiment is an optimized modification based on deep auto-encoding networks and ladder networks.

[0136] The forward conduction formula of the single-layer auto-encoding network is:

[0137] h(t)=f(x(t);ξ f )

[0138]

[0139] x(t) represents the input data, and h(t) represents the hidden layer neuron. The f function is the encoding mapping function, and the g function represents the decoding mapping function, as shown in the schematic diagram. Figure 6 .

[0140] The ladder network uses a horizontal connection method to connect each layer of the encoding to the decoding layer based on the auto-encoding network. The ladder network also introduces noise at each layer. The loss function of the ladder network is the sum of each layer loss function C. The schematic diagram is shown in Figure 7 .

[0141] The biggest difference between the semi-supervised gradient neural network and the ladder network is that a supervised loss function is connected to the highest layer of the encoding layer, and the unsupervised and supervised loss functions are added together as the total loss in the training process, and gradient descent training is performed.

[0142] Given N labeled sample data and M unlabeled sample data, the number of labeled sample data is much smaller than that of unlabeled data. The model aims to learn a function to determine the probability of data label P(y|x) respectively. As shown in the following formula (1), in the ladder network, the function is a deep denoising autoencoder, noise is added to all hidden layers, and the final loss function is the cross-entropy loss function of the labeled sample data and the Euclidean error loss function of the reconstruction of the noise decoder of the unlabeled sample data: Figure 8

[0143]

[0144] The Softmax algorithm of the embodiment is a generalization and deduction of the logistic regression algorithm, and its essence is the gradient logarithmic normalization of the discrete probability distribution of a limited term. Therefore, the Softmax algorithm has a wide application in various probability-based multi-classification problem methods including multinomial logistic regression, multinomial linear discriminant analysis, naive Bayes classifier and artificial neural network. In particular, in multinomial logistic regression and linear discriminant analysis, the input of the function is the result obtained from K different linear functions, and the probability of the sample vector x belonging to the jth classification is:

[0145]

[0146] The above only describes the preferred embodiments of the present application, and does not limit the patent protection scope of the present application, any equivalent structural transformation, direct or indirect application in other related technical fields, are also included in the protection scope of the present application.​

Claims

1. A multi-link business data tracing technology, characterized in that, include: The data acquisition module is used to collect metadata and business table data; The natural language processing module is used to convert natural language text that cannot be recognized by machines into numerical data; The data source standardization module is used to standardize the data source units of the data acquisition module and the natural language processing module, and generate a standard data source unit result library and a standard business scenario result library. The deep learning algorithm module is configured to: learn and train on business table data based on the clustering results of data source units and the clustering results of scenarios, and complete the field-level data tracing work; as well as The data storage module stores data generated by the data source standardization module and the deep learning algorithm module. The deep learning algorithm module comprises six sub-modules: natural language processing, convolutional neural networks, word vector similarity and word expansion algorithms, semi-supervised neural network label training, gradient neural networks, and multi-classification algorithms. The working method of the deep learning algorithm module includes: Generating word vectors for a field using Gensim; The word vectors in this part are pooled using a convolutional neural network to form a tensor database; Along with the standard data source unit result library, the standard business scenario result library, and the externally imported label library, a semi-supervised learning gradient neural network is imported, and semi-supervised training is performed based on the loss of classification error. The Softmax layer is used to classify the source units of the field data. Convolutional neural networks (CNNs) use pooling, which divides the input tensor matrix into several rectangular regions and outputs the maximum value for each sub-region, thus achieving feature compression. The forward propagation formula for a single-layer autoencoder network is: Indicates input data, Represents hidden layer neurons. The function is an encoding mapping function. The function represents the decoding mapping function; The staircase network, based on the autoencoder network, connects each layer of the encoding layer to the decoding layer using horizontal lines. The staircase network also introduces noise into each layer. The loss function of the staircase network is the sum of the loss functions C of each layer. Given N labeled data samples and M unlabeled data samples, where the number of labeled samples is much smaller than the number of unlabeled samples, the model aims to learn a function to determine the probability of labeling the data. In a staircase network, this function is a deep denoising autoencoder. Noise is added to all hidden layers, and the final loss function is the sum of the cross-entropy loss function for labeled sample data and the Euclidean error loss function after reconstruction by the noisy decoders of each layer for unlabeled sample data. ； And the semi-supervised gradient neural network is configured as follows: The highest layer of the encoding layer is connected to a supervised loss function. During training, the unsupervised and supervised loss functions are added together to obtain the total loss, and gradient descent training is performed. The operation method of the natural language processing module is as follows: Preprocessing of the text; Using the open-source libraries Jieba and Gensim, and loading a pre-trained corpus, we completed the word segmentation and vectorization of the text. The LDA algorithm is used to generate text topic terms. These topic terms are then used to calculate topic term relevance using the Apriori algorithm. This relevance is then used to perform clustering of the text corpus, which originally did not require manual feature annotation, using the DBSCAN algorithm to generate the final corpus. Preprocessing includes, but is not limited to, deduplication, removal of special symbols, and simplified / traditional character conversion. The data source standardization module's working method includes: Connect with data from various business systems; After completing the metadata collection, basic data cleaning and standardization work is carried out on the data; Import the data into the natural language processing module; Ultimately, two result libraries are generated: a standard data source unit result library and a standard business scenario result library. The semi-supervised learning gradient neural network uses a pseudo-label algorithm, and the operation method of the pseudo-label algorithm is as follows: The trained model is used to assign a pseudo-label to the unlabeled data; the training method is to use the trained model to predict the unlabeled data and use the category with the highest probability as the pseudo-label of the unlabeled data. By applying the idea of ​​entropy regularization, unsupervised data is transformed into a regularization term of the objective function; unlabeled data with pseudo-labels is treated as labeled data, and then cross-entropy is used to evaluate the error magnitude. The Dirichlet distribution is the conjugate prior probability distribution of the multinomial distribution. In the model, a document is generated as follows: 1) From the Dirichlet distribution Mid-sampling to generate documents Theme distribution ; 2) From the multinomial distribution of the topic Mid-sampling to generate documents No. The theme of each word ; 3) From the Dirichlet distribution Mid-sampling generates topics Word distribution ; 4) From the polynomial distribution of words Mid-sampling ultimately generates words ; Therefore, the joint distribution of all visible and hidden variables in the entire model is: The maximum likelihood estimate of the word distribution of a document can be obtained by integrating and summing the hidden variables in the above equation. according to The maximum likelihood estimation yields the subject terms of the document. The Apriori algorithm uses an iterative method, and its algorithm flow is shown below: Input: Data set D, support threshold ; Output: The largest frequent k-itemset; 1) Scan the entire dataset to obtain all the data that have appeared, and use them as candidate frequent 1-itemsets; k=1, and frequent 0-itemsets are empty sets; 2) Mining frequent k-itemsets: a) Calculate the support of candidate frequent k-itemsets from the scanned data; b) Remove the datasets whose support is below the threshold from the candidate frequent k-itemsets to obtain frequent k-itemsets; if the obtained frequent k-itemsets are empty, directly return the set of frequent k-1 itemsets as the algorithm result and the algorithm ends; if the obtained frequent k-itemsets have only one item, directly return the set of frequent k-itemsets as the algorithm result and the algorithm ends. c) Generate candidate frequent k+1 itemsets by connecting them based on frequent k-itemsets; 3) Let k = k + 1, then proceed to step 2). The DBSCAN algorithm divides the points in the sample space that need to be clustered into three different classes: core points, density-reachable points, and outliers, and then performs clustering on these three classes: 1) The DBSCAN model requires two basic parameters: the maximum neighborhood and the minimum number of points required to form a high-density region; 2) If a point has a range of points within it, then that point is called the core point; 3) Points within the core area are called directly reachable, i.e., density-reachable points; 4) If there exists a road with points q and q, and each point is directly reachable by q, where all points on the road except q are core points, then the road is said to be reachable by q. 5) All points that cannot be reached from any other point are called outside points; If a point is a core point, it forms a cluster with all points reachable from it. All reachable points include both core and non-core points. Each cluster has at least one core point. Non-core points can also be part of a cluster, but they are located at the "edge" of the cluster because they cannot reach more points. Each cluster in the DBSCAN algorithm model satisfies two properties: 6) Every two points in the same cluster are connected to each other; 7) If a point is reachable from points in a cluster, then it belongs to that cluster.

Images