Data analysis method, device, server and readable storage medium

Abstract

The embodiments of the present invention relate to the field of data analysis, and disclose a data analysis method, device, server, and readable storage medium. In the present invention, a data table call process sequence is obtained; a new call sequence with the same sequence length is generated according to the data table call process sequence; the new call sequence is input into a preset neural network for training to obtain a relationship matrix; the relationship matrix is ​​processed to obtain the relationship value between the data tables; and a hot and cold relationship heat map is constructed according to the relationship value between the data tables to analyze the data. The call relationship between the data tables is determined more accurately to improve the accuracy of the analysis conclusion.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data analysis, and in particular, to a data analysis method, device, server, and readable storage medium. Background Art

[0002] In the data warehouse, the original data is generally synchronized from the database of each business system. As the data warehouse's source layer data table, it is the basis for the subsequent construction of the upper-layer data model of the data warehouse. In order to better manage data assets, it is necessary to determine the popularity of each source layer data table and focus on monitoring the data tables with high popularity (i.e. high importance).

[0003] Related technologies generally judge the popularity of a data table by the frequency with which the data table is called during data processing. However, the method of only counting the frequency of calls is relatively simple and insufficient to accurately reflect the popularity of the data table, which may lead to errors in judging the importance of the data table. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a data analysis method, device, server and readable storage medium to more accurately determine the calling relationship between data tables and improve the accuracy of the analysis conclusion.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides a data analysis method, comprising the following steps:

[0006] Get the data table calling process sequence;

[0007] Generate a new calling sequence with the same sequence length according to the calling process sequence of the data table;

[0008] Inputting the new calling sequence into a preset neural network for training to obtain a relationship matrix;

[0009] Process the relationship matrix to obtain relationship values ​​between data tables;

[0010] According to the relationship values ​​between the data tables, a hot-cold relationship heat map is constructed to analyze the data.

[0011] An embodiment of the present invention further provides a data analysis device, comprising:

[0012] The acquisition module is used to obtain the data table calling process sequence;

[0013] A sequence processing module, used for generating a new calling sequence with the same sequence length according to the calling process sequence of the data table;

[0014] A network training module, used for inputting the new calling sequence into a preset neural network for training to obtain a relationship matrix;

[0015] A relationship calculation module, used to process the relationship matrix and obtain the relationship values ​​between the data tables;

[0016] The analysis module is used to construct a cold-hot relationship heat map according to the relationship values ​​between the data tables to analyze the data.

[0017] An embodiment of the present invention further provides a server, comprising: at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the data analysis methods.

[0020] An embodiment of the present invention further provides a computer-readable storage medium, and when the computer program is executed by a processor, any one of the data analysis methods described above is implemented.

[0021] Compared with the related art, the embodiment of the present invention does not simply use the number of calls as the basis for judging the relationship between data tables. The relationship between data tables is obtained through a pre-set neural network, which is more accurate than the traditional closeness judgment method, and avoids the situation where some data tables are not important despite a large number of calls. And the relationship values ​​between the data tables are used to form a heat map, and the close relationship between the data table time can be seen through the heat map, making the analysis of the data clearer and more intuitive. Mathematical methods are used to quantify the call relationship closeness information between each data table, and the relationship value between each data table is obtained through calculation. This relationship value not only includes the call frequency of the data table, but more importantly, it also includes the close co-occurrence between each data table. The heat table can more accurately reflect the heat level of the data table, providing a basis for subsequent key monitoring and guarantee.

[0022] In addition, the data analysis method provided by the embodiment of the present invention, after inputting the calling new sequence into a preset neural network for training, also includes: reducing the loss value of the neural network by a stochastic gradient descent method.

[0023] In addition, the data analysis method provided in an embodiment of the present invention, wherein the calling new sequence is input into a preset neural network for training to obtain a relationship matrix, includes: when the loss value is less than a preset loss threshold, the matrix generated by the neural network is used as the relationship matrix.

[0024] In addition, the data analysis method provided in an embodiment of the present invention, wherein the calling new sequence is input into a preset neural network for training to obtain a relationship matrix, includes: when the loss value is less than a preset loss threshold, the matrix generated by the neural network is used as the relationship matrix.

[0025] In addition, the data analysis method provided in an embodiment of the present invention processes the relationship matrix to obtain the relationship values ​​between data tables, including: performing a dot multiplication operation on the vectors in the relationship matrix, and obtaining the dot multiplication result as the relationship value, wherein each vector in the relationship matrix corresponds to one data table.

[0026] In addition, the data analysis method provided by the embodiment of the present invention, obtaining the loss value includes: processing the data table outside the intermediate data table of the new sequence being called to generate a predicted value of the intermediate data table; inputting the predicted value of the intermediate data table and the encoded value of the intermediate data table into the loss function to generate the loss value.

[0027] In addition, in the data analysis method provided by the embodiment of the present invention, the relationship value represents the closeness of the calling relationship between the data tables.

[0028] In addition, the data analysis method provided in an embodiment of the present invention generates a new calling sequence with a consistent sequence length based on the data table calling process sequence, including: performing a sliding window algorithm on the data table calling process sequence to generate a new calling sequence, wherein the sequence length of the new calling sequence is a preset sliding window value. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0030] Figure 1 is a flow chart of a data analysis method provided by a first embodiment of the present invention;

[0031] Figure 2 is a heat map of the data analysis method provided by the first embodiment of the present invention;

[0032] Figure 3 is a matrix C of the data analysis method provided by the second embodiment of the present invention;

[0033] Figure 4 is the matrix C2 of the data analysis method provided by the second embodiment of the present invention;

[0034] Figure 5is a matrix of the data analysis method provided by the second embodiment of the present invention;

[0035] Figure 6 is a schematic structural diagram of a data analysis device provided by a third embodiment of the present invention;

[0036] Figure 7 It is a schematic diagram of the structure of a server provided by the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the following will be described in detail with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that in the various embodiments of the present invention, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented.

[0038] The following embodiments are divided for the convenience of description and shall not constitute any limitation on the specific implementation of the present invention. The embodiments may be combined with each other and referenced to each other without contradiction.

[0039] The first embodiment of the present invention relates to a data analysis method. The specific process is as follows Figure 1 shown.

[0040] Step 101, obtaining a data table calling process sequence;

[0041] A large number of data tables are stored in the data warehouse. In the processing flow of the data warehouse, data tables call each other. The calling relationship between data tables in each data processing flow is extracted. For example, in a data processing flow, three source layer data tables A, B, and C are required. ABC means that in this data calling flow, data table A is called first, then data table B is called, and finally data table C is called. The number of data calling flows is the same as the number of calling flow sequences.

[0042] Step 102, generating a new calling sequence with the same sequence length according to the data table calling process sequence;

[0043] The data table calling process sequence is processed by a sliding window algorithm to generate a new calling sequence, wherein the sequence length of the new calling sequence is a preset sliding window value.

[0044] Specifically, a sliding window value is set, and the sliding window value is an odd number, so that the intermediate data table for calling the new sequence can be determined.

[0045] The calling program sequence is calculated using a sliding window algorithm according to the sliding window value to obtain a new calling sequence.

[0046] For example, the data table call sequence is ABC, ADEGJ, BCFH, where A, B, C, D, E, F, G, H, J are the source layer data tables. If the sliding window value is set to 3, the new call sequence can be ABC, ADE, DEG, EGJ, etc.

[0047] Step 103, inputting the new calling sequence into a preset neural network for training to obtain a relationship matrix;

[0048] The data table in the new sequence is encoded using onehot encoding to obtain a data table vector, and one dimension of each vector is 1, and the rest are 0. For example, assuming there are M data tables, M = 5, the data table vector can be (1,0,0,0,0), (0,1,0,0,0), (0,0,1,0,0), (0,0,0,1,0) or (0,0,0,0,1).

[0049] Randomly initialize matrix C and matrix C2, with sizes of MxK and KxM respectively, and the values ​​of the matrices are random values ​​between 0 and 1.

[0050] Specify the intermediate data table that calls the new sequence as the predicted value, and the table around the intermediate data table as the input value, that is, use the data table outside the intermediate data table of the sliding window to predict the intermediate data table.

[0051] Take calling the new sequence ABC as an example, where B corresponds to the predicted value, and A and C correspond to the input values.

[0052] Onehot encodes the data table in the new sequence, multiplies the obtained values ​​by matrix C, and then averages the result vectors. Multiply the average value by matrix C2, and then use the softmax function on the multiplication result to generate the prediction value of the intermediate data table.

[0053] The predicted value of the intermediate data table and the intermediate data table after onehot encoding are input into the cross entropy loss function to generate a loss value.

[0054] Then, according to the training method of stochastic gradient descent, the neural network updates the values ​​of matrix C and matrix C2 through back propagation, so that the loss value continues to decrease.

[0055] When the loss value is less than the preset loss threshold, it is determined that the neural network training has reached convergence. At this time, the relatively stable matrix generated by the neural network is the relationship matrix.

[0056] In a specific example, in calling the new sequence ABC, assuming M=5, the onehot encoding of data table A is (1,0,0,0,0), the onehot encoding of data table B is (1,0,0,0,0), and the encoding of data table C is (0,0,1,0,0).

[0057] A(1,0,0,0,0) multiplied by matrix C(MxK) gets (0.1,0.3,0.5,0.2,0.1), B(1,0,0,0,0) multiplied by matrix C(MxK) gets (0.9,0.7,0.3,0.8,0.6);

[0058] Taking the average of (0.1, 0.3, 0.5, 0.2, 0.1) and (0.9, 0.7, 0.3, 0.8, 0.6) we get (0.5, 0.5, 0.4, 0.5, 0.35);

[0059] Multiply (0.5, 0.5, 0.4, 0.5, 0.35) by matrix C2 (KxM) and perform softmax function calculation to obtain the predicted values ​​of the intermediate data table (0.2, 0.9, 0.2, 0.3, 0.1).

[0060] Use the cross entropy loss function to calculate (0.2, 0.9, 0.2, 0.3, 0.1) and B(1, 0, 0, 0, 0) to obtain the loss value (e.g. 1.2).

[0061] All the data in the new sequence are repeatedly input into the neural network in a random order for training. According to the training method of stochastic gradient descent, the neural network updates the values ​​of matrix C and matrix C2 by back propagation, so that the values ​​of matrix C and matrix C2 are continuously updated, and finally the loss value reaches a relatively stable and small value (such as 0.001). When this value is less than the preset loss threshold, it is considered that the neural network training has reached convergence, and the matrix generated by the neural network tends to be stable. The matrix at this time is used as the relationship matrix.

[0062] Step 104, processing the relationship matrix to obtain relationship values ​​between data tables;

[0063] Perform dot product on the vectors in the relationship matrix. The dot product value represents the relationship between the two vectors, which is the relationship value. The relationship value is a value greater than -1 and less than 1. The closer it is to 1 (that is, the smaller the difference between the relationship value and 1), the closer the calling relationship between the data tables is. For example, the dot product result of the vectors A (0.83, 0.55, 0.91, 0.96, 0.71) and B (0.56, 0.74, 0.53, 0.62, 0.52) is 0.96, which is close to 1. Therefore, it can be considered that the relationship between data tables A and B is very close.

[0064] Step 105, constructing a hot-cold relationship thermodynamic diagram according to the relationship values ​​between the data tables to analyze the data.

[0065] The result of two-way multiplication (relationship value) forms a hot-cold relationship heat map, such as Figure 2 As shown in the figure, the diagonal line is the correlation between the data table itself, which is always 1. In the hot-cold relationship heat map, the lighter the color, the closer the call relationship. Through the relationship value between data tables and the hot-cold relationship heat map, the call relationship between data tables can be analyzed. The data tables with lighter colors can be monitored and protected in key areas to facilitate data maintenance.

[0066] Compared with the related art, the embodiment of the present invention does not simply use the number of calls as the basis for judging the relationship between data tables. The relationship between data tables is obtained through a pre-set neural network, which is more accurate than the traditional closeness judgment method, and avoids the situation where some data tables are not important despite a large number of calls. And the relationship values ​​between the data tables are used to form a heat map, and the close relationship between the data table time can be seen through the heat map, making the analysis of the data clearer and more intuitive. Mathematical methods are used to quantify the call relationship closeness information between each data table, and the relationship value between each data table is obtained through calculation. This relationship value not only includes the call frequency of the data table, but more importantly, it also includes the close co-occurrence between each data table. The heat table can more accurately reflect the heat level of the data table, providing a basis for subsequent key monitoring and guarantee.

[0067] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0068] The second embodiment of the present invention relates to a data analysis method. Assume that the number of data tables is M, M=10; the dimension selected by the data table vector is K, K=5, that is, there are 10 data tables in total, and each data table is represented by a 5-dimensional vector. The matrix C is initialized to a 10x5 matrix using random values ​​of 0 to 1. Figure 3 As shown. The matrix C2 is randomly initialized to a 5x10 matrix as shown Figure 4 shown.

[0069] Get the first row of data ABC in the new sequence and input the values ​​of A and C to the neural network;

[0070] Assume that the onehot encoding of A is [1,0,0,0,0,0,0,0,0,0], and the onehot encoding of C is [0,0,1,0,0,0,0,0,0,0], according to the neural network structure diagram, the onehot encoding of A is multiplied by the matrix C to obtain [0.59, 0.29, 0.77, 0.01, 0.23];

[0071] The onehot encoding of C is multiplied by the matrix C to get [0.24, 0.8, 0.17, 0.3, 0.47];

[0072] Taking the average of these two vectors, we get [0.42, 0.54, 0.47, 0.16, 0.35];

[0073] Then multiply this vector by matrix C2 to get [1.02, 0.94, 1.06, 1.31, 0.8, 0.89, 0.87, 0.83, 0.88, 0.79];

[0074] Then input this vector into the softmax function to obtain [0.11, 0.1, 0.11, 0.14, 0.09, 0.09, 0.09, 0.09, 0.09, 0.09], which is the predicted value of the intermediate data table;

[0075] Then the onehot encoding of B [0,1,0,0,0,0,0,0,0,0] and the predicted value of the intermediate data table are input into the cross entropy loss function to obtain the loss value.

[0076] Then, according to the training method of stochastic gradient descent, the neural network updates the values ​​of matrix C and matrix C2 through back propagation, so that the loss value continues to decrease.

[0077] Furthermore, all the data in the new sequence P are repeatedly input into the neural network for training in a random order, so that the values ​​of the matrix C and the matrix C2 are continuously updated, and finally the loss value reaches a relatively stable and smaller value, at which point the neural network training reaches convergence.

[0078] After the training reaches convergence, a relatively stable matrix C is obtained. The matrix C is a dense vector representation of M data tables. Each table is a K-dimensional vector, and this K-dimensional vector contains the call relationship information between the data tables. Figure 5 shown.

[0079] exist Figure 5 In , each row is a k=5-dimensional vector, which represents the vector representation of a data table. There are 10 rows in total, indicating that there are 10 data tables in total, including data table A, data table B, and data table C.

[0080] The association value between two data tables is calculated by dot multiplication between vectors, and the pairwise call association matrix of all data tables is obtained to form a heat map as shown below: Figure 2 shown.

[0081] Compared with the related art, this embodiment selects the call order of data tables in the existing data warehouse to train the neural network, and finally obtains the K-dimensional vector representation of each data table, and the vector contains the call relationship closeness information between the data tables. Through the operation of this K-dimensional vector, the heat map between each data table can be obtained as the basis for subsequent key monitoring and protection.

[0082] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0083] A third embodiment of the present invention relates to a data analysis device, such as Figure 6 As shown, including:

[0084] An acquisition module 601 is used to acquire a data table calling process sequence;

[0085] A sequence processing module 602 is used to generate a new calling sequence with the same sequence length according to the data table calling process sequence;

[0086] A network training module 603 is used to input the new calling sequence into a preset neural network for training to obtain a relationship matrix;

[0087] A relationship calculation module 604 is used to process the relationship matrix to obtain relationship values ​​between data tables;

[0088] The analysis module 605 is used to construct a hot-cold relationship heat map according to the relationship values ​​between the data tables to analyze the data.

[0089] It is not difficult to find that this embodiment is a system embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.

[0090] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other units in this embodiment.

[0091] A fourth embodiment of the present invention relates to a server, such as Figure 7 As shown, including:

[0092] At least one processor 701; and a memory 702 communicatively connected to the at least one processor 701;

[0093] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the data calling methods.

[0094] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus links various circuits of one or more processors and memories together. The bus can also link various other circuits such as peripherals, voltage regulators, and power management circuits together, which are all well known in the art, so they are not further described in this article. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0095] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0096] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0097] Those skilled in the art will appreciate that the above-mentioned embodiments are specific examples for implementing the present invention, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A data analysis method, characterized in that: include: Get the data table calling process sequence; According to the data table calling process sequence, a sliding window algorithm is calculated according to a preset sliding window value to obtain a new calling sequence, wherein the lengths of the sequences in the new calling sequence are consistent, and the intermediate data table of the new sequence is called only when the sliding window value is an odd number; Input the calling new sequence into a preset neural network for training, encode the data table in the calling new sequence to obtain a data table vector, form a first matrix M×K and a second matrix K×M according to the number of data tables M and the vector K of the data table, multiply the data tables around the intermediate data table of the calling new sequence as input values ​​by the first matrix respectively, average the multiplication result vectors, multiply the average value by the second matrix, and input the multiplication result into a softmax function to generate a prediction value of the intermediate data table; input the prediction value of the intermediate data table and the encoded value of the intermediate data table into a loss function to generate a loss value; when the loss value is less than a preset loss threshold, use the first matrix generated by the neural network as a relationship matrix; Performing a dot multiplication operation on the vectors in the relationship matrix, and obtaining a relationship value between the data tables based on the dot multiplication result; the relationship value indicates the closeness of the calling relationship between the data tables; According to the relationship values ​​between the data tables, a hot-cold relationship heat map is constructed to analyze the data.

2. The data analysis method according to claim 1, characterized in that: After generating the loss value, the method further includes: The loss value of the neural network is reduced by stochastic gradient descent.

3. The data analysis method according to claim 1, characterized in that: The processing of the relationship matrix to obtain the relationship values ​​between the data tables includes: A dot product operation is performed on the vectors in the relationship matrix, and the obtained dot product result is used as the relationship value, wherein any vector in the relationship matrix corresponds to one data table.

4. The data analysis method according to claim 1, characterized in that: The relationship value indicates the closeness of the calling relationship between the data tables.

5. The data analysis method according to claim 1, characterized in that: The length of each sequence in the called new sequence is a preset sliding window value.

6. A data analysis device, characterized in that: include: The acquisition module is used to obtain the data table calling process sequence; A sequence processing module is used to call the process sequence according to the data table, perform sliding window algorithm calculation according to a preset sliding window value, and obtain a new calling sequence, wherein the lengths of the sequences in the new calling sequence are consistent, and the intermediate data table of the new sequence is called only when the sliding window value is an odd number; A network training module is used to input the calling new sequence into a preset neural network for training, encode the data table in the calling new sequence to obtain a data table vector, form a first matrix M×K and a second matrix K×M according to the number of data tables M and the vector K of the data table, multiply the data tables around the intermediate data table of the calling new sequence as input values ​​by the first matrix respectively, average the multiplication result vectors, multiply the average value by the second matrix, and input the multiplication result into a softmax function to generate a prediction value of the intermediate data table; input the prediction value of the intermediate data table and the encoded value of the intermediate data table into a loss function to generate a loss value; when the loss value is less than a preset loss threshold, use the matrix generated by the neural network as a relationship matrix; A relationship calculation module, used to perform a dot multiplication operation on the vectors in the relationship matrix, and obtain a relationship value between the data tables based on the dot multiplication result; the relationship value indicates the closeness of the calling relationship between the data tables; The analysis module is used to construct a hot-cold relationship heat map according to the relationship values ​​between the data tables to analyze the data.

7. A server, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data analysis method as described in any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data analysis method according to any one of claims 1 to 5 is implemented.

Images