Data distribution analysis method and device, equipment and medium

Abstract

The invention relates to the technical field of data processing, in particular to a data distribution analysis method and device, equipment and a medium. The method comprises the following steps: acquiring each data set corresponding to data stored in different storage environments, respectively counting a first occurrence frequency of each group of data in each data set, and determining a first data distribution matrix of the data set according to the first occurrence frequency of each group of data in the data set for each data set; and extracting target features corresponding to each first data distribution matrix based on a pre-trained feature extraction model, determining data distribution features of each data set, and determining whether data distribution between the data sets is similar based on the similarity between the target features. According to the method, the data distribution condition of each data set is determined according to the occurrence frequency of each group of data in each data set, so that whether data distribution among different data sets is consistent or not is determined, and rapid data distribution analysis is realized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data distribution analysis method, device, equipment and medium. Background Art

[0002] Due to external supervision or internal control requirements, in some cases, data in the production environment cannot be easily copied to the test environment for testing. For example, some data cannot be moved or must be desensitized before being moved. In addition, the data carrying capacity of the test environment and the production environment is different. Even if the data in the production environment can be directly copied to the test environment, the test environment cannot carry such a huge amount of data.

[0003] In order to conduct corresponding tests in the test environment, it is necessary to construct data similar to that in the production environment in the test environment, and simulate the actual operation conditions as much as possible during the test process. In the related art, when constructing data similar to that in the production environment, it is usually necessary to manually perform multi-dimensional statistics on the data distribution in the production environment, and construct corresponding data based on the statistical results. If the data distribution of the data constructed in the test environment is different from that in the production environment, then the test results must be inaccurate.

[0004] Therefore, how to determine whether the data distribution is consistent becomes an urgent problem to be solved. Summary of the invention

[0005] The present application provides a data distribution analysis method, device, equipment and medium for determining whether the data distribution is consistent.

[0006] The present application provides a data distribution analysis method, the method comprising:

[0007] Obtaining each data set to be subjected to data distribution analysis, and respectively counting the first occurrence frequency of each group of data in each data set, wherein the data of each data set is stored in different storage environments;

[0008] For each data set, determining a first data distribution matrix of the data set according to a first occurrence frequency of each group of data in the data set;

[0009] Extracting target features corresponding to each first data distribution matrix based on a pre-trained feature extraction model;

[0010] Based on the similarity between each of the target features, it is determined whether the data distribution between each of the data sets is similar.

[0011] Furthermore, the respectively counting the first occurrence frequency of each group of data in each data set includes:

[0012] For each group of data in each data set, each data in the data group is divided into corresponding sub-data groups based on the grouping strategy of the data group;

[0013] According to a first quantity of data included in each sub-data group and a second quantity of data included in the data set, a first occurrence frequency corresponding to each sub-data group of the data set is determined.

[0014] Further, determining a first data distribution matrix of the data set according to a first occurrence frequency of each group of data in the data set includes:

[0015] For each data group of the data set, according to the first occurrence frequency of each sub-data group of the data group, a preset number of target occurrence frequencies are selected;

[0016] According to the magnitude of each target occurrence frequency, the target occurrence frequencies are sequentially filled into the preset matrix template to obtain a second data distribution matrix of the data group;

[0017] A first data distribution matrix of the data set is determined according to the second data distribution matrix of each data group of the data set.

[0018] Furthermore, the extracting target features corresponding to each first data distribution matrix based on the pre-trained feature extraction model includes:

[0019] Input any first data distribution matrix into the input layer for data format conversion to obtain a first eigenvector;

[0020] Inputting the first feature vector into the embedding layer for mapping processing to obtain a dense vector;

[0021] Inputting the dense vector into a feature processing layer to obtain a second feature vector, wherein the feature processing layer includes at least one convolutional layer and at least one global maximum pooling layer;

[0022] Inputting the second feature vector into at least one fully connected layer for feature extraction to obtain a third feature vector;

[0023] The output layer outputs the third eigenvector in a set format to obtain the target feature corresponding to the first data distribution matrix.

[0024] Further, if the feature processing layer includes multiple convolutional layers and a global maximum pooling layer, the inputting the dense vector into the feature processing layer to obtain the second feature vector includes:

[0025] The multiple convolutional layers sequentially perform convolution processing on the dense vector to obtain a fourth eigenvector;

[0026] The fourth feature vector is input into the global maximum pooling layer for pooling processing to obtain the second feature vector.

[0027] Further, if the feature processing layer includes multiple convolutional layers and multiple global maximum pooling layers, any processing unit includes one convolutional layer and one global maximum pooling layer, and the global maximum pooling layer is connected after the convolutional layer, and the multiple processing units are connected in sequence;

[0028] The step of inputting the dense vector into a feature processing layer to obtain a second feature vector comprises:

[0029] The dense vector is input into the first convolution layer, so that the convolution layer and the global maximum pooling layer in the processing unit of the feature processing layer process the received vector, and the vector processed by the last processing unit is used as the second feature vector.

[0030] Furthermore, the training process of the feature extraction model includes:

[0031] Acquire a training set, wherein the training set includes sample data set pairs and labels corresponding to the sample data set pairs, wherein the labels are used to identify whether data distributions between each sample data set in the corresponding sample data set pair are similar;

[0032] For each sample data set in each sample data set pair, counting the second occurrence frequency of each group of data in the sample data set; determining a third data distribution vector of the sample data set according to the second occurrence frequency of each group of data in the sample data set;

[0033] Input each third data distribution vector of any sample data set pair into the original feature extraction model to obtain a prediction feature vector corresponding to each third data distribution vector; determine the prediction result of the sample data set pair according to the similarity between each prediction feature vector in the sample data set pair;

[0034] A loss value is determined according to the prediction result and the label corresponding to the sample data set, and the original feature extraction model is trained based on the loss value to obtain the feature extraction model.

[0035] The present application provides a data distribution analysis device, the device comprising:

[0036] An acquisition module, used for acquiring each data set to be subjected to data distribution analysis, wherein the data of each data set is stored in a different storage environment;

[0037] A determination module, configured to respectively count the first occurrence frequency of each group of data in each data set; for each data set, determine a first data distribution matrix of the data set according to the first occurrence frequency of each group of data in the data set;

[0038] The analysis module is used to extract the target features corresponding to each first data distribution matrix based on the pre-trained feature extraction model; and determine whether the data distribution between each data set is similar based on the similarity between each of the target features.

[0039] Furthermore, the determination module is specifically used to divide each data in each data set into corresponding sub-data groups based on the grouping strategy of the data group for each group of data in each data set; and determine the first occurrence frequency corresponding to each sub-data group of the data group according to the first number of data included in each sub-data group and the second number of data included in the data set.

[0040] Furthermore, the determination module is specifically used to select a preset number of target occurrence frequencies for each data group of the data set according to the first occurrence frequency of each sub-data group of the data group; fill the target occurrence frequencies into the preset matrix template in turn according to the size of each target occurrence frequency to obtain the second data distribution matrix of the data group; and determine the first data distribution matrix of the data set according to the second data distribution matrix of each data group of the data set.

[0041] Furthermore, the analysis module is specifically used to input any first data distribution matrix into the input layer for data format conversion to obtain a first eigenvector; input the first eigenvector into the embedding layer for mapping processing to obtain a dense vector; input the dense vector into the feature processing layer to obtain a second eigenvector, and the feature processing layer includes at least one convolutional layer and at least one global maximum pooling layer; input the second eigenvector into at least one fully connected layer for feature extraction to obtain a third eigenvector; the output layer outputs the third eigenvector in a set format to obtain the target feature corresponding to the first data distribution matrix.

[0042] Furthermore, if the feature processing layer includes multiple convolution layers and a global maximum pooling layer, the analysis module is specifically used to perform convolution processing on the dense vector in sequence by the multiple convolution layers to obtain a fourth feature vector; and input the fourth feature vector into the global maximum pooling layer for pooling processing to obtain the second feature vector.

[0043] Further, if the feature processing layer includes multiple convolutional layers and multiple global maximum pooling layers, any processing unit includes one convolutional layer and one global maximum pooling layer, and the global maximum pooling layer is connected after the convolutional layer, and the multiple processing units are connected in sequence;

[0044] The analysis module is specifically used to input the dense vector into the first convolution layer, so that the convolution layer and the global maximum pooling layer in the processing unit of the feature processing layer process the received vector, and use the vector processed by the last processing unit as the second feature vector.

[0045] Furthermore, the acquisition module is further used to acquire a training set, wherein the training set includes sample data set pairs and labels corresponding to the sample data set pairs, wherein the labels are used to identify whether data distributions between each sample data set in the corresponding sample data set pairs are similar;

[0046] The determination module is further configured to count the second occurrence frequency of each group of data in each sample data set in each sample data set pair; and determine the third data distribution vector of the sample data set according to the second occurrence frequency of each group of data in the sample data set;

[0047] A training module is used to input each third data distribution vector of any sample data set pair into the original feature extraction model respectively to obtain the predicted feature vector corresponding to each third data distribution vector; determine the prediction result of the sample data set pair according to the similarity between each predicted feature vector in the sample data set pair; determine the loss value according to the prediction result and the label corresponding to the sample data set pair, and train the original feature extraction model based on the loss value to obtain the feature extraction model.

[0048] An embodiment of the present application further provides an electronic device, the electronic device comprising a processor, the processor being configured to implement the steps of any of the above-described data distribution analysis methods when executing a computer program stored in a memory.

[0049] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, which implements the steps of any of the above-mentioned data distribution analysis methods when executed by a processor.

[0050] An embodiment of the present application also provides a computer program product, which includes: a computer program code, when the computer program code is executed on a computer, enables the computer to execute the steps of any of the above-mentioned data distribution analysis methods.

[0051] In the embodiment of the present application, each data set corresponding to the data stored in different storage environments is obtained, and the first occurrence frequency of each group of data in each data set is counted respectively, and then for each data set, the first data distribution matrix of the data set is determined according to the first occurrence frequency of each group of data in the data set, and based on the pre-trained feature extraction model, the target features corresponding to each first data distribution matrix are extracted to determine the data distribution features of each data set, and then based on the similarity between each target feature, it is determined whether the data distribution between each data set is similar. According to the frequency of occurrence of each group of data in each data set, the data distribution of each data set is determined, so as to determine whether the data distribution between different data sets is consistent, and realize rapid data distribution analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 A schematic diagram of a data distribution analysis process provided in an embodiment of the present application;

[0054] Figure 2 A schematic diagram of a process of extracting target features by a feature extraction model provided in an embodiment of the present application;

[0055] Figure 3 A schematic diagram of a data distribution analysis process provided in an embodiment of the present application;

[0056] Figure 4 A schematic diagram of the structure of a data distribution analysis device provided in an embodiment of the present application;

[0057] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0059] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0060] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0061] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0062] At present, in order to conduct tests closer to reality, data is generally moved directly from the production environment to the test environment for testing, or the test environment directly collects real data for testing, which is very likely to cause data leakage. Moreover, even if the compliance of data acquisition meets the requirements, the timeliness of data processing and the efficiency of comparison will be greatly reduced. At present, although data can be constructed manually and the data distribution can be analyzed manually, the efficiency is extremely low. Therefore, there is an urgent need for a convenient way to evaluate whether the data distribution of the production environment and the test environment is consistent, so as to ensure that the performance test implemented based on the test environment is real and effective. Considering the above problems, it is found through research that deep learning technology, especially deep neural networks, shows significant advantages in processing and identifying complex patterns in large-scale data. By applying deep learning technology to feature extraction of data, the distribution law and abnormal patterns of data can be captured more accurately, thereby generating more descriptive feature vectors.

[0063] Based on this, the present application provides a data distribution analysis method, device, equipment and medium. In an embodiment of the present application, each data set to be subjected to data distribution analysis is obtained, and the first occurrence frequency of each group of data in each data set is counted respectively; for each data set, the first data distribution matrix of the data set is determined according to the first occurrence frequency of each group of data in the data set; based on the pre-trained feature extraction model, the target features corresponding to each first data distribution matrix are extracted; based on the similarity between each target feature, it is determined whether the data distribution between each data set is similar.

[0064] Embodiment 1:

[0065] Figure 1 A schematic diagram of a data distribution analysis process provided in an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:

[0066] S101: Acquire each data set to be subjected to data distribution analysis, and count the first occurrence frequency of each group of data in each data set, wherein the data of each data set is stored in different storage environments.

[0067] A data distribution analysis method provided in an embodiment of the present application is applied to an electronic device, which may be a server, a PC, a smart terminal, etc.

[0068] When it is necessary to analyze the data distribution, in an embodiment of the present application, each data set to be analyzed for data distribution can be obtained. The data of different data sets can be stored in different storage environments, for example, the two storage environments can be a production environment and a test environment, respectively. In order to ensure data security, real data in the production environment is generally not used for testing during testing. Relevant personnel generally build a test database with the same production database structure as the production environment in the test environment, wherein the number of data tables in the test database and the production database, as well as the data table structure are the same. After building the test database, relevant personnel can analyze the real data recorded in each data table in the production database, and rewrite and expand the test data based on these real data to obtain the test data, and store the test data in the data table corresponding to the test database. In an embodiment of the present application, all data recorded in the production database can be determined as a data set, and all data recorded in the test database can be determined as a data set.

[0069] After obtaining each data set, the first occurrence frequency of each group of data in each data set can be counted respectively. In an embodiment of the present application, the associated data can be determined as a group of data. In other words, each data included in any data group is associated. Among them, data association can be understood as the existence of commonality in the data in the corresponding data group. Exemplarily, all data recorded in any data table in the database can be determined as a group of data, i.e., a data group.

[0070] Assume that data set A includes all data recorded in database A, and data set B includes all data recorded in database B. The number of data tables and data table structures included in database A and database B are the same, but the specific data recorded in each data table is different. When analyzing the data in data set A and data set B, all data recorded in one data table can be determined as a group of data.

[0071] When determining the first occurrence frequency of any data group, the number of data included in the data group may be counted, and the ratio of the number to the total number of data included in the data set corresponding to the data group may be determined as the first occurrence frequency.

[0072] S102: For each data set, determine a first data distribution matrix of the data set according to a first occurrence frequency of each group of data in the data set.

[0073] After determining the first occurrence frequency of each group of data, the first data distribution matrix of each data set can be determined according to the first occurrence frequency of each group of data in the data set. The first data distribution matrix can be an M*N order matrix, where M and N can be any positive integers, for example, M=1, N=3, or M=3, N=1, and can be configured as needed by those skilled in the art.

[0074] In a possible implementation, each group of data may be sorted according to the relevant information corresponding to each group of data. For example, after the data in the data set is grouped according to the data table where the data is located, each group of data may be sorted according to the first letter of the corresponding data table name, or may be sorted according to the number of strokes of the corresponding data table name. After sorting each group of data, the corresponding first occurrence frequency may be sequentially filled into the preset matrix template in this order to obtain the first data distribution matrix of the corresponding data set.

[0075] In a possible implementation, a preset matrix template may be saved, and for each data set, the first occurrence frequency of each group of data in the data set is filled into the position corresponding to the preset matrix template to obtain the first data distribution matrix of the data set. In order to ensure the similarity of data distribution as much as possible, when constructing data, relevant personnel generally ensure that the number of data groups included in each data set to be subjected to data distribution analysis is the same, that is, the number and structure of data tables are the same. In order to facilitate the subsequent determination of the first data distribution matrix, relevant personnel may add an identifier to each data group when constructing data; if each data group is determined by electronic device division, then after the electronic device divides a data group based on a certain division strategy, it may add an identifier to the data group.

[0076] When filling the preset matrix template, the first occurrence frequencies of two corresponding data groups in two data sets can be filled into the same position of the preset matrix template according to the identifier corresponding to each data group. For example, data set A includes data group A and data group B, and data set B includes data group A' and data group B', wherein data group A and data group A' are corresponding data groups, and the data in these two data groups are data recorded in data tables with the same data table structure, and data group B and data group B' are corresponding data groups, and the data in these two data groups are data recorded in data tables with the same data table structure. The first occurrence probability of data group A in data set A is filled into the first position to be filled in the preset matrix template A, and the first occurrence probability corresponding to data group A' in data set B is filled into the first position to be filled in the preset matrix template B; the first occurrence probability corresponding to data group B in data set A is filled into the second position to be filled in the preset matrix template A, and the first occurrence probability corresponding to data group B' in data set B is filled into the second position to be filled in the preset matrix template B, and so on.

[0077] S103: Extracting target features corresponding to each first data distribution matrix based on a pre-trained feature extraction model.

[0078] In order to capture the features in the first data distribution matrix, in an embodiment of the present application, after obtaining the first data distribution matrix, a deep neural network model can be used to process and analyze the first feature data distribution matrix. In an embodiment of the present application, the target features corresponding to each first data distribution matrix can be extracted based on a pre-trained feature extraction model. The target feature represents the data distribution feature of the corresponding data set and can reflect the data distribution characteristics of the corresponding data set. The target features extracted by the feature extraction model can be represented in the form of a vector or in the form of a matrix, or in other forms. The target features between data sets with similar data distributions are closer, and whether the data distributions between the corresponding data sets are similar can be indirectly determined based on whether the determined target features are similar. Among them, the feature extraction model can be any model that has feature extraction capabilities for the first data distribution matrix. The feature extraction model can include a multilayer perceptron (Multilayer Perceptron, MLP), a convolutional neural network (Convolutional Neural Networks, CNN) or a recursive neural network (recursive neuralnetwork, RNN). The specific model internal structure selection depends on the characteristics of the data and the required analysis depth, and technicians in this field can configure it as needed.

[0079] S104: Based on the similarity between each of the target features, determine whether the data distribution between each of the data sets is similar.

[0080] After obtaining the target feature corresponding to each data set, the similarity between each target feature can be determined, and based on the determined similarity, it can be determined whether the data distribution between each data set is similar.

[0081] Specifically, if the determined target feature is in the form of a matrix or a vector, the similarity between the matrices or vectors of any two data sets can be determined. After the similarity between the target features of any two data sets is determined, it can be determined whether the similarity is greater than a preset threshold. If so, it can be determined that the data distributions between the two data sets are similar. Otherwise, it is determined that the data distributions between the two data sets are not similar.

[0082] If the determined target feature is in other representation forms, such as using text to describe the data distribution characteristics of the corresponding data set, then after obtaining the target feature of each text description method, the semantic similarity of the target features of any two data sets can be determined, thereby determining whether the data distribution between the two data sets is similar based on the semantic similarity and a preset threshold.

[0083] In the embodiment of the present application, each data set corresponding to the data stored in different storage environments is obtained, and the first occurrence frequency of each group of data in each data set is counted respectively, and then for each data set, the first data distribution matrix of the data set is determined according to the first occurrence frequency of each group of data in the data set, and based on the pre-trained feature extraction model, the target features corresponding to each first data distribution matrix are extracted to determine the data distribution features of each data set, and then based on the similarity between each target feature, it is determined whether the data distribution between each data set is similar. According to the frequency of occurrence of each group of data in each data set, the data distribution of each data set is determined, so as to determine whether the data distribution between different data sets is consistent, and realize rapid data distribution analysis.

[0084] Embodiment 2:

[0085] In order to improve the accuracy of data distribution analysis, based on the above embodiment, in the embodiment of the present application, the first occurrence frequency of each group of data in each data set is counted separately, including:

[0086] For each group of data in each data set, each data in the data group is divided into corresponding sub-data groups based on the grouping strategy of the data group;

[0087] According to a first quantity of data included in each sub-data group and a second quantity of data included in the data set, a first occurrence frequency corresponding to each sub-data group of the data set is determined.

[0088] Since the same data group may record multiple data of the same person or the same enterprise, in order to accurately analyze the data distribution, in the embodiment of the present application, when counting the first occurrence frequency of each group of data in each data set, the data in each data group can be subdivided. Since the data structure of the data table corresponding to each data group is known, then, when constructing the data, the grouping strategy of the corresponding data group can be configured according to the field name. For example, the transfer records of each company are recorded in data group A, and any tool can have multiple transfer records. In the embodiment of the present application, the grouping strategy of "grouping by company name" can be saved for the data group A. For another example, the transaction records generated by commodity A in 2023 are recorded in data group B. In the embodiment of the present application, the grouping strategy of "grouping by month" can be saved for the data group B. For another example, the personal information of all students in the school is recorded in data group C. In the embodiment of the present application, the grouping strategy of "grouping by class" can be saved for the data group C. Those skilled in the art can configure different grouping strategies as needed, and the embodiment of the present application does not limit this.

[0089] In an embodiment of the present application, when determining the first occurrence frequency of each group of data in each data set, a grouping strategy saved for each group of data in each data set can be obtained, and each data in the data group can be grouped according to the grouping strategy to obtain multiple sub-data groups.

[0090] Specifically, if the data group records the transfer records of each company, multiple sub-data groups can be obtained based on the grouping strategy of "grouping by company name", and the data included in any data group are the transfer records of the same company. For example, the data in sub-data group 1 are all transfer records of company A, and the data in sub-data group 2 are all transfer records of company B.

[0091] If the data group records the transaction records of commodity A in 2023, multiple sub-data groups can be obtained based on the grouping strategy of "grouping by month". The data included in any data group are the transaction records of commodity A in the same month. For example, the data in sub-data group 1 are the transaction records of commodity A in January, and the data in sub-data group 2 are the transaction records of commodity A in February.

[0092] If the data group records the personal information of all students in the school, multiple sub-data groups can be obtained based on the grouping strategy of "grouping by class". The data included in any data group is the personal information of students in the same class. For example, the data in sub-data group 1 is the personal information of students in Class 2, Grade 3, and the data in sub-data group 2 is the personal information of students in Class 5, Grade 3.

[0093] After obtaining each sub-data group of the data group, the first occurrence frequency corresponding to each sub-data group of the data group may be determined according to the first quantity of data included in each sub-data group and the second quantity of data included in the data set.

[0094] Assume that the data set C includes 1000 pieces of data, that is, the second quantity is 1000, and a data group in the data set C includes 100 pieces of data, and the 100 pieces of data record 28 pieces of deposit data of company A. Then, the 28 pieces of deposit data about company A can be determined as a sub-data group of the data group. It can be determined that the first occurrence frequency of this sub-data group corresponding to company A is 28 / 1000=0.028.

[0095] In order to further improve the accuracy of data distribution analysis, based on the above embodiments, in the embodiment of the present application, the first data distribution matrix of the data set is determined according to the first occurrence frequency of each group of data in the data set, including:

[0096] For each data group of the data set, selecting a preset number of target occurrence frequencies according to the first occurrence frequency of each sub-data group of the data group;

[0097] According to the magnitude of each target occurrence frequency, the target occurrence frequencies are sequentially filled into the preset matrix template to obtain a second data distribution matrix of the data group;

[0098] A first data distribution matrix of the data set is determined according to the second data distribution matrix of each data group of the data set.

[0099] Through research and analysis, it is found that in a large amount of data, if the data distribution of most of the data is similar, and only the data distribution of a small part of the data is not similar, then the impact of these small parts of the data on the global data is not large. Based on this, in an embodiment of the present application, in order to further improve the accuracy of data distribution analysis and improve the efficiency of data distribution analysis, after determining the first frequency of occurrence of each sub-data group of each group of data, for each data set, according to the first frequency of occurrence of each group of data of the data set, when determining the first data distribution matrix of the data set, for each data set of the data set, in the data set, according to the first frequency of occurrence of each sub-data group, a preset number of target frequencies of occurrence are selected, and subsequently based on the preset number of target frequencies of occurrence, the second data distribution matrix of the data set is determined. Wherein, the preset number can be any integer, such as 10, 50, 99, 122, etc. The preset numbers corresponding to different data groups can be the same or different. In other words, for different data groups, the number of selected target frequencies of occurrence can be the same or different. The dimensions of the second data distribution matrix determined based on different numbers of target occurrence frequencies may be different. In order to facilitate the subsequent determination of the second data distribution matrix, the second data distribution matrix can be filled with preset values ​​so that the dimensions of each second data distribution matrix finally determined are the same.

[0100] After selecting a preset number of target occurrence frequencies, the target occurrence frequencies can be filled into the preset matrix template in turn according to the size of each target occurrence frequency to obtain a second data distribution matrix of the data group. In an embodiment of the present application, the second data distribution matrix can be a row of N columns, and the number of columns of the second data distribution matrix depends on the preset number. When filling the preset matrix template, if the target occurrence frequencies are all filled, and there are still unfilled items in the preset matrix template, the preset values ​​can be used to complete them, for example, "0" is used to complete the remaining items.

[0101] For example, the target occurrence frequencies corresponding to each sub-data group included in data group A are 0.3, 0.22, 0.35, and 0.13, respectively, and the target occurrence frequencies corresponding to each sub-data group included in data group B are 0.2, 0.61, and 0.19, respectively. The preset matrix template includes 4 items to be filled. Then, the second data distribution matrix corresponding to data group A can be [0.35, 0.3, 0.22, 0.13], and the second data distribution matrix corresponding to data group B can be [0.61, 0.2, 0.19].

[0102] After obtaining the second data distribution matrix of each data group of the data set, the first data distribution matrix of the data set can be determined according to the second data distribution matrix of each data group of the data set. For example, the second data distribution matrix of each data group is sequentially spliced ​​to obtain the first data distribution matrix of the data set. Continuing with the above example, if data set A includes data group A and data group B, wherein the second data distribution matrix of data group A is [0.35, 0.3, 0.22, 0.13], and the second data distribution matrix of data group B is [0.61, 0.2, 0.19], then the first data distribution matrix of the determined data set can be [0.35, 0.3, 0.22, 0.13, 0.61, 0.2, 0.19].

[0103] In a possible implementation, if data set A includes data group A and data group B, wherein the second data distribution matrix of data group A is [0.35, 0.3, 0.22, 0.13], and the second data distribution matrix of data group B is [0.61, 0.2, 0.19], then the first data distribution matrix of the determined data set can be That is to say, the generated first data distribution matrix can be an M*N order matrix, where M represents the number of data groups included in the corresponding data set, and N represents the number of corresponding target occurrence frequencies in each data group. When determining the data distribution matrix, preset values ​​such as "0" can be used for filling.

[0104] In the embodiment of the present application, the second data distribution matrix of the data group is determined according to the size of the target occurrence frequency, aiming to capture the central tendency and possible abnormal distribution of the data, and provide an empirical basis for feature extraction.

[0105] Embodiment 3:

[0106] In order to further improve the accuracy of data distribution analysis, on the basis of the above embodiments, in the embodiment of the present application, the feature extraction model based on the pre-trained process extracts the target features corresponding to each first data distribution matrix, including:

[0107] Input any first data distribution matrix into the input layer for data format conversion to obtain a first eigenvector;

[0108] Inputting the first feature vector into the embedding layer for mapping processing to obtain a dense vector;

[0109] Inputting the dense vector into a feature processing layer to obtain a second feature vector, wherein the feature processing layer includes at least one convolutional layer and at least one global maximum pooling layer;

[0110] Inputting the second feature vector into at least one fully connected layer for feature extraction to obtain a third feature vector;

[0111] The output layer outputs the third eigenvector in a set format to obtain the target feature corresponding to the first data distribution matrix.

[0112] When using a pre-trained feature extraction model to extract the target features corresponding to each first data distribution matrix, each first data distribution matrix can be input into the feature extraction model, and the feature extraction model extracts the features of each first data distribution matrix. In an embodiment of the present application, the feature extraction model may include an input layer, an embedding layer, a feature processing layer, a fully connected layer, and an output layer.

[0113] When using the feature extraction model to extract target features corresponding to any first data distribution matrix, the first data distribution matrix can be input into the input layer for data format conversion, and the first data distribution matrix can be converted into a first feature vector in a format that can be processed by the neural network.

[0114] After obtaining the first eigenvector, the first eigenvector can be input into the embedding layer for mapping processing to obtain a dense vector. In other words, the data distribution information of each data group is converted into a dense vector representation, which helps the neural network better understand the relationship between different data groups.

[0115] After obtaining the dense vector, the dense vector can be input into the feature processing layer for feature extraction to obtain a second feature vector. The feature processing layer may include at least one convolutional layer and at least one global maximum pooling layer (Global Max Pooling). That is, the dense vector can be feature extracted using a convolutional layer and a global maximum pooling layer, or multiple convolutional layers and multiple global maximum pooling layers can be stacked to extract features from the dense vector. In an embodiment of the present application, a convolutional layer is used to capture local features of data distribution. Optionally, the convolutional layer can be a one-dimensional convolutional layer (1DConvolutional Layer). The global maximum pooling layer is generally located after the convolutional layer, and the global maximum pooling layer can extract global features from the output of the convolutional layer, which helps the network focus on the most important data distribution features while reducing the number of parameters.

[0116] After obtaining the second feature vector, the second feature vector can be input into at least one fully connected layer for feature extraction to obtain a third feature vector. The fully connected layer is also called a dense layer. In an embodiment of the present application, feature extraction can be performed using one fully connected layer, or multiple fully connected layers can be used for feature extraction. For example, two fully connected layers are used for feature extraction, so that the model learns more abstract and comprehensive feature representations.

[0117] After the third eigenvector is obtained, the third eigenvector may be input to the output layer, so that the output layer outputs the third eigenvector in a set format to obtain the target feature corresponding to the first data distribution matrix.

[0118] In order to further improve the accuracy of data distribution analysis, based on the above embodiments, in the embodiment of the present application, if the feature processing layer includes multiple convolutional layers and a global maximum pooling layer, the dense vector is input into the feature processing layer to obtain a second feature vector, including:

[0119] The multiple convolutional layers sequentially perform convolution processing on the dense vector to obtain a fourth eigenvector;

[0120] The fourth feature vector is input into the global maximum pooling layer for pooling processing to obtain the second feature vector.

[0121] In an embodiment of the present application, if the feature extraction layer for feature extraction includes multiple convolutional layers and a global maximum pooling layer, then when the feature extraction layer is used to process the dense vector to obtain the second feature vector, the multiple convolutional layers can be used to convolve the dense vector in sequence to obtain a fourth feature vector. For example, the feature extraction layer includes three convolutional layers, then the first convolutional layer can convolve the dense vector output by the embedding layer to obtain a first intermediate vector, and then input the first intermediate vector into the second convolutional layer, and the second convolutional layer convolve the first intermediate vector to obtain a second intermediate vector, and then input the second intermediate vector into the third convolutional layer, and the third convolutional layer convolve the second intermediate vector to obtain a fourth feature vector.

[0122] After obtaining the fourth eigenvector output by the last convolutional layer, the fourth eigenvector may be input into a global maximum pooling layer for pooling processing to obtain a second eigenvector.

[0123] In order to further improve the accuracy of data distribution analysis, on the basis of the above embodiments, in the embodiment of the present application, if the feature processing layer includes multiple convolutional layers and multiple global maximum pooling layers, any processing unit includes a convolutional layer and a global maximum pooling layer, and the global maximum pooling layer is connected after the convolutional layer, and multiple processing units are connected in sequence;

[0124] The step of inputting the dense vector into a feature processing layer to obtain a second feature vector comprises:

[0125] The dense vector is input into the first convolution layer, so that the convolution layer and the global maximum pooling layer in the processing unit of the feature processing layer process the received vector, and the vector processed by the last processing unit is used as the second feature vector.

[0126] In an embodiment of the present application, if the feature extraction layer for feature extraction includes multiple convolutional layers and multiple global maximum pooling layers, then the feature extraction layer can be divided into multiple processing units. Among them, any processing unit includes a convolutional layer and a global maximum pooling layer, and the global maximum pooling layer is connected after the convolutional layer, and multiple processing units are connected in sequence. Exemplarily, the indirect method of each structural layer of the feature extraction layer can be convolutional layer 1-global maximum pooling layer 1-convolutional layer 2-global maximum pooling layer 2-convolutional layer 3-global maximum pooling layer 3. Among them, convolutional layer 1 and global maximum pooling layer 1 constitute a processing unit, convolutional layer 2 and global maximum pooling layer 2 constitute a processing unit, and convolutional layer 3 and global maximum pooling layer 3 constitute a processing unit.

[0127] When the dense vector is processed by the feature processing layer, the dense vector can be input into the first convolution layer, so that the convolution layer and the global maximum pooling layer in the first processing unit of the feature processing layer process the received vector, and the vector processed by the last processing unit is used as the second feature vector.

[0128] The following describes a process of determining the target feature corresponding to the first data distribution matrix in conjunction with a specific embodiment. Figure 2 A schematic diagram of a process of extracting target features by a feature extraction model provided in an embodiment of the present application, such as Figure 2 As shown, first, for any data set, for each data group of the data set, the top N target occurrence frequencies are selected according to the first occurrence frequency of each sub-data group of the data group. According to the top N target occurrence frequencies, the second data distribution matrix of the data group is determined.

[0129] After the second data distribution matrix of each data group in the data set is obtained, the first data distribution matrix of the data set is determined according to each second data distribution matrix.

[0130] After determining the first data distribution matrix, the first data distribution matrix is ​​input into the input layer to obtain the first eigenvector. The first eigenvector is input into the embedding layer to obtain a dense vector. The dense vector is input into the one-dimensional convolution layer to obtain the fourth eigenvector. The fourth eigenvector is input into the global maximum pooling layer to obtain the second eigenvector. The second eigenvector is input into the fully connected layer 1 to obtain the fifth eigenvector, and the fifth eigenvector is input into the fully connected layer 2 to obtain the third eigenvector. Finally, the output layer outputs the third eigenvector in a set format to obtain the target feature corresponding to the first data distribution matrix.

[0131] Embodiment 4:

[0132] In order to obtain a highly accurate feature extraction model, based on the above embodiments, in the embodiment of the present application, the training process of the feature extraction model includes:

[0133] Acquire a training set, wherein the training set includes sample data set pairs and labels corresponding to the sample data set pairs, wherein the labels are used to identify whether data distributions between each sample data set in the corresponding sample data set pair are similar;

[0134] For each sample data set in each sample data set pair, counting the second occurrence frequency of each group of data in the sample data set; determining a third data distribution vector of the sample data set according to the second occurrence frequency of each group of data in the sample data set;

[0135] Input each third data distribution vector of any sample data set pair into the original feature extraction model to obtain a prediction feature vector corresponding to each third data distribution vector; determine the prediction result of the sample data set pair according to the similarity between each prediction feature vector in the sample data set pair;

[0136] A loss value is determined according to the prediction result and the label corresponding to the sample data set, and the original feature extraction model is trained based on the loss value to obtain the feature extraction model.

[0137] In order to obtain a highly accurate feature extraction model, in an embodiment of the present application, a training set is pre-configured, and the original feature extraction model can be trained based on the sample data set pairs included in the training set. In order to facilitate the subsequent training of the original feature extraction model, the training set includes not only sample data set pairs, but also labels corresponding to each sample data set pair. The label is used to identify whether the data distribution between each sample data set in the corresponding sample data set pair is similar.

[0138] In a possible implementation, the sample data set pairs included in the training set can be historical data processed by the feature extraction model, and the corresponding labels can be historical results output by the feature extraction model, or historical output results after manual verification. In other words, the data in the data distribution analysis process can be used for the training and upgrading of the feature extraction model in the later stage.

[0139] In order to train the original feature extraction model, in an embodiment of the present application, after obtaining the training set, for each sample data set in each sample data set pair in the training set, the second occurrence frequency of each group of data in the sample data set can be counted, and the third data distribution vector of the sample data set can be determined according to the second occurrence frequency of each group of data in the sample data set. The process of determining the third data distribution vector is similar to the process of determining the first data distribution matrix. When determining the third data distribution vector, the above embodiment can be referred to, and the embodiment of the present application will not be repeated here.

[0140] After obtaining the third data distribution vector corresponding to each sample data set, each third data distribution vector of any sample data set pair can be input into the original feature extraction model to obtain the predicted feature vector corresponding to each third data distribution vector output by the original feature extraction model. Thus, the prediction result of the sample data set pair is determined according to the similarity between each predicted feature vector in the sample data set pair.

[0141] After obtaining the prediction result of the sample data set pair, the label of the sample data set pair can be obtained. Therefore, the loss value is determined based on the deviation between the prediction result and the label. And the original feature extraction model is trained based on the determined loss value. In an embodiment of the present application, the convergence condition can be pre-configured, and the feature extraction model obtained when the convergence condition is met can be used as the feature extraction model for data distribution analysis. By training the original feature extraction model, its efficiency in identifying and extracting key data features can be maximized. The deep neural network outputs target features representing the data distribution of the data by learning the intrinsic patterns and associations of the data. These target features comprehensively consider the multi-dimensional information of the data volume and data distribution.

[0142] Embodiment 5:

[0143] The process of data distribution analysis is described below in conjunction with a specific embodiment. In the embodiment of the present application, the main focus is on analyzing whether the data distributions in two databases are consistent. Figure 3 A schematic diagram of a data distribution analysis process provided in an embodiment of the present application, such as Figure 3 As shown, it mainly includes the following steps:

[0144] Statistical data: Statistical data information of each data table in the database, including the total number of data recorded in the database and the number of data recorded in each data table. All data recorded in the database can be regarded as a data set, and the data recorded in each data table can be regarded as a data group.

[0145] Parse index information: Get the index structure of each data table in the database, which may include primary key, foreign key and other index information. In the database design process, for some index columns used for query optimization, their types and other information are generally recorded in detail. Therefore, when getting each index structure, you can directly get it from the system table.

[0146] Statistical index data distribution: For each data group, that is, each data table, a group by operation is performed according to the index structure corresponding to the data table, and the data recorded in the data table is divided into multiple sub-data groups, so as to facilitate the statistical distribution of data in each data table. Then, according to the first occurrence frequency of each sub-data group, the top N target occurrence frequencies are selected, and then based on these N target occurrence frequencies, the first data distribution matrix of the corresponding database is determined.

[0147] Feature extraction: Use the pre-trained feature extraction model to extract the target features corresponding to the first data distribution matrix.

[0148] Similarity analysis: After obtaining the target features corresponding to each database to be analyzed, we can determine whether the data distribution between databases is similar based on the similarity between each target feature. We use the extracted target features to evaluate the similarity of data distribution between different databases. This analysis can help administrators identify tables with similar data distribution features, so as to conduct reasonable non-functional testing and optimization strategy deployment.

[0149] By automatically counting the amount of data and index information in the database table, and using deep neural networks to automatically analyze and generate feature vectors of the data, this technical solution greatly reduces the need for manual intervention. This automated processing not only improves the efficiency of data management in the test environment, but also makes the performance of non-functional tests closer to the actual production situation.

[0150] Based on the data distribution analysis method provided in this application, data optimization can be performed more intelligently and automatically, data processing efficiency can be improved, management complexity can be reduced, and dynamically changing data environments can be effectively responded to. The data distribution analysis method provided in this application will help optimize the overall structure and performance of the database, especially in the case of large amounts of data and high concurrency.

[0151] Embodiment 6:

[0152] Based on the same inventive concept, the embodiment of the present application provides a data distribution analysis device, Figure 4 For a schematic diagram of a data distribution analysis device provided in an embodiment of the present application, please refer to Figure 4 , the device comprises:

[0153] An acquisition module 401 is used to acquire each data set to be subjected to data distribution analysis, wherein the data of each data set is stored in a different storage environment;

[0154] A determination module 402 is used to count the first occurrence frequency of each group of data in each data set respectively; for each data set, determine a first data distribution matrix of the data set according to the first occurrence frequency of each group of data in the data set;

[0155] The analysis module 403 is used to extract the target features corresponding to each first data distribution matrix based on the pre-trained feature extraction model; and determine whether the data distribution between each data set is similar based on the similarity between each of the target features.

[0156] In a possible implementation, the determination module 402 is specifically used to divide each data group in each data set into corresponding sub-data groups based on the grouping strategy of the data group; and determine the first occurrence frequency corresponding to each sub-data group of the data group according to the first number of data included in each sub-data group and the second number of data included in the data set.

[0157] In a possible implementation, the determination module 402 is specifically used to select a preset number of target occurrence frequencies for each data group of the data set according to the first occurrence frequency of each sub-data group of the data group; fill the target occurrence frequencies into a preset matrix template in turn according to the size of each target occurrence frequency to obtain a second data distribution matrix of the data group; and determine the first data distribution matrix of the data set according to the second data distribution matrix of each data group of the data set.

[0158] In one possible implementation, the analysis module 403 is specifically used to input any first data distribution matrix into the input layer for data format conversion to obtain a first eigenvector; input the first eigenvector into the embedding layer for mapping processing to obtain a dense vector; input the dense vector into the feature processing layer to obtain a second eigenvector, the feature processing layer includes at least one convolutional layer and at least one global maximum pooling layer; input the second eigenvector into at least one fully connected layer for feature extraction to obtain a third eigenvector; the output layer outputs the third eigenvector in a set format to obtain the target feature corresponding to the first data distribution matrix.

[0159] In a possible implementation, if the feature processing layer includes multiple convolutional layers and a global maximum pooling layer, the analysis module 403 is specifically used to perform convolution processing on the dense vector in sequence by the multiple convolutional layers to obtain a fourth feature vector; and input the fourth feature vector into the global maximum pooling layer for pooling processing to obtain the second feature vector.

[0160] In a possible implementation, if the feature processing layer includes multiple convolutional layers and multiple global maximum pooling layers, any processing unit includes one convolutional layer and one global maximum pooling layer, and the global maximum pooling layer is connected after the convolutional layer, and the multiple processing units are connected in sequence;

[0161] The analysis module 403 is specifically used to input the dense vector into the first convolution layer, so that the convolution layer and the global maximum pooling layer in the processing unit of the feature processing layer process the received vector, and use the vector processed by the last processing unit as the second feature vector.

[0162] In a possible implementation, the acquisition module 401 is further used to acquire a training set, wherein the training set includes sample data set pairs and labels corresponding to the sample data set pairs, wherein the labels are used to identify whether data distributions between each sample data set in the corresponding sample data set pairs are similar;

[0163] The determination module 402 is further configured to, for each sample data set in each sample data set pair, count the second occurrence frequency of each group of data in the sample data set; and determine a third data distribution vector of the sample data set according to the second occurrence frequency of each group of data in the sample data set;

[0164] The training module 404 is used to input each third data distribution vector of any sample data set pair into the original feature extraction model respectively to obtain the predicted feature vector corresponding to each third data distribution vector; determine the prediction result of the sample data set pair according to the similarity between each predicted feature vector in the sample data set pair; determine the loss value according to the prediction result and the label corresponding to the sample data set pair, and train the original feature extraction model based on the loss value to obtain the feature extraction model.

[0165] Embodiment 7:

[0166] Based on the same inventive concept, an embodiment of the present application provides an electronic device, which can implement the steps of the data distribution analysis method discussed above. Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 5As shown, it includes: a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504;

[0167] The memory 503 stores a computer program. When the program is executed by the processor 501, the processor 501 performs the following steps:

[0168] Obtaining each data set to be subjected to data distribution analysis, and respectively counting the first occurrence frequency of each group of data in each data set, wherein the data of each data set is stored in different storage environments;

[0169] For each data set, determining a first data distribution matrix of the data set according to a first occurrence frequency of each group of data in the data set;

[0170] Extracting target features corresponding to each first data distribution matrix based on a pre-trained feature extraction model;

[0171] Based on the similarity between each of the target features, it is determined whether the data distribution between each of the data sets is similar.

[0172] In a possible implementation, respectively counting the first occurrence frequency of each group of data in each data set includes:

[0173] For each group of data in each data set, each data in the data group is divided into corresponding sub-data groups based on the grouping strategy of the data group;

[0174] According to a first quantity of data included in each sub-data group and a second quantity of data included in the data set, a first occurrence frequency corresponding to each sub-data group of the data set is determined.

[0175] In a possible implementation, determining a first data distribution matrix of the data set according to a first occurrence frequency of each group of data in the data set includes:

[0176] For each data group of the data set, according to the first occurrence frequency of each sub-data group of the data group, a preset number of target occurrence frequencies are selected;

[0177] According to the magnitude of each target occurrence frequency, the target occurrence frequencies are sequentially filled into the preset matrix template to obtain a second data distribution matrix of the data group;

[0178] A first data distribution matrix of the data set is determined according to the second data distribution matrix of each data group of the data set.

[0179] In a possible implementation, extracting target features corresponding to each first data distribution matrix based on a pre-trained feature extraction model includes:

[0180] Input any first data distribution matrix into the input layer for data format conversion to obtain a first eigenvector;

[0181] Inputting the first feature vector into the embedding layer for mapping processing to obtain a dense vector;

[0182] Inputting the dense vector into a feature processing layer to obtain a second feature vector, wherein the feature processing layer includes at least one convolutional layer and at least one global maximum pooling layer;

[0183] Inputting the second feature vector into at least one fully connected layer for feature extraction to obtain a third feature vector;

[0184] The output layer outputs the third eigenvector in a set format to obtain the target feature corresponding to the first data distribution matrix.

[0185] In a possible implementation, if the feature processing layer includes multiple convolutional layers and a global maximum pooling layer, inputting the dense vector into the feature processing layer to obtain the second feature vector includes:

[0186] The multiple convolutional layers sequentially perform convolution processing on the dense vector to obtain a fourth eigenvector;

[0187] The fourth feature vector is input into the global maximum pooling layer for pooling processing to obtain the second feature vector.

[0188] In a possible implementation, if the feature processing layer includes multiple convolutional layers and multiple global maximum pooling layers, any processing unit includes one convolutional layer and one global maximum pooling layer, and the global maximum pooling layer is connected after the convolutional layer, and the multiple processing units are connected in sequence;

[0189] The step of inputting the dense vector into a feature processing layer to obtain a second feature vector comprises:

[0190] The dense vector is input into the first convolution layer, so that the convolution layer and the global maximum pooling layer in the processing unit of the feature processing layer process the received vector, and the vector processed by the last processing unit is used as the second feature vector.

[0191] In a possible implementation, the training process of the feature extraction model includes:

[0192] Acquire a training set, wherein the training set includes sample data set pairs and labels corresponding to the sample data set pairs, wherein the labels are used to identify whether data distributions between each sample data set in the corresponding sample data set pair are similar;

[0193] For each sample data set in each sample data set pair, counting the second occurrence frequency of each group of data in the sample data set; determining a third data distribution vector of the sample data set according to the second occurrence frequency of each group of data in the sample data set;

[0194] Input each third data distribution vector of any sample data set pair into the original feature extraction model to obtain a prediction feature vector corresponding to each third data distribution vector; determine the prediction result of the sample data set pair according to the similarity between each prediction feature vector in the sample data set pair;

[0195] A loss value is determined according to the prediction result and the label corresponding to the sample data set, and the original feature extraction model is trained based on the loss value to obtain the feature extraction model.

[0196] Since the principle of solving the problem by the above electronic device is similar to the data distribution analysis method, the implementation of the above electronic device can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0197] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 502 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0198] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0199] Embodiment 8:

[0200] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, in which a computer program executable by a processor is stored. When the program runs on the processor, the processor executes any of the data distribution analysis methods discussed above. Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to that of the data distribution analysis method, the implementation of the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0201] Embodiment 9:

[0202] Based on the same inventive concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the data distribution analysis methods discussed above. Since the principle of solving the problem by the above computer program product is similar to that of the data distribution analysis method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0203] In an embodiment of the present application, each data set corresponding to the data stored in different storage environments is obtained, and the first occurrence frequency of each group of data in each data set is counted respectively, and then for each data set, the first data distribution matrix of the data set is determined according to the first occurrence frequency of each group of data in the data set, and based on the pre-trained feature extraction model, the target features corresponding to each first data distribution matrix are extracted to determine the data distribution features of each data set, and then based on the similarity between each target feature, it is determined whether the data distribution between each data set is similar. According to the frequency of occurrence of each group of data in each data set, the data distribution of each data set is determined, so as to determine whether the data distribution between different data sets is consistent, thereby realizing rapid data distribution analysis.

[0204] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0205] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0206] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0207] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0208] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data distribution analysis method, characterized in that: The method comprises: Obtaining each data set to be subjected to data distribution analysis, and respectively counting the first occurrence frequency of each group of data in each data set, wherein the data of each data set is stored in different storage environments; For each data set, determining a first data distribution matrix of the data set according to a first occurrence frequency of each group of data in the data set; Extracting target features corresponding to each first data distribution matrix based on a pre-trained feature extraction model; Based on the similarity between each of the target features, it is determined whether the data distribution between each of the data sets is similar.

2. The method according to claim 1, characterized in that The respectively counting the first occurrence frequency of each group of data in each data set comprises: For each group of data in each data set, each data in the data group is divided into corresponding sub-data groups based on the grouping strategy of the data group; According to a first quantity of data included in each sub-data group and a second quantity of data included in the data set, a first occurrence frequency corresponding to each sub-data group of the data set is determined.

3. The method according to claim 2, characterized in that The step of determining a first data distribution matrix of the data set according to a first occurrence frequency of each group of data in the data set comprises: For each data group of the data set, according to the first occurrence frequency of each sub-data group of the data group, a preset number of target occurrence frequencies are selected; According to the magnitude of each target occurrence frequency, the target occurrence frequencies are sequentially filled into the preset matrix template to obtain a second data distribution matrix of the data group; A first data distribution matrix of the data set is determined according to the second data distribution matrix of each data group of the data set.

4. The method according to claim 1, characterized in that: The extracting target features corresponding to each first data distribution matrix based on the pre-trained feature extraction model includes: Input any first data distribution matrix into the input layer for data format conversion to obtain a first eigenvector; Inputting the first feature vector into the embedding layer for mapping processing to obtain a dense vector; Inputting the dense vector into a feature processing layer to obtain a second feature vector, wherein the feature processing layer includes at least one convolutional layer and at least one global maximum pooling layer; Inputting the second feature vector into at least one fully connected layer for feature extraction to obtain a third feature vector; The output layer outputs the third eigenvector in a set format to obtain the target feature corresponding to the first data distribution matrix.

5. The method according to claim 4, characterized in that If the feature processing layer includes a plurality of convolutional layers and a global maximum pooling layer, the inputting the dense vector into the feature processing layer to obtain a second feature vector includes: The multiple convolutional layers sequentially perform convolution processing on the dense vector to obtain a fourth eigenvector; The fourth feature vector is input into the global maximum pooling layer for pooling processing to obtain the second feature vector.

6. The method according to claim 4, characterized in that If the feature processing layer includes multiple convolutional layers and multiple global maximum pooling layers, any processing unit includes one convolutional layer and one global maximum pooling layer, and the global maximum pooling layer is connected after the convolutional layer, and the multiple processing units are connected in sequence; The step of inputting the dense vector into a feature processing layer to obtain a second feature vector comprises: The dense vector is input into the first convolution layer, so that the convolution layer and the global maximum pooling layer in the processing unit of the feature processing layer process the received vector, and the vector processed by the last processing unit is used as the second feature vector.

7. The method according to claim 1, characterized in that The training process of the feature extraction model includes: Acquire a training set, wherein the training set includes sample data set pairs and labels corresponding to the sample data set pairs, wherein the labels are used to identify whether data distributions between each sample data set in the corresponding sample data set pair are similar; For each sample data set in each sample data set pair, counting the second occurrence frequency of each group of data in the sample data set; determining a third data distribution vector of the sample data set according to the second occurrence frequency of each group of data in the sample data set; Input each third data distribution vector of any sample data set pair into the original feature extraction model to obtain a prediction feature vector corresponding to each third data distribution vector; determine the prediction result of the sample data set pair according to the similarity between each prediction feature vector in the sample data set pair; A loss value is determined according to the prediction result and the label corresponding to the sample data set, and the original feature extraction model is trained based on the loss value to obtain the feature extraction model.

8. A data distribution analysis device, characterized in that: The device comprises: An acquisition module, used for acquiring each data set to be subjected to data distribution analysis, wherein the data of each data set is stored in a different storage environment; A determination module, configured to respectively count the first occurrence frequency of each group of data in each data set; for each data set, determine a first data distribution matrix of the data set according to the first occurrence frequency of each group of data in the data set; The analysis module is used to extract the target features corresponding to each first data distribution matrix based on the pre-trained feature extraction model; and determine whether the data distribution between each data set is similar based on the similarity between each of the target features.

9. The device according to claim 8, characterized in that The determination module is specifically used to divide each group of data in each data set into corresponding sub-data groups based on the grouping strategy of the data group; and determine the first occurrence frequency corresponding to each sub-data group of the data group according to the first number of data included in each sub-data group and the second number of data included in the data set.

10. The device according to claim 9, characterized in that The determination module is specifically used to select a preset number of target occurrence frequencies for each data group of the data set according to the first occurrence frequency of each sub-data group of the data group; fill the target occurrence frequencies into a preset matrix template in turn according to the size of each target occurrence frequency to obtain the second data distribution matrix of the data group; and determine the first data distribution matrix of the data set according to the second data distribution matrix of each data group of the data set.

11. The device according to claim 8, characterized in that The analysis module is specifically used to input any first data distribution matrix into the input layer for data format conversion to obtain a first eigenvector; input the first eigenvector into the embedding layer for mapping processing to obtain a dense vector; input the dense vector into the feature processing layer to obtain a second eigenvector, and the feature processing layer includes at least one convolutional layer and at least one global maximum pooling layer; input the second eigenvector into at least one fully connected layer for feature extraction to obtain a third eigenvector; the output layer outputs the third eigenvector in a set format to obtain the target feature corresponding to the first data distribution matrix.

12. The device according to claim 11, characterized in that If the feature processing layer includes multiple convolution layers and a global maximum pooling layer, the analysis module is specifically used to perform convolution processing on the dense vector in sequence by the multiple convolution layers to obtain a fourth feature vector; and input the fourth feature vector into the global maximum pooling layer for pooling processing to obtain the second feature vector.

13. The device according to claim 11, characterized in that If the feature processing layer includes multiple convolutional layers and multiple global maximum pooling layers, any processing unit includes one convolutional layer and one global maximum pooling layer, and the global maximum pooling layer is connected after the convolutional layer, and the multiple processing units are connected in sequence; The analysis module is specifically used to input the dense vector into the first convolution layer, so that the convolution layer and the global maximum pooling layer in the processing unit of the feature processing layer process the received vector, and use the vector processed by the last processing unit as the second feature vector.

14. The device according to claim 8, characterized in that The acquisition module is further used to acquire a training set, wherein the training set includes sample data set pairs and labels corresponding to the sample data set pairs, wherein the labels are used to identify whether data distributions between each sample data set in the corresponding sample data set pair are similar; The determination module is further configured to count the second occurrence frequency of each group of data in each sample data set in each sample data set pair; and determine the third data distribution vector of the sample data set according to the second occurrence frequency of each group of data in the sample data set; A training module is used to input each third data distribution vector of any sample data set pair into the original feature extraction model respectively to obtain the predicted feature vector corresponding to each third data distribution vector; determine the prediction result of the sample data set pair according to the similarity between each predicted feature vector in the sample data set pair; determine the loss value according to the prediction result and the label corresponding to the sample data set pair, and train the original feature extraction model based on the loss value to obtain the feature extraction model.

15. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to implement the steps of the data distribution analysis method as claimed in any one of claims 1 to 7 when executing a computer program stored in a memory.

16. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the data distribution analysis method as described in any one of claims 1 to 7.

17. A computer program product, characterized in that The computer program product comprises: a computer program code, which, when executed on a computer, enables the computer to execute the steps of the data distribution analysis method as claimed in any one of claims 1 to 7.

Images