Data distribution management method and device, equipment, storage medium and product

Abstract

The invention discloses a data distribution management method, and relates to the technical field of computers, and the data distribution management method comprises the steps that to-be-stored data sent by a second area cluster is acquired, data feature information of the to-be-stored data is extracted, and the data feature information comprises attribute information; directory matching is conducted on the data feature information, a directory matching result is obtained, and the directory matching result comprises a target directory hierarchy; and storing the to-be-stored data to a directory layer corresponding to the target directory hierarchy under a preset federated data directory so as to realize hierarchical management of data of different attribute information. According to the method, the data feature information of the to-be-stored data of the second regional cluster is matched with the federated data directory, so that ordered flow of the data between the first regional cluster and the second regional cluster and hierarchical management of the data of different attribute information are realized, and the reasonability of data distribution management is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to data distribution management methods, devices, equipment, storage media and products. Background Art

[0002] At present, with the development of big data technology, the "East Data West Computing" project has been proposed. It aims to guide the computing power demand in the east to the west in an orderly manner by building a new computing power network system integrating data centers, cloud computing, and big data, optimize the construction layout of data centers, and promote coordinated linkage between the east and the west. That is, the "East Data West Computing" project is to send the data of the eastern data center to the western data center for storage and calculation, so that the computing power resources of the western data center can more fully support the calculation of the data in the eastern data center.

[0003] Under the big framework of "East Data and West Computing", a large amount of data flows from east to west. More and more data will be stored in western data centers, or data will be placed in western data centers for calculation. How to improve the rationality of data distribution management is a technical problem that urgently needs to be solved.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a data distribution management method, aiming to solve the technical problem of how to improve the rationality of data distribution management.

[0006] To achieve the above objectives, the present application proposes a data distribution management method, the data distribution management method comprising:

[0007] Acquire the data to be stored sent by the second regional cluster, and extract data feature information of the data to be stored, wherein the data feature information includes attribute information;

[0008] Performing directory matching on the data feature information to obtain a directory matching result, wherein the directory matching result includes a target directory level;

[0009] The data to be stored is saved in a directory layer corresponding to the target directory level under a preset federal data directory to achieve hierarchical management of data with different attribute information.

[0010] Optionally, the step of performing directory matching on the data feature information to obtain a directory matching result includes:

[0011] Based on the data feature information, directory matching is performed through a preset directory allocation model to obtain a directory matching result;

[0012] The directory allocation model is obtained by performing unsupervised iterative training on a model to be trained based on data feature samples and directory feature samples, and the model to be trained is a generative adversarial network model.

[0013] Optionally, before the step of performing directory matching based on the data feature information by using a preset directory allocation model to obtain a directory matching result, the method includes:

[0014] Obtain data feature samples and catalog feature samples;

[0015] According to the preset data codes, the data feature samples and the catalog feature samples are subjected to matrix classification and integration to obtain a feature combination matrix, wherein the feature combination matrix includes a plurality of data codes, each of which includes a plurality of data feature samples and a plurality of catalog feature samples;

[0016] Calculate the similarity of the data feature samples and directory feature samples of each data code in the feature combination matrix with the data feature samples and directory feature samples of other data codes respectively, obtain the similarity value of each data code with respect to each data feature sample and directory feature sample, and form a feature similarity matrix of each data code based on the similarity values;

[0017] The feature similarity matrix is ​​visualized to obtain a data feature image of each data encoding;

[0018] The data feature image is input into a preset model to be trained for unsupervised iterative training to obtain a directory allocation model, wherein the model to be trained is a generative adversarial network model.

[0019] Optionally, the step of imaging the feature similarity matrix to obtain a data feature image of each data encoding includes:

[0020] Get the number of feature dimensions;

[0021] Dividing the feature similarity matrix into multi-dimensional feature similarity matrices corresponding to the number of feature dimensions;

[0022] Based on the preset binning rules, each similarity value in the feature similarity matrix of each dimension is equally divided into pixel values ​​to obtain the pixel grayscale value corresponding to each similarity value;

[0023] Based on the pixel grayscale values, a data feature image corresponding to the multi-dimensional feature similarity matrix of each data encoding is generated.

[0024] Optionally, the model to be trained includes a generation model and a discrimination model, and the step of inputting the data feature image into a preset model to be trained for unsupervised iterative training to obtain a directory allocation model includes:

[0025] Inputting the data feature image into the generation model, performing positive sample forgery processing based on the generation model, and generating a forged positive sample;

[0026] Based on the forged positive sample and the data feature image, solving the optimal parameter by a preset first parameter optimization method to obtain a first optimal parameter, and judging whether the first optimal parameter satisfies a preset first parameter threshold condition; if the first optimal parameter does not satisfy the first parameter threshold condition, returning to the step of inputting the data feature image into the generation model, performing positive sample forging processing based on the generation model, and generating a forged positive sample, until the first optimal parameter satisfies the first parameter threshold condition, and obtaining a target forged positive sample of the current iteration round and a trained generation model;

[0027] Input the data feature image and the target forged positive sample into the discriminant model, solve the optimal parameter by a preset second parameter optimization method, obtain the second optimal parameter, and judge whether the second optimal parameter meets the preset second parameter threshold condition; if the second optimal parameter does not meet the second parameter threshold condition, return to the step of inputting the data feature image into the generation model, performing positive sample forging processing based on the generation model, and generating forged positive samples, until the second optimal parameter meets the second parameter threshold condition, and the trained discriminant model is obtained;

[0028] Based on the trained generative model and the trained discriminative model, a directory allocation model is obtained by combining them.

[0029] Optionally, the step of performing directory matching based on the data feature information by using a preset directory allocation model to obtain a directory matching result includes:

[0030] Based on the data feature information, data coding is classified and processed by a preset directory allocation model to obtain data coding corresponding to the data feature information after classification;

[0031] Classify the classified data codes and directory information to obtain directory classification results of the data codes;

[0032] The data path corresponding to the directory classification result is determined, and the data encoding is matched with the directory corresponding to the data path to obtain a directory matching result.

[0033] In addition, to achieve the above purpose, the present application also proposes a data distribution management device, the data distribution management device comprising:

[0034] an acquisition module, configured to acquire the data to be stored sent by the second regional cluster, and extract data feature information of the data to be stored, wherein the data feature information includes attribute information;

[0035] A matching module, used for performing directory matching on the data feature information to obtain a directory matching result, wherein the directory matching result includes a target directory level;

[0036] The saving module is used to save the data to be stored in a directory layer corresponding to the target directory level under a preset federal data directory to achieve hierarchical management of data with different attribute information.

[0037] In addition, to achieve the above objectives, the present application also proposes a data distribution management device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the data distribution management method described above.

[0038] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the data distribution management method described above are implemented.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data distribution management method described above are implemented.

[0040] One or more technical solutions proposed in this application have at least the following technical effects:

[0041] The present application proposes that the first regional cluster matches the data feature information of the data to be stored in the second regional cluster with the federal data directory, so as to obtain a directory matching result including a target directory hierarchy. It can be understood that the directory matching process realizes automatic matching of the data to be stored in the second regional cluster to the federal data directory under the first regional cluster, and then realizes reasonable allocation of the data to be stored in the second regional cluster to the target directory hierarchy of the federal data directory under the first regional cluster, thereby establishing a federal data directory mount and data cache, so as to realize the orderly flow of data between the first regional cluster and the second regional cluster and the hierarchical management of data with different attribute information, thereby improving the rationality of data distribution management. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A flowchart of the first embodiment of the data distribution management method of the present application is provided;

[0045] Figure 2 This is a schematic diagram of the overall architecture of the data distribution management method of this application;

[0046] Figure 3 A schematic diagram of data flow in the application scenario of the data distribution management method of this application;

[0047] Figure 4 This is a schematic diagram of the central and western data center for the data distribution management method of this application;

[0048] Figure 5 This is a schematic diagram of the eastern data center in the data distribution management method of this application;

[0049] Figure 6 A flowchart of the second embodiment of the data distribution management method of the present application is provided;

[0050] Figure 7 This is a schematic diagram of the feature engineering process under the data distribution management method of this application;

[0051] Figure 8 This is a schematic diagram of the feature similarity matrix in the data distribution management method of this application;

[0052] Fig. 9 This is a schematic diagram of the multi-dimensional feature similarity matrix in the data distribution management method of this application;

[0053] Fig.10 A three-dimensional feature image diagram in the data distribution management method of this application;

[0054] Fig.11 It is a schematic diagram of the four-dimensional feature image in the data distribution management method of this application;

[0055] Fig.12 This is a schematic diagram of the module structure of the data distribution management device according to an embodiment of the present application;

[0056] Fig.13 Schematic diagram of the device structure of the hardware operating environment involved in the data distribution management method in the embodiment of the present application.

[0057] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0058] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0059] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0060] The main solution of the embodiment of the present application is: obtaining the data to be stored sent by the second regional cluster, and extracting data feature information of the data to be stored, wherein the data feature information includes attribute information; performing directory matching on the data feature information to obtain a directory matching result, wherein the directory matching result includes a target directory level; saving the data to be stored to a directory layer corresponding to the target directory level under a preset federal data directory to realize hierarchical management of data with different attribute information.

[0061] In this embodiment, for ease of description, the following description is made with the first regional cluster as the execution entity.

[0062] Under the big framework of "East Data and West Computing", a large amount of data flows from east to west. More and more data will be stored in western data centers, or data will be placed in western data centers for calculation. How to improve the rationality of data distribution management is a technical problem that urgently needs to be solved.

[0063] The present application provides a solution to achieve the orderly flow of data between the first regional cluster and the second regional cluster and the hierarchical management of data with different attribute information, thereby improving the rationality of data distribution management.

[0064] It can be seen from the above embodiments that the present application proposes that the first regional cluster matches the data feature information of the data to be stored in the second regional cluster with the federal data directory, so as to obtain a directory matching result including a target directory hierarchy. It can be understood that the directory matching process realizes the automatic matching of the data to be stored in the second regional cluster to the federal data directory under the first regional cluster, and then realizes the reasonable allocation of the data to be stored in the second regional cluster to the target directory hierarchy of the federal data directory under the first regional cluster, so as to establish a federal data directory mount and data cache, so as to realize the orderly flow of data between the first regional cluster and the second regional cluster and the hierarchical management of data with different attribute information, thereby improving the rationality of data distribution management.

[0065] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a terminal system, etc. The following takes the bastion host as an example to illustrate this embodiment and the following embodiments.

[0066] Based on this, the present application embodiment provides a data distribution management method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the data distribution management method of the present application.

[0067] In this embodiment, the data distribution management method includes steps S100 to S300:

[0068] Step S100, acquiring the data to be stored sent by the second regional cluster, and extracting data feature information of the data to be stored, wherein the data feature information includes attribute information;

[0069] It should be noted that both the first regional cluster and the second regional cluster are clusters deployed in the data center for storing data. The first regional cluster and the second regional cluster are deployed in different data centers. Since this application scenario involves the "East Data West Computing" project, the first regional cluster is deployed in the western data center, and the second regional cluster is deployed in the eastern data center. The functions of the western data center are mainly to establish federal data, store and pull caches according to the set algorithm strategies, or store warm and cold data in multiple eastern data centers for use in offline data processing scenarios, AI training, video rendering, supercomputing scenarios, data archiving and other scenarios. The functions of the eastern data center are similar to those of the western data center, mainly to identify which data can be shared and which data is confidential. It can also obtain certain cached data from the data federation of the western data center or combine stored data with local processing. Reference Figure 2 The construction of each data center is similar, but the usage methods are different. The western data center focuses on federal data, while the eastern data center focuses on hot data and necessary joint queries.

[0070] Further, refer to Figure 3 In the framework of East-West Data Computing, a large amount of data flows from east to west for non-real-time application scenarios, and some data flows from west to east through technical means such as verification authorization and desensitization. In this application, the main problem is how data flows between the east and west data centers and how data is calculated, that is, to define the data flow and make the setting of the data flow more reasonable.

[0071] In a specific implementation, the data to be stored refers to data from the second regional cluster of the eastern data center to be mounted / or stored in the first regional cluster of the western data center. The present application predefines the data and specifically extracts data feature information of the data to be stored from data asset management, wherein the data feature information includes attribute information, and the attribute information refers to whether the data is cold attribute data, hot attribute data, directory attribute data, etc., wherein cold (attribute) data refers to part of the data that is not allowed to be deleted from the log but is not related to the business, and hot (attribute) data refers to part of the data that is related to the business. In addition to attribute information, data features also include different categories of data such as numerical features, classification features, time features, text features, image features, and spatial features.

[0072] In the specific implementation, refer to Figure 4 , the data storage systems in the data center are diverse, which can be S3, HDFS, MinIO or other data storage systems. This solution uses Alluxio to connect to these data storage systems. There can be multiple Alluxio clusters in the same data center. Each cluster can store or cache data in different storage systems in the data center according to the application scenario. The way to read the data of the remote eastern data center is to cache the data of the eastern data center through Alluxio and query it through tools such as hive or presto. The data pulled from the eastern data center does not contain its private data. At the same time, the western data center can also perform ACL permission control on federal data.

[0073] Further, refer to Figure 5 The construction of the Eastern Data Center is roughly the same as that of the Western Data Center. The difference is that the federal data of the Eastern Data Center is based on the federal data of the Western Data Center, and the scale of data pulled from the federal data of the Western Data Center is relatively small.

[0074] Step S200, performing directory matching on the data feature information to obtain a directory matching result, wherein the directory matching result includes a target directory level;

[0075] It should be noted that the federal data directory design includes: the first level is the root directory; the second level is the name of each federal data center used in different application scenarios, such as data archiving; the third level is the name of each eastern data center; the fourth level is the directory name mounted and cached in the data archive under each eastern data center, such as 202301, 202302, etc.; the fifth level is the directory level under the directory name mounted and cached.

[0076] In a specific implementation, the first regional cluster matches the data feature information of the data to be stored in the second regional cluster with the federal data directory to obtain a directory matching result including a target directory hierarchy. It can be understood that the directory matching process realizes automatic matching of the data to be stored in the second regional cluster to the federal data directory under the first regional cluster, and then realizes reasonable allocation of the data to be stored in the second regional cluster to the target directory hierarchy of the federal data directory under the first regional cluster, thereby establishing a federal data directory mount and data cache, so as to realize the orderly flow of data between the first regional cluster and the second regional cluster and the hierarchical management of data with different attribute information, thereby improving the rationality of data distribution management.

[0077] In a specific implementation, the first regional cluster performs directory matching on the data feature information to obtain a directory matching result, including:

[0078] Based on the data feature information, directory matching is performed through a preset directory allocation model to obtain a directory matching result; wherein the directory allocation model is obtained by performing unsupervised iterative training on a model to be trained based on data feature samples and directory feature samples, and the model to be trained is a generative adversarial network model.

[0079] In the specific implementation, the first regional cluster analyzes the characteristic attributes of storage data and the characteristic attributes of mounting and cache directories, and uses these two types of characteristic data as model input, uses collaborative filtering to calculate the similarity of data features, and then processes the similarity into a data feature similarity image, combines the "generative adversarial network" algorithm to perform image classification modeling, and finally outputs the data and directory correspondence data.

[0080] In a specific implementation, the first regional cluster performs directory matching based on the data feature information through a preset directory allocation model to obtain a directory matching result, including:

[0081] Based on the data feature information, data coding is classified and processed through a preset directory allocation model to obtain the data coding corresponding to the data feature information after classification; the classified data coding and directory information are classified to obtain a directory classification result of the data coding; the data path corresponding to the directory classification result is determined, and the data coding is matched with the directory corresponding to the data path to obtain a directory matching result.

[0082] In the specific implementation, the first regional cluster classifies the data encoding through the directory allocation model, classifies the data encoding, and each type of data encoding has a high similarity. Then, through the data encoding characteristics and the directory information in the data directory matrix, the topN directory classifications in the same category are summarized (N can be set according to actual business needs), and then the actual path of the directory classification in the running environment is matched. The corresponding directory of the data encoding can be obtained, and the data encoding and cache directory matching are completed. The function of this solution is realized, that is, the matching of the feature visualization algorithm based on similarity is realized to achieve the purpose of optimizing data distribution.

[0083] Step S300, saving the data to be stored to a directory layer corresponding to the target directory level under a preset federal data directory, so as to realize hierarchical management of data with different attribute information.

[0084] In a specific implementation, the first regional cluster saves the data to be stored to a directory layer corresponding to the target directory level under a preset federal data directory to realize hierarchical management of data with different attribute information. That is, the above method establishes a federal data directory mounting and data caching method to orderly flow data between the eastern and western data centers to realize hierarchical management of hot and cold data.

[0085] The present application proposes that the first regional cluster matches the data feature information of the data to be stored in the second regional cluster with the federal data directory, so as to obtain a directory matching result including a target directory hierarchy. It can be understood that the directory matching process realizes automatic matching of the data to be stored in the second regional cluster to the federal data directory under the first regional cluster, and then realizes reasonable allocation of the data to be stored in the second regional cluster to the target directory hierarchy of the federal data directory under the first regional cluster, thereby establishing a federal data directory mount and data cache, so as to realize the orderly flow of data between the first regional cluster and the second regional cluster and the hierarchical management of data with different attribute information, thereby improving the rationality of data distribution management.

[0086] Based on the above first embodiment, the present application also proposes another embodiment, referring to Figure 6 , the data distribution management method comprises:

[0087] Before the step of the first regional cluster performing directory matching based on the data feature information by using a preset directory allocation model to obtain a directory matching result, the method includes the following steps A100-A500:

[0088] Step A100, obtaining data feature samples and directory feature samples;

[0089] In a specific implementation, the data feature samples and the directory feature samples are feature samples used for model training. The first regional cluster obtains the data feature samples by extracting data features from data asset management, and the first regional cluster obtains the directory feature samples by obtaining directory features from system integration management.

[0090] Step A200, according to the preset data codes, matrix classification and integration of the data feature samples and the catalog feature samples are performed to obtain a feature combination matrix, wherein the feature combination matrix includes a plurality of data codes, each of which includes a plurality of data feature samples and a plurality of catalog feature samples;

[0091] In the specific implementation, data feature samples and directory feature samples are associated according to data codes and integrated into data feature and directory feature matrices. Each data code contains n data features and m directory features. Finally, data samples are generated, as shown in Table 1 below:

[0092]

[0093]

[0094] Furthermore, after determining the feature combination matrix of the first regional cluster, it is necessary to perform data feature engineering analysis on it, referring to Figure 7 The feature engineering process involved in this application mainly includes: feature understanding, feature cleaning and enhancement, feature construction, feature selection, etc.

[0095] Step A300, performing similarity calculations on the data feature samples and directory feature samples of each data code in the feature combination matrix with the data feature samples and directory feature samples of other data codes, obtaining similarity values ​​of each data code with respect to each data feature sample and directory feature sample, and composing a feature similarity matrix of each data code based on the similarity values;

[0096] In the specific implementation, in order to more accurately separate the key features of the data and improve the accuracy of the model algorithm. It is necessary to perform similarity analysis and calculation on the data features after feature engineering processing and the directory feature matrix. The accuracy of the similarity calculation between data directly affects the effect of model training, thereby affecting the accuracy of the data directory output by the model. Specifically, this application uses a collaborative filtering method to calculate the similarity between the data and directory features in the matrix according to the data encoding. Traverse each data encoding, calculate the similarity between the current data encoding and all other data encodings on each feature, obtain the similarity value of any two data encodings on each feature, and generate a feature similarity matrix for each data encoding. For details, please refer to Figure 8 shown.

[0097] Step A400, imaging the feature similarity matrix to obtain a data feature image of each data code;

[0098] It should be noted that visualization is the process of converting a feature similarity matrix into a feature similarity image. This application proposes dimensional feature visualization, which converts abstract data encoding similarity into an image, and then uses the advantage algorithm of the image solution to process the data feature image.

[0099] In a specific implementation, the first regional cluster visualizes the feature similarity matrix to obtain a data feature image of each data encoding, including:

[0100] Acquire the number of feature dimensions; divide the feature similarity matrix into multi-dimensional feature similarity matrices corresponding to the number of feature dimensions; based on a preset binning rule, divide each similarity value in the feature similarity matrix of each dimension equally by pixel value to obtain a pixel grayscale value corresponding to each similarity value; based on the pixel grayscale value, generate a data feature image corresponding to the multi-dimensional feature similarity matrix of each data encoding.

[0101] In the specific implementation, the first regional cluster first splits the data feature similarity matrix into a multi-dimensional feature similarity matrix, and outputs a multi-dimensional matrix for each data code in the form of: number of data codes * number of features * feature dimension. (The number of data refers to the number of other data codes for comparison; the number of features refers to the number of features under the dimension; the feature dimension is to split all the features of the data code into multiple dimensions as needed, such as dividing the basic information of the data into the basic data dimension, dividing the information related to the classification features such as business domain classification information, period classification, etc. into the data classification dimension, and dividing the data period in the time feature into the data time dimension)

[0102] For example, in a dataset of 100 data codes in 3 dimensions (10 features in each dimension), the result is: the first element of the first matrix represents the similarity value between the first feature of data code 1 in dimension 1 and the first feature of data code 2 in dimension 1. Similarly, generate a similarity matrix between data code 1 in maintenance 1 and the other 99 data codes. Similarly, generate data in the other 2 dimensions. Finally, generate a multidimensional matrix of the similarity between data code 1 in 3 dimensions and the other 99 data codes. The similarity calculation result for all 100 data codes is the multidimensional matrix of the similarity between 100 data codes, each data code in 3 dimensions and 10 features in each dimension, and the other 99 data codes. For details, please refer to Fig. 9 shown.

[0103] Furthermore, the three-dimensional matrix of the three dimensions (basic data dimension, data classification dimension, and data time dimension) described above is similar to the red, green, and blue three-pipeline pixel matrix in image processing. Each element (i.e., similarity value) in each dimensional matrix is ​​binned according to a specific rule for encoding the full amount of data. The rule can be the range (or other rules, such as normalization, quartile difference, etc.), and the range (maximum value - minimum value) is divided into 256 intervals, corresponding to 0 to 255 intervals from small to large, forming 256 pixel grayscales from light to dark.

[0104] The process of dimensional feature visualization is to convert the data encoding features into a feature image. The red, green and blue channels of the image are the three business dimensions of the data encoding; the pixels in each channel are the values ​​of 256 bin groups of similarity values ​​of a feature of each data encoding and the corresponding features of other data encodings; each data encoding has a set of such multidimensional matrices, that is, each data encoding corresponds to a dimensional feature image, which can be referred to for details. Fig.10 shown.

[0105] It should be noted that according to the special needs of different business scenarios, this application can also process features into more dimensions, such as extending features to location tracks, friend circles, family circles, etc. Here, four dimensions are taken as an example to form a 4-dimensional matrix, and the transparency pipeline is increased with reference to PNG images to form a 4-channel image. For details, please refer to Fig.11 As shown. Based on the above steps, each data encoding has been transformed into a picture based on the similarity of multi-dimensional features with other data encodings. The picture set of all data encodings can be used as input, and then the picture classification algorithm is used for model training to classify the pictures by similarity.

[0106] Step A500, inputting the data feature image into a preset model to be trained for unsupervised iterative training to obtain a directory allocation model, wherein the model to be trained is a generative adversarial network model.

[0107] In the specific implementation, this application uses the data encoding dimension feature image obtained by dimensional feature image processing as input, and uses the generative adversarial network model framework (CFGAN) based on collaborative filtering to perform unsupervised classification on the image. Since each image is a data code, and each pixel of the image is the feature similarity value between data codes, the data codes under the same category are a set of data codes with high similarity.

[0108] In a specific implementation, the first regional cluster inputs the data feature image into a preset model to be trained for unsupervised iterative training to obtain a directory allocation model, including:

[0109] The data feature image is input into the generative model, and positive sample forgery processing is performed based on the generative model to generate a forged positive sample; based on the forged positive sample and the data feature image, the optimal parameter is solved by a preset first parameter optimization method to obtain a first optimal parameter, and it is determined whether the first optimal parameter meets the preset first parameter threshold condition; if the first optimal parameter does not meet the first parameter threshold condition, the step of inputting the data feature image into the generative model, and positive sample forgery processing is performed based on the generative model to generate a forged positive sample is returned until the first optimal parameter meets the first parameter threshold condition to obtain the target forged positive sample of the current iteration round and the training set. a trained generative model; inputting the data feature image and the target forged positive sample into the discriminative model, solving the optimal parameter by a preset second parameter optimization method to obtain a second optimal parameter, and judging whether the second optimal parameter meets the preset second parameter threshold condition; if the second optimal parameter does not meet the second parameter threshold condition, returning to the step of inputting the data feature image into the generative model, performing positive sample forging processing based on the generative model, and generating a forged positive sample, until the second optimal parameter meets the second parameter threshold condition to obtain the trained discriminative model; combining the trained generative model and the trained discriminative model to obtain a directory allocation model.

[0110] In the specific implementation, the generative adversarial network consists of a generative model G and a discriminative model D, which compete in adversarial mode: D tries to distinguish between real positive samples and G's forged positive samples, while G tries to deceive D. When the adversarial relationship is relatively balanced, GAN training is completed, and the cost function of GAN is:

[0111]

[0112] Where m is the sample in the training set, U represents the data encoding set, un represents the nth data encoding, φ and θ represent the parameters of the discriminant model D and the generative model G respectively.

[0113] Train the discriminant model, that is, maximize the probability of correctly distinguishing true positive samples from G's forged positive samples, as shown in the following formula:

[0114]

[0115] Among them, m+ and m- represent positive samples and negative samples respectively, and mg represents the positive samples forged by the generative model G. In contrast to the discriminative model D, the generative model G aims to minimize the correct discrimination probability of the discriminative model D, that is, to deceive the discriminative model. As shown in the following formula 1: G extracts negative samples from the sample pool according to the relevant probability distribution, so the parameter optimization of the generative model is shown in the following formula 2, where M represents the sample set:

[0116]

[0117] Since the samples are discrete during sampling, the optimal solution cannot be obtained directly through the gradient descent of the above formula. It is necessary to use the policy gradient method based on reinforcement learning for approximate solution. K is the size of the list of sample pairs that need to be judged, as shown in the following formula:

[0118]

[0119] The process of using this formula to find the optimal solution for the feature data set encoded by the sample data is the process of model generation. In this process, the precision (Precision) and recall (Recall) of the TOP N sorting and the F1-score are used to evaluate the model, and the more robust mean absolute percentage error (Mean Absolute Percent Error) is combined to evaluate the model output results to reduce the impact of the absolute error caused by individual outliers. The training and evaluation process is iterated until the model output is closer to the optimal solution of the algorithm, and finally the data encoding feature image classification is output, that is, the data encoding is classified according to the feature similarity image.

[0120] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data distribution management method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0121] This application also provides a data distribution management device, please refer to Fig.12 , the data distribution management device comprises:

[0122] An acquisition module 10 is used to acquire the data to be stored sent by the second regional cluster, and extract data feature information of the data to be stored, wherein the data feature information includes attribute information;

[0123] A matching module 20, configured to perform directory matching on the data feature information to obtain a directory matching result, wherein the directory matching result includes a target directory level;

[0124] The saving module 30 is used to save the data to be stored in a directory layer corresponding to the target directory level under a preset federal data directory, so as to realize hierarchical management of data with different attribute information.

[0125] Optionally, the matching module 20 includes:

[0126] The directory matching module is used to perform directory matching based on the data feature information through a preset directory allocation model to obtain a directory matching result; wherein the directory allocation model is obtained by performing unsupervised iterative training on a model to be trained based on data feature samples and directory feature samples, and the model to be trained is a generative adversarial network model.

[0127] Optionally, the data distribution management device further includes:

[0128] A sample acquisition module is used to acquire data feature samples and directory feature samples;

[0129] An integration module, used to perform matrix classification and integration on the data feature samples and the catalog feature samples according to preset data codes to obtain a feature combination matrix, wherein the feature combination matrix includes a plurality of data codes, each of which includes a plurality of data feature samples and a plurality of catalog feature samples;

[0130] A calculation module, used for calculating the similarity between the data feature samples and the directory feature samples of each data code in the feature combination matrix and the data feature samples and the directory feature samples of other data codes, respectively, to obtain the similarity value of each data code with respect to each data feature sample and directory feature sample, and to form a feature similarity matrix of each data code based on the similarity values;

[0131] An imaging module, used to image the feature similarity matrix to obtain a data feature image of each data encoding;

[0132] The training module is used to input the data feature image into a preset model to be trained for unsupervised iterative training to obtain a directory allocation model, wherein the model to be trained is a generative adversarial network model.

[0133] Optionally, the imaging module includes:

[0134] A dimension quantity acquisition module is used to obtain the number of feature dimensions;

[0135] A matrix division module, used for dividing the feature similarity matrix into multi-dimensional feature similarity matrices corresponding to the number of feature dimensions;

[0136] The pixel value equal division module is used to divide the similarity values ​​in the feature similarity matrix of each dimension into equal pixel values ​​based on the preset binning rules to obtain the pixel grayscale value corresponding to each similarity value;

[0137] The image generation module is used to generate a data feature image corresponding to the multi-dimensional feature similarity matrix of each data encoding based on the pixel gray value.

[0138] Optionally, the training module includes:

[0139] A fake image generation module, used for inputting the data feature image into the generation model, performing positive sample forging processing based on the generation model, and generating a forged positive sample;

[0140] A generation model optimization module, used to solve the optimal parameters based on the forged positive sample and the data feature image by a preset first parameter optimization method to obtain a first optimal parameter, and determine whether the first optimal parameter meets a preset first parameter threshold condition. If the first optimal parameter does not meet the first parameter threshold condition, return to the step of inputting the data feature image into the generation model, performing positive sample forging processing based on the generation model, and generating a forged positive sample, until the first optimal parameter meets the first parameter threshold condition, and obtains a target forged positive sample of the current iteration round and a trained generation model;

[0141] a discriminant model optimization module, used for inputting the data feature image and the target forged positive sample into the discriminant model, solving the optimal parameter by a preset second parameter optimization method, obtaining a second optimal parameter, and judging whether the second optimal parameter satisfies a preset second parameter threshold condition; if the second optimal parameter does not satisfy the second parameter threshold condition, returning to the step of inputting the data feature image into the generative model, performing positive sample forging processing based on the generative model, and generating a forged positive sample, until the second optimal parameter satisfies the second parameter threshold condition, and obtaining a trained discriminant model;

[0142] The combination module is used to combine the trained generation model and the trained discrimination model to obtain a directory allocation model.

[0143] Optionally, the matching module 20 includes:

[0144] A classification module, used to perform classification processing of data codes based on the data feature information through a preset directory allocation model to obtain data codes corresponding to the data feature information after classification;

[0145] A classification module is used to classify the classified data codes and directory information to obtain a directory classification result of the data codes;

[0146] The directory matching result determination module is used to determine the data path corresponding to the directory classification result, and match the data code with the directory corresponding to the data path to obtain the directory matching result.

[0147] The data distribution management device provided by the present application adopts the data distribution management method in the above embodiment, which can solve the technical problem of data distribution management. Compared with the prior art, the beneficial effects of the data distribution management device provided by the present application are the same as the beneficial effects of the data distribution management method provided by the above embodiment, and other technical features in the data distribution management device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0148] The present application provides a data distribution management device, which includes: at least one processor; and a memory that is communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data distribution management method in the above-mentioned embodiment one.

[0149] Reference below Fig.13 , which shows a schematic diagram of the structure of a data distribution management device suitable for implementing the embodiment of the present application. The data distribution management device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.13 The data distribution management device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0150] like Fig.13As shown, the data distribution management device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data distribution management device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data distribution management device to communicate with other devices wirelessly or by wire to exchange data. Although the data distribution management device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0151] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0152] The data distribution management device provided by the present application adopts the data distribution management method in the above embodiment to solve the technical problem of data distribution management. Compared with the prior art, the beneficial effects of the data distribution management device provided by the present application are the same as the beneficial effects of the data distribution management method provided by the above embodiment, and the other technical features in the data distribution management device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0153] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0154] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0155] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the data distribution management method in the above-mentioned embodiment.

[0156] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0157] The computer-readable storage medium may be included in the data distribution management device; or may exist independently without being assembled into the data distribution management device.

[0158] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the data distribution management device, the data distribution management device performs data distribution management.

[0159] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0160] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0161] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0162] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data distribution management method, and can solve the technical problems of data distribution management. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data distribution management method provided in the above-mentioned embodiment, and will not be described in detail here.

[0163] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data distribution management method when executed by a processor.

[0164] The computer program product provided by this application can solve the technical problem of data distribution management. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the data distribution management method provided by the above embodiment, which will not be repeated here.

[0165] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data distribution management method, characterized in that: Applied to a first regional cluster, the first regional cluster is communicatively connected with a second regional cluster, and the data distribution management method includes: Acquire the data to be stored sent by the second regional cluster, and extract data feature information of the data to be stored, wherein the data feature information includes attribute information; Performing directory matching on the data feature information to obtain a directory matching result, wherein the directory matching result includes a target directory level; The data to be stored is saved in a directory layer corresponding to the target directory level under a preset federal data directory to achieve hierarchical management of data with different attribute information.

2. The data distribution management method according to claim 1, characterized in that: The step of performing directory matching on the data feature information to obtain a directory matching result includes: Based on the data feature information, directory matching is performed through a preset directory allocation model to obtain a directory matching result; The directory allocation model is obtained by performing unsupervised iterative training on a model to be trained based on data feature samples and directory feature samples, and the model to be trained is a generative adversarial network model.

3. The data distribution management method according to claim 2, characterized in that: Before the step of performing directory matching based on the data feature information through a preset directory allocation model to obtain a directory matching result, the method includes: Obtain data feature samples and catalog feature samples; According to the preset data codes, the data feature samples and the catalog feature samples are subjected to matrix classification and integration to obtain a feature combination matrix, wherein the feature combination matrix includes a plurality of data codes, each of which includes a plurality of data feature samples and a plurality of catalog feature samples; Calculate the similarity of the data feature samples and directory feature samples of each data code in the feature combination matrix with the data feature samples and directory feature samples of other data codes respectively, obtain the similarity value of each data code with respect to each data feature sample and directory feature sample, and form a feature similarity matrix of each data code based on the similarity values; The feature similarity matrix is ​​visualized to obtain a data feature image of each data encoding; The data feature image is input into a preset model to be trained for unsupervised iterative training to obtain a directory allocation model, wherein the model to be trained is a generative adversarial network model.

4. The data distribution management method according to claim 3, characterized in that: The step of imaging the feature similarity matrix to obtain a data feature image of each data encoding includes: Get the number of feature dimensions; Dividing the feature similarity matrix into multi-dimensional feature similarity matrices corresponding to the number of feature dimensions; Based on the preset binning rules, each similarity value in the feature similarity matrix of each dimension is equally divided into pixel values ​​to obtain the pixel grayscale value corresponding to each similarity value; Based on the pixel grayscale values, a data feature image corresponding to the multi-dimensional feature similarity matrix of each data encoding is generated.

5. The data distribution management method according to claim 3, characterized in that: The model to be trained includes a generation model and a discrimination model. The step of inputting the data feature image into a preset model to be trained for unsupervised iterative training to obtain a directory allocation model includes: Inputting the data feature image into the generation model, performing positive sample forgery processing based on the generation model, and generating a forged positive sample; Based on the forged positive sample and the data feature image, solving the optimal parameter by a preset first parameter optimization method to obtain a first optimal parameter, and judging whether the first optimal parameter satisfies a preset first parameter threshold condition; if the first optimal parameter does not satisfy the first parameter threshold condition, returning to the step of inputting the data feature image into the generation model, performing positive sample forging processing based on the generation model, and generating a forged positive sample, until the first optimal parameter satisfies the first parameter threshold condition, and obtaining a target forged positive sample of the current iteration round and a trained generation model; Input the data feature image and the target forged positive sample into the discriminant model, solve the optimal parameter by a preset second parameter optimization method, obtain the second optimal parameter, and judge whether the second optimal parameter meets the preset second parameter threshold condition; if the second optimal parameter does not meet the second parameter threshold condition, return to the step of inputting the data feature image into the generation model, performing positive sample forging processing based on the generation model, and generating forged positive samples, until the second optimal parameter meets the second parameter threshold condition, and the trained discriminant model is obtained; Based on the trained generative model and the trained discriminative model, a directory allocation model is obtained by combining them.

6. The data distribution management method according to claim 2, characterized in that: The step of performing directory matching based on the data feature information through a preset directory allocation model to obtain a directory matching result includes: Based on the data feature information, data coding is classified and processed by a preset directory allocation model to obtain data coding corresponding to the data feature information after classification; Classify the classified data codes and directory information to obtain directory classification results of the data codes; The data path corresponding to the directory classification result is determined, and the data encoding is matched with the directory corresponding to the data path to obtain a directory matching result.

7. A data distribution management device, characterized in that: The device comprises: an acquisition module, configured to acquire the data to be stored sent by the second regional cluster, and extract data feature information of the data to be stored, wherein the data feature information includes attribute information; A matching module, used for performing directory matching on the data feature information to obtain a directory matching result, wherein the directory matching result includes a target directory level; The saving module is used to save the data to be stored in a directory layer corresponding to the target directory level under a preset federal data directory to achieve hierarchical management of data with different attribute information.

8. A data distribution management device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data distribution management method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data distribution management method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data distribution management method according to any one of claims 1 to 6 are implemented.

Images