A data processing method and device, electronic equipment and storage medium

Abstract

The application discloses a data processing method and device, electronic equipment and a storage medium, and the method comprises the following steps: determining a data processing model structure in a data processing library according to a data type of a data source; generating a data processing model based on a training set of the data source and the data processing model structure; and determining shared data corresponding to original data of the data source according to the data processing model, wherein the shared data has the same manifold structure and probability distribution characteristics as the original data. The privacy of the original data is processed by the data processing model, the difficulty of data sharing is reduced, and the utilization rate of the data is increased.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, and in particular to a data processing method and device, electronic equipment and storage medium. BACKGROUND

[0002] The three major elements of machine learning are data, algorithm and computing power. With the development of hardware and software technologies, the algorithm and computing power have been greatly improved. With the advent of big data, machine learning has become a hot research topic. Although the application of machine learning in various fields has brought an intelligent revolution to the industry, it is not easy to land a machine learning project. The reason is that data acquisition has become a bottleneck for machine learning. In machine learning, there are two requirements for data storage: 1. Data can be fully collected; 2. Data can be used centrally. However, these requirements are often not met in practical applications. For example, the data collection cost is too high for small-scale companies, resulting in insufficient data collection; commercial data secrecy requirements are too high, making it difficult to use data centrally. Real-world machine learning is difficult to implement.

[0003] The current solution to the above problem is data sharing. Common data sharing methods mainly include federated learning and secure multi-party computation. Federated learning requires data owners and users to be online at the same time to complete the computing task together, and the computing power required by each party is proportional to the amount of data. Secure multi-party computation requires each party to participate in the calculation online, and each party needs to communicate data due to the need for underlying protocols. The existing data sharing method requires data providers and data users to be online at the same time to complete the training of the machine learning task together. This method binds data providers and data users, increasing the difficulty of data sharing. There is an urgent need for a data processing method that decouples data providers and data users, reduces the difficulty of data sharing, and improves the efficiency of sharing. SUMMARY

[0004] The present application provides a data processing method, device, electronic equipment and storage medium to realize the decoupling of data providers and data users, reduce the difficulty of data sharing, and increase the utilization rate of data.

[0005] In a first aspect, the embodiments of the present application provide a data processing method, wherein the method comprises:

[0006] determining a data processing model structure in a data processing library according to the data type of the data source;

[0007] generating a data processing model based on the training set of the data source and the data processing model structure;

[0008] determine shared data corresponding to the original data of the data source according to the data processing model, wherein the shared data has the same manifold structure and probability distribution characteristics as the original data.

[0009] In a second aspect, an embodiment of the present application further provides a data processing apparatus, and the apparatus comprises:

[0010] a model structure module configured to determine a data processing model structure according to a data type of a data source in a data processing library;

[0011] a model training module configured to generate a data processing model based on a training set of the data source and the data processing model structure;

[0012] a shared data module configured to determine shared data corresponding to the original data of the data source according to the data processing model, wherein the shared data has the same manifold structure and probability distribution characteristics as the original data.

[0013] In a third aspect, an embodiment of the present application further provides an electronic device, and the electronic device comprises:

[0014] one or more processors;

[0015] a memory configured to store one or more programs,

[0016] when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of the embodiments of the present application.

[0017] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the data processing method according to any one of the embodiments of the present application.

[0018] In the embodiments of the present application, the data processing model structure corresponding to the data type of the data source is selected in the data processing library, the data processing model structure is trained using the training set from the data source to generate the data processing model, and the original data of the data source is processed using the data processing model to generate the shared data, and the shared data has the same manifold structure and probability distribution characteristics as the original data. The embodiments of the present application realize the privacy processing of the original data through the data processing model, can reduce the difficulty of data sharing, and thus increase the utilization rate of data. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of a data sharing method provided by an embodiment of the present application;

[0020] Figure 2is a flow chart of a data sharing method provided by the second embodiment of the present application;

[0021] Figure 3 is a training schematic diagram of an encoder and a decoder provided by the second embodiment of the present application;

[0022] Figure 4 is a training schematic diagram of an encoder and a decoder provided by the second embodiment of the present application;

[0023] Figure 5 is a training schematic diagram of an encoder and a decoder provided by the second embodiment of the present application;

[0024] Figure 6 is an example diagram of a data sharing method provided by the second embodiment of the present application;

[0025] Figure 7 is a structural schematic diagram of a data sharing device provided by the third embodiment of the present application;

[0026] Figure 8 is a structural schematic diagram of an electronic device provided by the fourth embodiment of the present application. DETAILED DESCRIPTION

[0027] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures, and in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0028] Embodiment one

[0029] Figure 1 is a flow chart of a data sharing method provided by the first embodiment of the present application, the embodiment can be applicable to the case of data sharing, and the method can be executed by a data sharing device, which can be realized in the form of hardware and / or software, see Figure 1 , the method provided by the embodiment of the present application specifically includes the following steps:

[0030] Step 110, determining a data processing model structure in a data processing library according to the data type of the data source.

[0031] The data source can be a data storage location of a data owner, and the data source can include a logical address or a physical address of the data. The data processing library can be a model structure library pre-built by a data sharing scheme provider, and the data processing library can include one or more data processing model structures that can be used to generate shared data having the same manifold structure and probability distribution characteristics as the original data. It can be understood that the data processing model structure can be a manifold learning model structure, and can perform dimension reduction processing on high-dimensional data.

[0032] In the embodiment of the present application, the data sharing scheme provider can pre-build a data processing library, and the data owner can query a data processing model structure for processing the data type in the data source in the data processing library when sharing data.

[0033] Step 120, generating a data processing model based on the training set of the data source and the data processing model structure.

[0034] The training set can be a data set for training the data processing model structure, and the data in the training set can include at least one of image data, table data, or medical detection data.

[0035] Specifically, the data in the data source can be read as a training set, and the training set can be input into the data processing model structure for training. The parameters of the data processing model structure can be continuously adjusted during the training process until the output result of the data processing model corresponding to the data processing structure meets the training end condition.

[0036] Step 130, determining shared data corresponding to the original data of the data source according to the data processing model, wherein the shared data has the same manifold structure and probability distribution characteristics as the original data.

[0037] The original data can be data in the data source, and can be information containing private data. The original data cannot be shared with a third party. The shared data can be de-identified data, and the shared data can have the same probability distribution characteristics as the original data. The shared data can have the same effect as the original data in the field of machine learning. The manifold structure and the probability distribution characteristics can be the manifold structure and the probability law of the value of the original data.

[0038] In the embodiment of the present application, after the training of the data processing model is completed, the original data can be read from the data source, and the data processing model can be used to process the original data. The data output by the data processing model can be used as shared data. It can be understood that the data processing model can process the original data to generate shared data that is different from the original data but contains the manifold structure and the probability distribution characteristics of the original data. The shared data can be used by a third party.

[0039] In the embodiment of the application, the data processing model structure corresponding to the data type of the data source is selected in the data processing library, the data processing model structure is trained using the training set from the data source to generate a data processing model, and the original data of the data source is processed using the data processing model to generate shared data, which has the same manifold structure and probability distribution characteristics as the original data. The embodiment of the application realizes the privacy processing of the original data through the data processing model, reduces the difficulty of data sharing, and thus increases the utilization rate of data.

[0040] Embodiment two

[0041] Figure 2 is a flowchart of a data sharing method provided by the second embodiment of the application. The embodiment of the application is a specific embodiment based on the above-mentioned application embodiment. Referring to Figure 2 The method provided by the embodiment of the application specifically includes the following steps:

[0042] Step 210, reading the data type of the original data in the data source.

[0043] The original data can be data stored in the data source, and the original data can include private data that cannot be directly shared with a third party.

[0044] In the embodiment of the application, the data type of the original data can be read from the data source, for example, the data type of the original data can be extracted from the database metadata.

[0045] Step 220, finding a data processing model structure matching the data type in the data processing library.

[0046] Specifically, the data processing library can store the data type and the data processing model structure in association. When the original data of the data source is processed, the corresponding data processing model structure can be found in the data processing library according to the extracted data type. For example, the data processing model structure in the data processing library can have the same identification number as its corresponding data type.

[0047] Step 230, collecting original data from the data source as a training set.

[0048] Specifically, a connection can be established with the data source, and the connection can be used to read a threshold number of original data as a training set. The threshold number can be determined by the data processing model structure. For example, different data processing model structures can be configured with a threshold number, which can be the minimum requirement for the amount of data in the training set. The more data in the training set, the more it reflects the manifold structure and probability distribution of the data, and the more accurate the training of the data processing structure.

[0049] Step 240, training the data processing model structure according to the training set to generate an encoder and a decoder of the data processing model.

[0050] Wherein, the encoder can be a machine learning model for processing high-dimensional original data into low-dimensional data, the decoder can be a machine learning model for processing low-dimensional data into high-dimensional data, the encoder and the decoder jointly constitute a flow type learning model, the decoder can be opposite to the processing process of the encoder, and the encoder and the decoder can be a convolutional neural network model or a fully connected network model.

[0051] In the embodiment of the application, the data processing model can include an encoder and a decoder, the encoder and the decoder can be trained using the collected training set, and the trained encoder and decoder can be used as the data processing model.

[0052] Step 250, inputting the original data of the data source into the encoder of the data processing model to generate point cloud data in a low-dimensional space.

[0053] Wherein, the point cloud data can be original data processed by the encoder, the point cloud data can be composed of one or more original data processed by the encoder, and the dimension of the data included in the point cloud data is determined by the user-specified hyperparameter.

[0054] In the embodiment of the application, the trained encoder can perform dimension reduction processing on the original data read by the data source, and the processing result can be used as point cloud data, and each data coordinate in the point cloud data can correspond to an original data of the data source.

[0055] Step 260, data reduction is performed on the point cloud data.

[0056] In the embodiment of the application, the point cloud data can be subjected to data reduction, and the probability of each data in the point cloud data can be converted from a point representation form to a weight representation form, so that the convergence stability and efficiency of the subsequent mapping relationship can be improved.

[0057] Step 270, determining the mapping relationship between the data distribution probability corresponding to the point cloud data and the specified probability distribution.

[0058] Wherein, the specified probability distribution can be a pre-set probability distribution, and can include uniform distribution and Gaussian distribution.

[0059] Specifically, the mapping relationship can be solved based on the probability distribution of the point cloud data and the specified probability distribution, and the mapping can reflect the mapping condition of the specified probability distribution to the probability distribution of the point cloud data.

[0060] Step 280, sampling to generate a data point set according to the specified probability distribution.

[0061] In the embodiment of the present application, the data resampling can be performed in the data space according to the specified probability distribution, so that the collected data meets the requirement of the specified probability, and the data obtained by resampling can be composed into a data point set.

[0062] In step 290, the data point set is mapped to the data distribution probability according to the mapping relationship.

[0063] Specifically, the data point set can be processed according to the mapping relationship obtained above, so that each data in the mapped data point set meets the data distribution probability of the original data.

[0064] In step 2100, data with a similarity greater than a similarity threshold in the point cloud data is removed from the mapped data point set.

[0065] The similarity threshold can be a parameter for judging the similarity of the original data and the shared data, and the similarity threshold can be set by the user according to the device performance and processing speed.

[0066] In the embodiment of the present application, the similarity of each data in the data point set and the point cloud data can be determined, which can include probability value, vector distance or Euclidean distance, etc. When the similarity of the data in the data point set and the data in the point cloud data is greater than the similarity threshold, the data is removed from the data point set.

[0067] In step 2110, the data point set is input into the decoder to generate shared data.

[0068] Specifically, the data in the data point set can be input into the decoder, and the data can be lifted from the low-dimensional space to the high-dimensional space corresponding to the original data by the decoder. The data after lifting the dimension can be used as shared data.

[0069] In the embodiment of the present application, the data type of the original data in the data source is read, the matching data processing model structure is obtained according to the data type, the original data in the data source is collected as a training set, the encoder and the decoder in the data processing model structure are trained using the training set, the original data of the data source is processed into point cloud data in a low-dimensional space using the encoder, the mapping relationship between the data distribution probability and the specified distribution probability is determined by data reduction on the point cloud data, the data point set is generated by resampling in the data space according to the specified distribution probability, the data point set is mapped to the data distribution probability according to the mapping relationship, the data with a similarity greater than a similarity threshold in the data point set is removed, and the data point set is converted into shared data using the decoder. The embodiment of the present application realizes the privacy processing of the original data through the data processing model, preserves the manifold structure and probability distribution characteristics of the original data, realizes the sharing and utilization of the shared data, removes the data with too high similarity, further prevents privacy leakage, and reduces the difficulty of sharing the private data.

[0070] Furthermore, based on the above embodiments of the invention, training the data processing model structure according to the training set to generate the encoder and decoder of the data processing model includes:

[0071] The encoder and decoder are trained layer by layer from low resolution to high resolution using the training set of the image data type; mean square error is used as the loss function to control the parameter updates of the encoder and decoder; wherein the encoder and decoder are graph convolutional neural network models, each including at least a convolutional layer, a linear rectified layer and a batch normalization layer.

[0072] In this embodiment of the invention, when the data source is an image data type, the encoder and decoder in the data processing model can be graph convolutional neural networks (GNNs). These GNNs include at least convolutional layers, linear rectified layers, and batch normalization layers. The encoder and decoder can be trained using a training set composed of image data. The training process can be performed multiple times at different resolutions: first, training is performed on the low-resolution convolutional layers, then on the high-resolution convolutional layers. After each training iteration, the mean squared error (MSE) between the encoder / decoder output and the input image data in the training set can be calculated. The MSE can be used to determine the training status of the encoder and decoder. When the MSE is less than a preset error value, the encoder and decoder training is considered complete. See also... Figure 3 For image data, embodiments of this invention can use a Convolutional Neural Network (CNN) model to implement an autoencoder including an encoder and a decoder. The autoencoder is constructed using a progressive resolution feature extraction method to learn the manifold structure of the image data. The number of layers in the autoencoder can be related to the resolution of the image data. Taking a 128*128 image as an example, the encoder and decoder can be trained layer by layer starting from a 4*4 resolution. Each layer of the encoder module contains one convolutional (Cov) layer, one batch normalization (BN) layer, and one linear rectification function (RELU) layer. This autoencoder can be trained layer by layer starting from a low resolution, for example, first training a 4x4 encoder and decoder, then training at an 8x8 resolution, and so on, gradually reaching the maximum resolution. During training, mean squared error is used as the loss function to guide the parameter updates of the encoder and decoder.

[0073] Furthermore, based on the above embodiments of the invention, training the data processing model structure according to the training set to generate the encoder and decoder of the data processing model includes:

[0074] The encoder and decoder are trained using the training set of the medical test data type; cross-entropy is used as a loss function to control the parameter updates of the encoder and decoder; wherein the encoder and decoder are fully connected layer networks, and the number of hidden layers and the number of hidden layer dimensions of the fully connected layer network are determined by the dimensions of the medical test data.

[0075] In this embodiment of the invention, when the data source is medical data, the encoder and decoder can be fully connected layer networks. The fully connected layer network can include multiple hidden layers, and each hidden layer can include multiple dimensions. When training the fully connected layer network corresponding to the encoder and decoder, the number of hidden layers and their dimensions can be determined by the number of dimensions of the medical data. The fully connected layer network can be trained multiple times. After each training, the cross-entropy damage function can be used to measure the training effect of the fully connected layer network each time. When the value of the cross-entropy damage function meets the training termination condition, the encoder and decoder training is completed.

[0076] In one exemplary implementation, Figure 4 This is a training diagram of an encoder and decoder provided in Embodiment 2 of the present invention. See also... Figure 4 The medical testing data is one-hot data, where each dimension represents the negative or positive result of a test indicator. Correlation may exist between indicators. A self-reading hot data autoencoder can be constructed using a fully connected layer network model. This autoencoder can perform manifold learning and includes an encoder and a decoder. The number of hidden layers in the fully connected layer and the number of dimensions in each hidden layer are selected based on the input dimension. For one-hot data, cross-entropy is used as the loss function to guide the encoder and decoder in updating their parameters. Training stops when the loss function reaches a certain value.

[0077] Furthermore, based on the above embodiments of the invention, training the data processing model structure according to the training set to generate the encoder and decoder of the data processing model includes:

[0078] For the training set of tabular data type, numerical data and categorical data are extracted respectively; Gaussian fitting normalization is performed on the numerical data, and entity embedding encoding is performed on the categorical data; the category vector generated by entity embedding encoding and the Gaussian fitting normalized numerical data are used to train the encoder and the decoder; wherein, the encoder and the decoder are fully connected layer networks, and the number of hidden layers and the number of hidden layer dimensions of the fully connected layer network are determined by the dimensions of the tabular data.

[0079] In this embodiment of the invention, when the data source is tabular data, the encoder and decoder processing the tabular data can use a fully connected layer network structure. Numerical and categorical data in the tabular data need to be pre-processed. Numerical data can be normalized using Gaussian fitting, for example, by normalizing the mean or variance. Categorical data can be vectorized using entity embedding, preserving the relationships between categories as much as possible. The pre-processed categorical and numerical data are then input into the fully connected layer network structure to adjust its parameters. After each training iteration, a loss function can be used to determine if training is complete. It is understood that different loss functions can be used for categorical and numerical data; for example, cross-entropy can be used as the loss function for categorical data, while mean squared error can be used for numerical data.

[0080] In one exemplary implementation, see Figure 5 Since the tabular data includes different data types, it can be divided into numerical data and categorical data. We can first preprocess the different types of data separately: Gaussian fitting normalization is applied to the numerical data, and entity embedding encoding is performed on the categorical data. Then, a fully connected feedforward network is used to construct an autoencoder. For each categorical data type, entity embedding is used to convert discrete category labels into continuous numerical types. For the numerical data, normalization is performed using the mean, variance, etc., to obtain preprocessed data. These two sets of data are then combined as input to the fully connected layer. The number of hidden layers and the dimensions of each hidden layer are set according to the dimensions of the data. During the training of the fully connected layer, different loss functions can be sampled to control the training process. Cross-entropy is used as the loss function for categorical data; mean squared error is used as the loss function for numerical data. Training stops when the value of the loss function meets a threshold.

[0081] In one exemplary implementation, Figure 6 This is an example diagram of a data sharing method provided in Embodiment 2 of the present invention. The data sharing method can be implemented based on a data generation framework, which may include manifold learning models for different types of data. In this framework, the manifold learning model, in conjunction with an optimal transport mapping, learns data patterns (including the manifold structure of the data and the probability density distribution on the manifold) from the original privacy data. Data is then resampled in the data space according to the data patterns to obtain generated data that conforms to the data patterns and is different from the original data. The original data is always retained by the data owner, and the generated data is used for sharing. See also... Figure 6The generation of shared data can include a learning phase and a generation phase. The learning phase includes: 1. Learning the data manifold structure through an autoencoder for different types of data; the output of manifold learning is point cloud data in the low-dimensional space of the manifold expansion; 2. A data reduction module transforms the point cloud data, converting the point representation of probabilities into a weighted representation; 3. Finally, the data enters the optimal transport map solver to obtain a mapping from a specified probability distribution (uniform distribution, Gaussian distribution) to the data probability distribution. The generation phase includes: 1. A specified probability distribution data point sampling module generates a data point set conforming to the specified probability distribution; 2. A data mapping module uses the optimal transport map obtained in the learning phase to map the data point set to the original data probability distribution; 3. A data filtering module removes data in the generated data that is too similar to the original data; 4. Finally, a data decoder learned in the learning phase is used to obtain different types of generated data.

[0082] Example 3

[0083] Figure 7 This is a schematic diagram of a data sharing device provided in Embodiment 3 of the present invention. The data sharing method provided in this embodiment of the present invention can be implemented by software and / or hardware, and can generally be integrated into a server. See [link to documentation]. Figure 7 The device provided in this embodiment of the invention may specifically include: a model structure module 301, a model training module 302, and a shared data module 303.

[0084] Model structure module 301 is used to determine the data processing model structure in the data processing library based on the data type of the data source.

[0085] The model training module 302 is used to generate a data processing model based on the training set of the data source and the data processing model structure.

[0086] The shared data module 303 is used to determine the shared data corresponding to the original data of the data source according to the data processing model, wherein the shared data and the original data have the same popular structure and probability distribution characteristics.

[0087] In this embodiment of the invention, the model structure module selects the data processing model structure corresponding to the data type of the data source in the data processing library. The model training module uses the training set from the data source to train the data processing model structure to generate a data processing model. The shared data module uses the data processing model to process the original data from the data source to generate shared data. The shared data has the same manifold structure and probability distribution characteristics as the original data. This embodiment of the invention achieves privacy processing of the original data through the data processing model, which can reduce the difficulty of data sharing and thus increase the utilization rate of data.

[0088] Furthermore, based on the above embodiments of the invention, the model structure module 301 in the device includes:

[0089] The type reading unit is used to read the data type of the original data in the data source.

[0090] The structure determination unit is used to search in the data processing library for the data processing model structure that matches the data type.

[0091] Furthermore, based on the above embodiments of the invention, the model training module 302 in the device includes:

[0092] The training set generation unit is used to collect raw data from the data source as a training set.

[0093] A model training unit is used to train a data processing model structure based on the training set to generate the encoder and decoder of the data processing model.

[0094] Furthermore, based on the above embodiments of the invention, the model training unit is specifically used to: train the encoder and the decoder layer by layer from low resolution to high resolution using the training set of the image data type; use mean square error as a loss function to control the parameter update of the encoder and the decoder; wherein the encoder and the decoder are graph convolutional neural network models, each including at least a convolutional layer, a linear rectified layer, a pooling layer and a loss function layer.

[0095] Furthermore, based on the above embodiments of the invention, the model training unit is specifically used to: train the encoder and the decoder using the training set of medical test data type; and use cross-entropy as a loss function to control the parameter update of the encoder and the decoder; wherein the encoder and the decoder are fully connected layer networks, and the number of hidden layers and the number of hidden layer dimensions included in the fully connected layer network are determined by the dimension of the medical test data.

[0096] Furthermore, based on the above embodiments of the invention, the model training unit is specifically used to: extract the numerical data and categorical data of the training for the tabular data type training set, respectively;

[0097] The numerical data is normalized using Gaussian fitting, and the categorical data is encoded using entity embedding. The category vectors generated by the entity embedding and the normalized numerical data are used to train the encoder and the decoder. The encoder and the decoder are fully connected layer networks, and the number of hidden layers and the number of hidden layer dimensions of the fully connected layer network are determined by the dimensions of the tabular data.

[0098] Furthermore, based on the above embodiments of the invention, the shared data module 303 includes:

[0099] The data encoding module is used to input the raw data from the data source into the encoder of the data processing model to generate point cloud data in low-dimensional space.

[0100] The data reduction module is used to perform data reduction on the point cloud data.

[0101] The optimal transport mapping solver module is used to determine the mapping relationship between the data distribution probability corresponding to the point cloud data and the specified probability distribution.

[0102] The data point sampling module is used to sample and generate a data point set according to the specified probability distribution.

[0103] The data mapping module is used to map the data point set to the data distribution probability according to the mapping relationship.

[0104] The data filtering module is used to remove data from the mapped data point set that has a similarity threshold greater than that of the point cloud data.

[0105] A data decoding module is used to input the data point set into a decoder to generate the shared data.

[0106] Example 4

[0107] Figure 8 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention, as shown below. Figure 8 As shown, the electronic device includes a processor 40, a memory 41, an input device 42, and an output device 43; the number of processors 40 in the electronic device can be one or more. Figure 8 Taking a processor 40 as an example; the processor 40, memory 41, input device 42, and output device 43 in the electronic device can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0108] The memory 41, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the data sharing method in this embodiment of the invention (e.g., model structure module 301, model training module 302, and shared data module 303 in the data sharing device). The processor 40 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 41, thereby realizing the aforementioned data sharing method.

[0109] The memory 41 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 41 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 41 may further include memory remotely located relative to the processor 40, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0110] Input device 42 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the electronic device. Output device 43 may include display devices such as a display screen.

[0111] Example 5

[0112] Embodiment 5 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a data sharing method, the method comprising:

[0113] The data processing model structure is determined in the data processing library based on the data type of the data source;

[0114] A data processing model is generated based on the training set of the data source and the data processing model structure;

[0115] The shared data corresponding to the original data of the data source is determined according to the data processing model, wherein the shared data and the original data have the same manifold structure and probability distribution characteristics.

[0116] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the method operations described above, but can also perform related operations in the data sharing method provided in any embodiment of the present invention.

[0117] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0118] It is worth noting that in the embodiments of the above data sharing device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0119] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: The data processing model structure is determined in the data processing library according to the data type of the data source, wherein the data type includes at least image data type, medical detection data type and table data type; A data processing model is generated based on the training set from the data source and the data processing model structure: raw data is collected from the data source as a training set; the data processing model structure is trained according to the training set to generate the encoder and decoder of the data processing model, wherein training the data processing model structure according to the training set to generate the encoder and decoder of the data processing model includes: The encoder and decoder are trained layer by layer from low resolution to high resolution using the training set of the image data type; mean square error is used as the loss function to control the parameter update of the encoder and decoder; wherein the encoder and decoder are graph convolutional neural network models, each including at least a convolutional layer, a linear rectified layer, a pooling layer and a loss function layer; The encoder and decoder are trained using the training set of the medical test data type; cross-entropy is used as a loss function to control the parameter updates of the encoder and decoder; wherein the encoder and decoder are fully connected layer networks, and the number of hidden layers and the number of hidden layer dimensions of the fully connected layer network are determined by the dimensions of the medical test data; For the training set of tabular data type, numerical data and categorical data are extracted from the training set respectively; Gaussian fitting normalization is performed on the numerical data, and entity embedding encoding is performed on the categorical data; the category vector generated by entity embedding encoding and the Gaussian fitting normalized numerical data are used to train the encoder and the decoder; wherein, the encoder and the decoder are fully connected layer networks, and the number of hidden layers and the number of hidden layer dimensions of the fully connected layer network are determined by the dimensions of the tabular data; The shared data corresponding to the original data of the data source is determined according to the data processing model, wherein the shared data and the original data have the same manifold structure and probability distribution characteristics.

2. The method according to claim 1, characterized in that, The step of determining the data processing model structure in the data processing library based on the data source data includes: Read the data type of the original data within the data source; Search the data processing model structure that matches the data type in the data processing library.

3. The method according to claim 1, characterized in that, The step of determining the shared data corresponding to the original data of the data source based on the data processing model includes: The raw data from the data source is input into the encoder of the data processing model to generate point cloud data in low-dimensional space; Perform data reduction on the point cloud data; Determine the mapping relationship between the data distribution probability corresponding to the point cloud data and the specified probability distribution; Data point sets are generated by sampling according to the specified probability distribution; Map the set of data points to the data distribution probability according to the mapping relationship; Data points with a similarity threshold greater than the point cloud data are removed from the mapped data point set; The data point set is input into the decoder to generate the shared data.

4. A data processing apparatus, characterized in that, The device includes: The model structure module is used to determine the data processing model structure in the data processing library according to the data type of the data source, wherein the data type includes at least image data type, medical detection data type and table data type; A model training module is used to generate a data processing model based on a training set from the data source and the data processing model structure: collecting raw data from the data source as a training set; training the data processing model structure according to the training set to generate the encoder and decoder of the data processing model, wherein the step of training the data processing model structure according to the training set to generate the encoder and decoder of the data processing model includes: The encoder and decoder are trained layer by layer from low resolution to high resolution using the training set of the image data type; mean square error is used as the loss function to control the parameter update of the encoder and decoder; wherein the encoder and decoder are graph convolutional neural network models, each including at least a convolutional layer, a linear rectified layer, a pooling layer and a loss function layer; The encoder and decoder are trained using the training set of the medical test data type; cross-entropy is used as a loss function to control the parameter updates of the encoder and decoder; wherein the encoder and decoder are fully connected layer networks, and the number of hidden layers and the number of hidden layer dimensions of the fully connected layer network are determined by the dimensions of the medical test data; For the training set of tabular data type, numerical data and categorical data are extracted respectively; Gaussian fitting normalization is performed on the numerical data, and entity embedding encoding is performed on the categorical data; the category vector generated by entity embedding encoding and the Gaussian fitting normalized numerical data are used to train the encoder and the decoder; wherein, the encoder and the decoder are fully connected layer networks, and the number of hidden layers and the number of hidden layer dimensions of the fully connected layer network are determined by the dimensions of the tabular data; A shared data module is used to determine the shared data corresponding to the original data of the data source according to the data processing model, wherein the shared data and the original data have the same manifold structure and probability distribution characteristics.

5. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it is a data processing method as described in any one of claims 1-3.

Images