Simulation service platform data similarity evaluation method and device and electronic equipment

Abstract

The invention relates to a simulation service platform data similarity evaluation method and device and electronic equipment, and relates to the technical field of simulation data processing. The method comprises the following steps: acquiring real data, generating synthetic data through a generative adversarial network based on the real data, calculating a compression rate from the real data to the synthetic data and a decompression rate from the synthetic data to the real data, determining a compression distance between the real data and the synthetic data according to the compression rate and the decompression rate, and evaluating the similarity between the real data and the synthetic data based on the compression distance. According to the scheme provided by the invention, the synthetic data highly similar to real data distribution can be generated through the generative adversarial network, powerful support is provided for data enhancement and simulation experiments by combining the neural network model, and similarity is evaluated by calculating the compression distance value between the real data and the synthetic data. And an effective means is provided for simulation data analysis and processing.

Description

Technical Field

[0001] The present application relates to the technical field of simulation data processing, and in particular to a method, device, and electronic device for evaluating data similarity of a simulation service platform. Background Art

[0002] In the data-driven era, data generation, discrimination and similarity assessment technologies of industrial design and virtual simulation service platforms are of vital importance to improving product design quality, optimizing simulation experiment results and accelerating product development cycles.

[0003] With the continuous advancement of technology, industrial design and virtual simulation service platforms have made significant progress in data generation, discrimination and similarity assessment, and model-based data generation methods have gradually emerged. These methods capture the distribution characteristics in the data by training specific machine learning models, thereby generating synthetic data that is closer to the real data distribution. However, the accuracy of these methods is still limited by the complexity of the model and the diversity of training data, and they still face problems such as low data generation quality, insufficient accuracy of the discrimination model, and lack of intuitiveness of the similarity assessment method. Summary of the invention

[0004] In order to solve or partially solve the problems existing in the related technology, the present application provides a simulation service platform data similarity evaluation method, device, and electronic device, which can generate synthetic data that is highly similar to the real data distribution through a generative adversarial network, provide strong support for data enhancement and simulation experiments by combining a neural network model, and evaluate the similarity by calculating the compressed distance value between the real data and the synthetic data, providing an effective means for data analysis and processing.

[0005] The first aspect of the present application provides a simulation service platform data similarity evaluation method, comprising: Acquire real data, and generate synthetic data based on the real data through a generative adversarial network; calculating a compression rate from the real data to the synthetic data, and a decompression rate from the synthetic data to the real data; determining a compression distance between the real data and the synthetic data according to the compression rate and the decompression rate; The similarity between the real data and the synthetic data is evaluated based on the compression distance.

[0006] Preferably, generating synthetic data based on the real data by generating an adversarial network comprises: generating noise according to data characteristics of the real data; The synthetic data is generated through the generative adversarial network based on the noise and the real data.

[0007] Preferably, the calculating of the compression rate from the real data to the synthetic data and the decompression rate from the synthetic data to the real data comprises: Calculating the variance of the noise, and calculating the compression rate according to the variance; Subtracting the noise from the synthetic data to generate simulated real data; The decompression rate is calculated based on the simulated real data and the synthetic data.

[0008] Preferably, the calculating of the compression rate from the real data to the synthetic data and the decompression rate from the synthetic data to the real data further comprises: Calculating feature matrices of the real data and the synthetic data respectively; Calculate eigenvalues ​​and eigenvectors based on their respective characteristic matrices; Determining the number of principal components to be retained for the real data and the synthetic data according to their respective eigenvalues; The real data and the synthetic data are respectively input into a preset compression rate function in combination with the number of principal components to generate a compression rate from the real data to the synthetic data and a decompression rate from the synthetic data to the real data.

[0009] Preferably, the step of inputting the real data matrix and the synthetic data matrix into a preset compression rate function in combination with the number of principal components to respectively generate a compression rate from the real data to the synthetic data and a decompression rate from the synthetic data to the real data comprises: Determine the number of eigenvalues ​​of the principal components and the target eigenvectors corresponding to the number of eigenvalues ​​of the principal components from the eigenvalues, and determine the target eigenvectors as the principal components; Projecting the real data matrix and the synthetic data matrix onto their respective principal components, respectively, to generate reduced-dimensional real data and reduced-dimensional synthetic data; Calculate the compression rate of the real data and the compression rate of the synthetic data respectively through the preset compression rate function according to the number of the respective eigenvalues ​​and the number of the principal components; The compression rate of the synthetic data is determined as the compression rate, and the compression rate of the real data is determined as the decompression rate.

[0010] Preferably, determining the compression distance between the real data and the synthetic data according to the compression rate and the decompression rate comprises: comparing the compression rate and the decompression rate; The larger one of the compression rate and the decompression rate is determined as the compression distance.

[0011] Preferably, the generative adversarial network includes a generative network and an adversarial network, both of which are multi-layer fully connected neural networks, and the generative adversarial network is obtained by the following method: Inputting the first training sample into the initial generative adversarial network; The initial generation network generates a first synthetic sample of the first training sample; Using the first training sample and the first synthetic sample to train the initial generative adversarial network to obtain a preliminary generative adversarial network; Inputting a second training sample into the preliminary generative adversarial network; The preliminary generation network generates a second synthetic sample of the second training sample; The second training sample and the second synthetic sample are used to train a preliminary discriminant network to generate a trained generative adversarial network.

[0012] Preferably, the adopting the first training sample and the first synthetic sample to train the initial generative adversarial network to obtain a preliminary generative adversarial network comprises: Inputting the first training sample and the first synthetic sample into an initial adversarial network respectively, calculating a first loss using a cross entropy loss function, and adjusting the initial generative adversarial network according to the first loss until a first preset condition is met, thereby obtaining the preliminary generative adversarial network; The using the second training sample and the second synthetic sample to train the preliminary discriminant network to generate a trained generative adversarial network comprises: adding labels to the second training sample and the second synthetic sample respectively to construct a training data set pair; The preliminary discriminant network is trained using the training data set until the preliminary discriminant network meets a second preset condition, thereby generating the generative adversarial network.

[0013] A second aspect of the present application provides a simulation service platform data similarity evaluation device, comprising: An acquisition module, used to acquire real data and generate synthetic data based on the real data through a generative adversarial network; A calculation module, used for calculating a compression rate from the real data to the synthetic data, and a decompression rate from the synthetic data to the real data; a determination module, configured to determine a compression distance between the real data and the synthetic data according to the compression rate and the decompression rate; An evaluation module is used to evaluate the similarity between the real data and the synthetic data based on the compression distance.

[0014] A third aspect of the present application provides an electronic device, including: Processor; and The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is caused to execute the method as described above.

[0015] The technical solution provided by the present application may include the following beneficial effects: An embodiment of the present application discloses a method for evaluating data similarity of a simulation service platform, including obtaining real data, and generating synthetic data based on the real data through a generative adversarial network, calculating the compression rate from the real data to the synthetic data, and the decompression rate from the synthetic data to the real data, determining the compression distance between the real data and the synthetic data according to the compression rate and the decompression rate, evaluating the similarity between the real data and the synthetic data based on the compression distance, being able to generate synthetic data that is highly similar to the real data distribution through a generative adversarial network, providing strong support for data enhancement and simulation experiments by combining a neural network model, evaluating the similarity by calculating the compression distance value between the real data and the synthetic data, and providing an effective means for simulation data analysis and processing.

[0016] The technical solution of the present application can also: directly reflect the degree of information loss of data during the compression and decompression process through the compression distance, making the evaluation results more intuitive and easy to understand, and can be applied to any one-dimensional or multi-dimensional continuous data set, and is not limited to a specific compression algorithm or denoising technology. It has broad application prospects in the fields of data compression effect evaluation, data denoising, signal recovery, etc., and can provide users with valuable information support.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail the exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0019] Figure 1 It is a flowchart of a method for evaluating data similarity of a simulation service platform shown in an embodiment of the present application; Figure 2 It is another flow chart of the method for evaluating data similarity of a simulation service platform shown in an embodiment of the present application; Figure 3 It is another flow chart of the method for evaluating data similarity of a simulation service platform shown in an embodiment of the present application; Figure 4It is a flowchart of a method for generating a generative adversarial network according to an embodiment of the present application; Figure 5 It is a structural diagram of a simulation service platform data similarity evaluation device shown in an embodiment of the present application; Figure 6 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0021] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0022] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0023] With the continuous advancement of technology, industrial design and virtual simulation service platforms have made significant progress in data generation, discrimination and similarity assessment, and model-based data generation methods have gradually emerged. These methods capture the distribution characteristics in the data by training specific machine learning models, thereby generating synthetic data that is closer to the real data distribution. However, the accuracy of these methods is still limited by the complexity of the model and the diversity of training data, and they still face problems such as low data generation quality, insufficient accuracy of the discrimination model, and lack of intuitiveness of the similarity assessment method.

[0024] In response to the above problems, an embodiment of the present application provides a method for evaluating data similarity of a simulation service platform, which can generate synthetic data that is highly similar to the real data distribution through a generative adversarial network, provide strong support for data enhancement and simulation experiments, and evaluate the similarity by calculating the compressed distance value between the real data and the synthetic data, thereby providing an effective means for data analysis and processing.

[0025] The technical solution of the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0026] Figure 1 It is a flow chart of a simulation service platform data similarity evaluation method shown in an embodiment of the present application.

[0027] See also Figure 1 , the method comprising: Step 101, obtain real data, and generate synthetic data based on the real data through a generative adversarial network.

[0028] Generative Adversarial Networks (GAN) consists of a generator network and a discriminator network. The generator network is responsible for capturing the distribution of sample data, while the discriminator network is generally a binary classifier that determines whether the input is real data or a generated sample.

[0029] In the embodiment of the present application, the real data X can be a vector containing n m dimensions, representing the observed values ​​of n samples on m features, or an original continuous data set. The real data X can be any form of data, such as images, text, or time series data. The generative adversarial network can generate synthetic data Y that is as close to the real data as possible based on the real data X.

[0030] Step 102, calculating the compression rate from real data to synthetic data, and the decompression rate from synthetic data to real data.

[0031] After the synthetic data Y is generated, the compression rate from the real data X to the synthetic data Y and the decompression rate from the synthetic data Y to the real data X are calculated respectively.

[0032] Step 103: Determine the compression distance between the real data and the synthetic data according to the compression rate and the decompression rate.

[0033] The compression distance between the real data X and the synthetic data Y can be determined according to the compression rate and the decompression rate. The compression rate from the real data X to the synthetic data Y, the decompression rate from the synthetic data Y to the real data X, and the compression distance between the real data X and the synthetic data Y can also be output for user reference and analysis.

[0034] Step 104 , evaluating the similarity between the real data and the synthetic data based on the compression distance.

[0035] The similarity between the real data X and the synthetic data Y is evaluated according to the compression distance. The larger the compression distance, the greater the difference in information loss between the real data X and the synthetic data Y during the compression process, which reflects the lower the similarity or correlation between them.

[0036] The embodiment of the present application discloses a method for evaluating data similarity of a simulation service platform, including obtaining real data, and generating synthetic data based on the real data through a generative adversarial network, calculating the compression rate from the real data to the synthetic data, and the decompression rate from the synthetic data to the real data, determining the compression distance between the real data and the synthetic data according to the compression rate and the decompression rate, and evaluating the similarity between the real data and the synthetic data based on the compression distance. The method can generate synthetic data that is highly similar to the real data distribution through a generative adversarial network, provide strong support for data enhancement and simulation experiments by combining a neural network model, evaluate the similarity by calculating the compression distance value between the real data and the synthetic data, and provide an effective means for simulation data analysis and processing.

[0037] Figure 2 It is another flow chart of the simulation service platform data similarity evaluation method shown in the embodiment of the present application.

[0038] See also Figure 2 , the method comprising: Step 201 , obtaining real data, and generating noise according to data characteristics of the real data.

[0039] The embodiments of the present application indirectly reflect the similarities or differences between data by simulating the compression and decompression process of real data affected by noise, thereby providing strong support for tasks such as data compression effect evaluation, data denoising and signal recovery.

[0040] The real data X can be a raw continuous data set, which can be any one-dimensional or multi-dimensional continuous data point set such as time series, image pixel data, sensor readings, etc. According to the application scenario and the data characteristics of the real data, a reasonable noise variance Y_noise_variance is set, and a random number generator (such as a Gaussian noise generator) is used to generate noise N of the same size as the real data X (that is, the same number of data points, and each data point has the same dimension) according to the given noise variance Y_noise_variance.

[0041] Step 202, generating synthetic data through a generative adversarial network based on noise and real data.

[0042] Using the generated noise N, the generative adversarial network adds the generated noise N to each corresponding data point of the real data X to obtain synthetic data Y, that is, synthetic data Y = X + N. This step simulates the noise pollution that the data may suffer during actual transmission, storage or processing.

[0043] Step 203: Calculate the variance of the noise, and calculate the compression rate according to the variance.

[0044] Calculate the variance of the noise N in the synthetic data Y, denoted as Var(N), and take the logarithm (base 10) of the variance of the noise N Var(N) as an approximation of the compression rate R_X_given_Y from the real data X to the synthetic data Y, that is, The compression rate can be regarded as a quantitative indicator of the degree of information loss. The larger the noise variance, the more information loss and the higher the compression rate.

[0045] Step 204, subtracting noise from the synthetic data to generate simulated real data.

[0046] Since the added noise N is known, the noise N can be directly subtracted from the synthetic data Y to generate the simulated real data X_hat, that is, X_hat = Y - N. This step simulates the complete decompression process in the ideal case, that is, completely removing the noise and restoring the original data.

[0047] Step 205, calculating the decompression rate according to the simulated real data and the synthetic data.

[0048] The decompression rate R_Y_given_X can be calculated based on the simulated real data X_hat and the synthetic data Y. In the embodiment of the present application, in order to simplify the calculation process, the decompression rate R_Y_given_X is directly set to be the same as the compression rate R_X_given_Y, that is, R_Y_given_X = R_X_given_Y. However, in practical applications, if the efficiency of the decompression algorithm or the quality of data recovery has a significant impact on the result, a more complex model can be introduced to calculate the decompression rate. The embodiment of the present application does not limit the calculation method of the decompression rate.

[0049] Step 206: Compare the compression rate and the decompression rate.

[0050] After obtaining the decompression rate R_Y_given_X, the compression rate R_X_given_Y and the decompression rate R_Y_given_X can be compared.

[0051] Step 207: The larger of the compression rate and the decompression rate is determined as the compression distance.

[0052] The larger value of the compression rate R_X_given_Y and the decompression rate R_Y_given_X is used as the overlapping compression distance CD_lsy between continuous data, that is, CD_lsy = max(R_X_given_Y, R_Y_given_X).

[0053] Step 208 , evaluating the similarity between the real data and the synthetic data based on the compression distance.

[0054] For ease of understanding, the compression distance CD_lsy can be converted into a more understandable percentage or ratio form. When evaluating the similarity, a preset threshold can be set for the compression distance CD_lsy, and the compression distance CD_lsy can be compared with the preset threshold to evaluate whether the similarity or difference between the real data and the synthetic data meets the predetermined standard. In an example, the calculated compression distance CD_lsy can be output to the user and can be presented in the form of a graphical user interface (GUI), a command-line interface (CLI) or a data report. The user can judge the degree of similarity or difference between the real data X and the synthetic data Y based on the value of the compression distance CD_lsy, and then make corresponding data analysis or decision support.

[0055] In an optional embodiment of the present application, if the real data X and the synthetic data Y have complete information overlap, for this special case, a random process M is introduced to simulate the complete overlap between the data, and based on the process, the calculation of the bidirectional compression rate between the real data X and the synthetic data Y is realized, thereby obtaining an accurate compression distance. The random process M can be Gaussian white noise, Brownian motion, or any other independent random process that does not depend on the real data X, ensuring that the generation of M can cover all possible ranges of variation of X while maintaining its independence. The specific process is as follows: First, a complete overlap model under continuous data is constructed, and an independent random process M is superimposed on the data set X (real data X), and synthetic data Y (Y=X+M) is generated through a generative adversarial network. The parameters of M, such as mean, variance, or correlation function, can be adjusted according to actual needs and application scenarios. The synthetic data Y (i.e., X+ M) is compressed using a preset compressor to obtain a compressed data representation S_(X+ M), and the preset compressor can effectively process and compress data containing random noise. The length or number of bits of S_(X+ M) is defined as the compression distance R_(Y|X)(D) required for successful decompression from real data X to synthetic data Y. This process simulates the information transmission and compression process from real data X to synthetic number Y. The synthetic data Y is compressed using a preset compressor to obtain compressed data S_Y, and the synthetic data Y is decompressed to obtain the estimated value Ŷ of the synthetic data Y. The real data X (or approximate X) is recovered from the estimated value Ŷ through filtering, denoising algorithms or model-based methods. The length or number of bits of S_Y is defined as the compression distance R_(X|Y)(D) from synthetic data Y to real data X. R_(Y|X)(D) and R_(X|Y)(D) are compared, and the larger of the two is determined as the compression distance CD_lsy(X,Y). The compression distance is output as an indicator for evaluating the similarity or compression characteristics between real data X and synthetic data Y, which can be used in multiple fields such as performance evaluation of data compression algorithms, data de-redundancy, and feature extraction.

[0056] The embodiment of the present application discloses a method for evaluating data similarity of a simulation service platform, including obtaining real data, generating noise according to data characteristics of the real data, generating synthetic data through a generative adversarial network based on the noise and the real data, calculating the variance of the noise, calculating the compression rate according to the variance, subtracting the noise from the synthetic data, generating simulated real data, calculating the decompression rate according to the simulated real data and the synthetic data, comparing the compression rate and the decompression rate, determining the larger of the compression rate and the decompression rate as the compression distance, evaluating the similarity between the real data and the synthetic data based on the compression distance, being able to directly reflect the degree of information loss of the data during the compression and decompression process through the compression distance, making the evaluation result more intuitive and easy to understand, being able to be applied to any one-dimensional or multi-dimensional continuous data set, and being not limited to a specific compression algorithm or denoising technology, having broad application prospects in the fields of data compression effect evaluation, data denoising, signal recovery, etc., and being able to provide users with valuable information support.

[0057] Figure 3 It is another flow chart of the simulation service platform data similarity evaluation method shown in the embodiment of the present application.

[0058] See also Figure 3 , the method comprising: Step 301, obtain real data, and generate synthetic data based on the real data through a generative adversarial network.

[0059] The embodiment of the present application adopts the compression rate simulation method of principal component analysis (PCA), and uses this method to calculate the compression rate to evaluate the similarity or correlation between two data sets. By simulating the data compression process, the amount of information loss before and after the data is compressed is calculated, thereby indirectly reflecting the dependence and similarity between the data. PCA is a commonly used data dimensionality reduction technology. It maps the original data to a new coordinate system through linear transformation, so that the variance of the transformed data on the first coordinate (i.e., the first principal component) is the largest, the variance on the second coordinate is the second largest, and so on. In the embodiment of the present application, PCA is used to simulate the data compression process.

[0060] In the embodiment of the present application, it is assumed that the real data X is stored in a matrix form, where each real data X contains n m-dimensional vectors, representing the observed values ​​of n samples on m features. The generative adversarial network can generate synthetic data Y that is as close to the real data as possible based on the real data X. The synthetic data Y also contains n m-dimensional vectors. If the real data X and the synthetic data Y are not in matrix form, they can be converted into matrix form.

[0061] Step 302, respectively calculate the feature matrices of the real data and the synthetic data.

[0062] The feature matrices of the real data and the synthetic data are calculated respectively. The feature matrices can be covariance matrices or correlation matrices.

[0063] Step 303: Calculate eigenvalues ​​and eigenvectors based on their respective characteristic matrices.

[0064] Based on their respective feature matrices, the eigenvalues ​​and eigenvectors are calculated. The eigenvectors correspond to the main direction of change of the data, i.e., the principal components. The number of eigenvalues ​​m is recorded at the same time.

[0065] Step 304, determining the number of principal components to be retained for the real data and the synthetic data according to their respective eigenvalues.

[0066] According to the size of the eigenvalue, determine the number of principal components numComponents to be retained for the real data X and the synthetic data Y, and select the eigenvectors corresponding to the first numComponents largest eigenvalues ​​as the principal component.

[0067] Step 305 , input the real data and the synthetic data into a preset compression rate function in combination with the number of principal components, and generate a compression rate from the real data to the synthetic data and a decompression rate from the synthetic data to the real data, respectively.

[0068] Define a preset compression rate function simulatedCompressionRate(data, numComponents), which receives the real data X, the synthetic data Y, and the number of principal components numComponents to be retained as input. In one example, if the PCA transformation has not been performed, the PCA transformation is first performed on the real data X or the synthetic data Y inside the preset compression rate function, and then the compression rate from the real data X to the synthetic data Y and the decompression rate from the synthetic data Y to the real data X are generated respectively.

[0069] In an optional embodiment of the present application, step 305 includes: Sub-step 3051, determining the number of eigenvalues ​​of the principal components and the target eigenvectors corresponding to the number of eigenvalues ​​of the principal components from the eigenvalues, and determining the target eigenvectors as the principal components.

[0070] According to the size of the eigenvalue, determine the number of principal components numComponents to be retained for the real data X and the synthetic data Y, select the eigenvectors corresponding to the first numComponents largest eigenvalues ​​as the target eigenvector, and determine the target eigenvector as the principal component. Then retain the first numComponents principal components, which will be used to construct the new coordinate system.

[0071] Sub-step 3052, projecting the real data and the synthetic data onto their respective principal components, respectively, to generate the real data after dimensionality reduction and the synthetic data after dimensionality reduction, respectively.

[0072] The real data X and the synthetic data Y will be projected along the principal components. The real data X and the synthetic data Y will be projected onto their respective principal components to obtain the reduced dimension data sets X_reduced and Y_reduced.

[0073] Sub-step 3053, respectively calculating the compression ratio of the real data and the compression ratio of the synthetic data by using a preset compression ratio function according to the number of their respective eigenvalues ​​and the number of principal components.

[0074] The compression ratio L(X) of real data and the compression ratio L(Y) of synthetic data can be calculated by log2(the number of eigenvalues ​​m / the number of principal components retained numComponents).

[0075] Sub-step 3054, determining the compression ratio of the synthetic data as the compression rate, and determining the compression ratio of the real data as the decompression rate.

[0076] In order to evaluate the similarity or correlation between the real data X and the synthetic data Y, it is assumed that the compression rate L(Y|X) from the real data X to the synthetic data Y can be approximated by projecting Y onto the principal component of X and calculating the compression rate. Similarly, the compression rate L(X|Y) from the synthetic data Y to the real data X can also be approximated by projecting X onto the principal component of Y and calculating the compression rate. In one example, in order to improve the calculation speed, a simplified approximation method can be adopted, that is, the principal components of the real data X and the synthetic data Y are used to calculate the compression rate and the decompression rate, that is, the compression rate from the real data X to the synthetic data Y is L(Y|X) =L(Y), and the decompression rate from the synthetic data Y to the real data X is L(X|Y) =L(X).

[0077] Step 306: compare the compression rate and the decompression rate.

[0078] The compression rate L(Y) and the decompression rate L(X) are compared to determine the larger of the two.

[0079] Step 307: The larger of the compression rate and the decompression rate is determined as the compression distance.

[0080] The larger value of L(Y) and L(X) is taken as the compression distance CD(X,Y) between the real data X and the synthetic data Y. This choice is based on the assumption that if the difference in information loss during compression between two data sets is large, then the similarity or correlation between them may be low. The calculated compression rate from real data X to synthetic data Y (approximately L(Y)), the decompression rate from synthetic data Y to real data X (approximately L(X)), and the compression distance CD(X,Y) between real data X and synthetic data Y can also be displayed to the user in the form of a table or graph.

[0081] Step 308 , evaluating the similarity between the real data and the synthetic data based on the compression distance.

[0082] According to the value of the compression distance CD(X,Y), the similarity or correlation between the real data X and the synthetic data Y can be evaluated. For example, if CD(X,Y) is small, it means that the difference in information loss between the real data X and the synthetic data Y during the compression process is not large, and they may have a high similarity or correlation; conversely, if CD(X,Y) is large, it means that the similarity or correlation between the real data X and the synthetic data Y may be low.

[0083] In an optional embodiment of the present application, if the real data X and the synthetic data Y have complete information overlap, for this special case, a random process M is introduced to simulate the complete overlap between the data, and based on the process, the calculation of the bidirectional compression rate between the real data X and the synthetic data Y is realized, thereby obtaining an accurate compression distance. The random process M can be Gaussian white noise, Brownian motion, or any other independent random process that does not depend on the real data X, ensuring that the generation of M can cover all possible ranges of variation of X while maintaining its independence. The specific process is as follows: First, a complete overlap model under continuous data is constructed, and an independent random process M is superimposed on the data set X (real data X), and synthetic data Y (Y=X+M) is generated through a generative adversarial network. The parameters of M, such as mean, variance, or correlation function, can be adjusted according to actual needs and application scenarios. The synthetic data Y (i.e., X+ M) is compressed using a preset compressor to obtain a compressed data representation S_(X+ M), and the preset compressor can effectively process and compress data containing random noise. The length or number of bits of S_(X+ M) is defined as the compression distance R_(Y|X)(D) required for successful decompression from real data X to synthetic data Y. This process simulates the information transfer and compression process from real data X to synthetic number Y. The synthetic data Y is compressed using a preset compressor to obtain compressed data S_Y, and the synthetic data Y is decompressed to obtain the estimated value Ŷ of the synthetic data Y. The real data X (or approximate X) is recovered from the estimated value Ŷ through filtering, denoising algorithms or model-based methods. The length or number of bits of S_Y is defined as the compression distance R_(X|Y)(D) from Y to X. R_(Y|X)(D) and R_(X|Y)(D) are compared, and the larger of the two is determined as the compression distance CD_lsy(X,Y). The compression distance is output as an indicator for evaluating the similarity or compression characteristics between real data X and synthetic data Y, and can be used in multiple fields such as performance evaluation of data compression algorithms, data de-redundancy, and feature extraction.

[0084] The embodiment of the present application discloses a method for evaluating data similarity of a simulation service platform, including obtaining real data, and generating synthetic data based on the real data through a generative adversarial network, respectively calculating feature matrices of the real data and the synthetic data, respectively calculating eigenvalues ​​and eigenvectors based on their respective feature matrices, determining the number of principal components to be retained for the real data and the synthetic data according to their respective eigenvalues, inputting the real data and the synthetic data into a preset compression rate function in combination with the number of principal components, respectively generating a compression rate from the real data to the synthetic data and a decompression rate from the synthetic data to the real data, respectively generating the compression rate and the decompression rate, comparing the compression rate and the decompression rate, and determining the larger of the compression rate and the decompression rate as the compression distance, so that a user can adjust the number of principal components retained according to actual needs to balance the compression effect and the degree of information retention. The method can be applied to various types of data sets, including but not limited to images, texts, time series, etc., has a wide range of applicability, can play an important role in data compression, data de-redundancy, feature extraction, anomaly detection and other fields, and provide strong support for research and application in the fields of data science and machine learning.

[0085] Figure 4 It is a flow chart of a method for generating a generative adversarial network shown in an embodiment of the present application.

[0086] See also Figure 4 The generative adversarial network includes a generative network and an adversarial network, both of which are multi-layer fully connected neural networks. The method includes: Step 401: input a first training sample into an initial generative adversarial network.

[0087] The generative adversarial network includes a generator network (G) and an adversarial network (D). The generator network is responsible for receiving random noise or real data as input, and generating synthetic data that is as close to the real data distribution as possible; the discriminator network is responsible for distinguishing whether the input data is real or generated by the generator network. In an embodiment of the present application, both the generator network and the discriminator network may include multiple fully connected layers, ReLU (Rectified Linear Unit, linear rectifier function) activation layers, and specific output layers (such as regression layers or sigmoid layers). The generator network is a multi-layer fully connected neural network, and the input layer receives a random noise vector of a fixed length (for example, a vector of length 100), followed by several hidden layers, each hidden layer contains a certain number of neurons (such as 256 neurons per layer), and uses ReLU as the activation function. The number of neurons in the output layer matches the dimension of the input data. For example, for image data, the number of neurons in the output layer should be equal to the number of pixels in the image. The discriminant network is also a multi-layer fully connected neural network. The input layer receives data of the same dimension as the real data (including the data output by the generated network and the real data), followed by several hidden layers (each layer contains 512 neurons, activated by ReLU), and finally an output layer containing one neuron, which uses the sigmoid function to output the judgment result (0 for synthetic data and 1 for real data). Before training, the weights and biases of the generated network and the discriminant network can be randomly initialized, and then the first training sample is input into the initial generated network of the initial generated adversarial network.

[0088] Step 402: The initial generation network generates a first synthetic sample of a first training sample.

[0089] The initial generation network generates a corresponding first synthetic sample according to the first training sample and random noise.

[0090] Step 403: Input the first training sample and the first synthetic sample into the initial adversarial network respectively, calculate the first loss using the cross entropy loss function, adjust the initial generative adversarial network according to the first loss, until the first preset condition is met, and obtain the preliminary generative adversarial network.

[0091] The first training sample and the first synthetic sample are respectively input into the initial adversarial network. The initial adversarial network calculates the first loss according to the cross entropy loss function, and uses the back propagation algorithm and an appropriate optimizer (such as the Adam optimizer) to update the parameters of the initial generative network and the initial discriminant network until the first preset condition is met to obtain a preliminary generative adversarial network. The first preset condition may be reaching a preset maximum number of iterations or satisfying other stop conditions. The embodiment of the present application does not limit the first preset condition.

[0092] Step 404: input the second training sample into the preliminary generative adversarial network.

[0093] After obtaining the preliminary generative adversarial network, further training can be performed by inputting the second training sample into the preliminary generative adversarial network.

[0094] Step 405: The network is initially generated to generate a second synthetic sample of the second training sample.

[0095] The preliminary generative network of the preliminary generative adversarial network generates a corresponding second synthetic sample according to the second training sample.

[0096] Step 406: Add labels to the second training sample and the second synthetic sample respectively to construct a training data set pair.

[0097] Labels are added to the second training sample and the second synthetic sample respectively, for example, label "1" is added to the second training sample and label "0" is added to the second synthetic sample. For any second training sample, a training data pair (X_train, Y_train) is constructed according to the second training sample and the second synthetic sample corresponding to the second training sample. Each second training sample is traversed to generate a training data set pair.

[0098] Step 407: Use the training data set to train the preliminary discriminant network until the preliminary discriminant network meets the second preset condition, thereby generating a generative adversarial network.

[0099] A preliminary discriminant network is trained using a training data set to improve its ability to distinguish between real data and synthetic data. The training process is repeated until the preliminary discriminant network meets the second preset condition to generate a final generative adversarial network. The second preset condition may be that the performance of the discriminant network is stable, a preset training goal is achieved, the similarity between the synthetic data and the real data reaches a preset condition, etc. The embodiment of the present application does not limit the second preset condition.

[0100] In an optional embodiment of the present application, the method further includes: Step S11, obtaining real data, and generating synthetic data based on the real data through a generative adversarial network.

[0101] After generating the final generative adversarial network, we can obtain real data X and input the real data X into the generative adversarial network to generate synthetic data Y.

[0102] Step S12, calculating the compression distance between the real data and the synthetic data by using a preset compression distance function.

[0103] Select a suitable preset compression distance function, such as Kolmogorov complexity or approximate compression distance based on a specific compression algorithm (such as compression rate difference based on ZIP or LZ77 algorithms), compress the real data and synthetic data respectively using the preset compression distance function, and record their compressed sizes. Calculate the difference (or ratio) between the compressed sizes of the two sets of data as the compression distance between them.

[0104] Step S13, evaluating the similarity between the real data and the synthetic data based on the compression distance.

[0105] The calculated compression distance value is output as the evaluation result of the similarity between the synthetic data Y and the real data X. According to actual needs, a threshold can be set to determine whether the quality of the synthetic data meets the requirements. It is understandable that if the synthetic data does not meet the requirements, the generative adversarial network can continue to be trained, or the parameters of the generative adversarial network can be changed until the quality of the synthetic data meets the requirements.

[0106] The embodiment of the present application discloses a method for generating a generative adversarial network, wherein the generative adversarial network includes a generative network and an adversarial network, both of which are multi-layer fully connected neural networks. The method includes: inputting a first training sample into an initial generative adversarial network, wherein the initial generative network generates a first synthetic sample of the first training sample, inputting the first training sample and the first synthetic sample into the initial adversarial network respectively, calculating a first loss using a cross entropy loss function, adjusting the initial generative adversarial network according to the first loss until a first preset condition is met to obtain a preliminary generative adversarial network, inputting a second training sample into the preliminary generative adversarial network, and wherein the preliminary generative network generates a second synthetic sample of the second training sample. , add labels to the second training sample and the second synthetic sample respectively, construct a training data set pair, use the training data set pair to train the preliminary discriminant network until the preliminary discriminant network meets the second preset condition, and generate a generative adversarial network. It can generate synthetic data that is highly similar to the real data distribution based on the trained generative adversarial network, and provide strong support for data enhancement and simulation experiments. Using real data and synthetic data to train the discriminant network can significantly improve the discriminant model's ability to distinguish between real data and synthetic data. This method can be flexibly applied to data generation and discrimination tasks in different fields, and the network structure and training parameters can be adjusted according to actual needs to adapt to different application scenarios.

[0107] Corresponding to the aforementioned application function implementation method embodiment, the present application also provides a simulation service platform data similarity evaluation device, electronic device and corresponding embodiments.

[0108] Figure 5 It is a structural diagram of a simulation service platform data similarity evaluation device shown in an embodiment of the present application.

[0109] See also Figure 5 , the data similarity evaluation device 500 comprises: An acquisition module 501 is used to acquire real data and generate synthetic data based on the real data through a generative adversarial network; A calculation module 502, for calculating a compression rate from real data to synthetic data, and a decompression rate from synthetic data to real data; A determination module 503, configured to determine a compression distance between real data and synthetic data according to a compression rate and a decompression rate; The evaluation module 504 is used to evaluate the similarity between the real data and the synthetic data based on the compression distance.

[0110] In an optional embodiment of the present application, the acquisition module 501 includes: A noise submodule, used to generate noise according to the data characteristics of real data; The synthetic data submodule is used to generate synthetic data based on noisy and real data through a generative adversarial network.

[0111] In an optional embodiment of the present application, the calculation module 502 includes: The variance submodule is used to calculate the variance of the noise and calculate the compression rate based on the variance; A simulation submodule, used to subtract noise from synthetic data to generate simulated real data; The decompression submodule is used to calculate the decompression rate based on simulated real data and synthetic data.

[0112] In an optional embodiment of the present application, the calculation module 502 further includes: The matrix submodule is used to calculate the feature matrices of real data and synthetic data respectively; The feature submodule is used to calculate the eigenvalues ​​and eigenvectors based on their respective feature matrices; The principal component submodule is used to determine the number of principal components to be retained for the real data and synthetic data based on their respective eigenvalues; The calculation submodule is used to input the real data and the synthetic data into a preset compression rate function in combination with the number of principal components, and generate a compression rate from the real data to the synthetic data and a decompression rate from the synthetic data to the real data.

[0113] In an optional embodiment of the present application, the calculation submodule includes: A principal component unit is used to determine the number of principal components eigenvalues ​​and the target eigenvectors corresponding to the number of principal components eigenvalues ​​from the eigenvalues, and determine the target eigenvectors as the principal components; A projection unit, used to project the real data matrix and the synthetic data matrix onto their respective principal components, to generate the real data after dimensionality reduction and the synthetic data after dimensionality reduction respectively; A calculation unit, used to calculate the compression rate of the real data and the compression rate of the synthetic data respectively according to the number of their respective eigenvalues ​​and the number of principal components through a preset compression rate function; The determination unit is used to determine the compression rate of the synthetic data as the compression rate and the compression rate of the real data as the decompression rate.

[0114] In an optional embodiment of the present application, the determination module 503 includes: A comparison submodule, used to compare the compression rate and the decompression rate; The compression distance submodule is used to determine the larger of the compression rate and the decompression rate as the compression distance.

[0115] In an optional embodiment of the present application, the generative adversarial network includes a generative network and an adversarial network, both of which are multi-layer fully connected neural networks. The device includes: A first input module, used to input a first training sample into an initial generative adversarial network; A first synthesis module, used for initially generating a network to generate a first synthesis sample of a first training sample; A first training module is used to train an initial generative adversarial network using a first training sample and a first synthetic sample to obtain a preliminary generative adversarial network; A second input module, used for inputting a second training sample into a preliminary generative adversarial network; A second synthesis module, used for preliminarily generating a second synthesis sample of a second training sample generated by a network; The second training module is used to train the preliminary discriminant network with the second training sample and the second synthetic sample to generate a trained generative adversarial network.

[0116] In an optional embodiment of the present application, the first training module includes: An adjustment submodule, used to input the first training sample and the first synthetic sample into the initial adversarial network respectively, calculate the first loss using the cross entropy loss function, and adjust the initial generative adversarial network according to the first loss until the first preset condition is met, thereby obtaining a preliminary generative adversarial network; The second training module includes: A label submodule, used to add labels to the second training sample and the second synthetic sample respectively, so as to construct a training data set pair; The second training submodule is used to train the preliminary discriminant network using the training data set until the preliminary discriminant network meets the second preset condition, thereby generating a generative adversarial network.

[0117] The embodiment of the present application discloses a data similarity evaluation device for a simulation service platform, comprising obtaining real data, and generating synthetic data based on the real data through a generative adversarial network, calculating the compression rate from the real data to the synthetic data, and the decompression rate from the synthetic data to the real data, determining the compression distance between the real data and the synthetic data according to the compression rate and the decompression rate, and evaluating the similarity between the real data and the synthetic data based on the compression distance. The device can generate synthetic data that is highly similar to the real data distribution through a generative adversarial network, and the neural network model provides strong support for data enhancement and simulation experiments. The similarity is evaluated by calculating the compression distance value between the real data and the synthetic data, which provides an effective means for simulation data analysis and processing.

[0118] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0119] Figure 6 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application.

[0120] See also Figure 6 , the electronic device 600 includes a memory 610 and a processor 620 .

[0121] The processor 620 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0122] The memory 610 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 620 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, an optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 6010 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 610 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0123] The memory 610 stores executable codes, and when the executable codes are processed by the processor 620 , the processor 620 can execute part or all of the methods described above.

[0124] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0125] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0126] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for evaluating data similarity of a simulation service platform, characterized in that: The method comprises: Acquire real data, and generate synthetic data based on the real data through a generative adversarial network; calculating a compression rate from the real data to the synthetic data, and a decompression rate from the synthetic data to the real data; determining a compression distance between the real data and the synthetic data according to the compression rate and the decompression rate; The similarity between the real data and the synthetic data is evaluated based on the compression distance.

2. The method according to claim 1, characterized in that Generating synthetic data based on the real data by generating an adversarial network comprises: generating noise according to data characteristics of the real data; The synthetic data is generated through the generative adversarial network based on the noise and the real data.

3. The method according to claim 2, characterized in that The calculating of the compression rate from the real data to the synthetic data and the decompression rate from the synthetic data to the real data comprises: Calculating the variance of the noise, and calculating the compression rate according to the variance; Subtracting the noise from the synthetic data to generate simulated real data; The decompression rate is calculated based on the simulated real data and the synthetic data.

4. The method according to claim 1, characterized in that: The calculating of the compression rate from the real data to the synthetic data and the decompression rate from the synthetic data to the real data further comprises: Calculating feature matrices of the real data and the synthetic data respectively; Calculate eigenvalues ​​and eigenvectors based on their respective characteristic matrices; Determining the number of principal components to be retained for the real data and the synthetic data according to their respective eigenvalues; The real data and the synthetic data are respectively input into a preset compression rate function in combination with the number of principal components to generate a compression rate from the real data to the synthetic data and a decompression rate from the synthetic data to the real data.

5. The method according to claim 4, characterized in that The step of inputting the real data matrix and the synthetic data matrix into a preset compression rate function in combination with the number of principal components to generate a compression rate from the real data to the synthetic data and a decompression rate from the synthetic data to the real data comprises: Determine the number of eigenvalues ​​of the principal components and the target eigenvectors corresponding to the number of eigenvalues ​​of the principal components from the eigenvalues, and determine the target eigenvectors as the principal components; Projecting the real data matrix and the synthetic data matrix onto their respective principal components, respectively, to generate reduced-dimensional real data and reduced-dimensional synthetic data; Calculate the compression rate of the real data and the compression rate of the synthetic data respectively through the preset compression rate function according to the number of the respective eigenvalues ​​and the number of the principal components; The compression rate of the synthetic data is determined as the compression rate, and the compression rate of the real data is determined as the decompression rate.

6. The method according to claim 1, characterized in that The determining the compression distance between the real data and the synthetic data according to the compression rate and the decompression rate comprises: comparing the compression rate and the decompression rate; The larger one of the compression rate and the decompression rate is determined as the compression distance.

7. The method according to claim 1, characterized in that The generative adversarial network includes a generative network and an adversarial network, both of which are multi-layer fully connected neural networks. The generative adversarial network is obtained by the following method: Inputting the first training sample into the initial generative adversarial network; The initial generation network generates a first synthetic sample of the first training sample; Using the first training sample and the first synthetic sample to train the initial generative adversarial network to obtain a preliminary generative adversarial network; Inputting a second training sample into the preliminary generative adversarial network; The preliminary generation network generates a second synthetic sample of the second training sample; The second training sample and the second synthetic sample are used to train a preliminary discriminant network to generate a trained generative adversarial network.

8. The method according to claim 7, characterized in that The adopting the first training sample and the first synthetic sample to train the initial generative adversarial network to obtain a preliminary generative adversarial network comprises: Inputting the first training sample and the first synthetic sample into an initial adversarial network respectively, calculating a first loss using a cross entropy loss function, and adjusting the initial generative adversarial network according to the first loss until a first preset condition is met, thereby obtaining the preliminary generative adversarial network; The using the second training sample and the second synthetic sample to train the preliminary discriminant network to generate a trained generative adversarial network comprises: adding labels to the second training sample and the second synthetic sample respectively to construct a training data set pair; The preliminary discriminant network is trained using the training data set until the preliminary discriminant network meets a second preset condition, thereby generating the generative adversarial network.

9. A simulation service platform data similarity evaluation device, characterized in that: The device comprises: An acquisition module, used to acquire real data and generate synthetic data based on the real data through a generative adversarial network; A calculation module, used for calculating a compression rate from the real data to the synthetic data, and a decompression rate from the synthetic data to the real data; a determination module, configured to determine a compression distance between the real data and the synthetic data according to the compression rate and the decompression rate; An evaluation module is used to evaluate the similarity between the real data and the synthetic data based on the compression distance.

10. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 8.

Images