Data processing method and device, equipment and medium

Abstract

The invention relates to the technical field of data migration, in particular to a data processing method and device, equipment and a medium. In the embodiment of the invention, the reconstruction matrix is generated by using the VAE model, so that on the premise of not leaking the real content of the data in the database, the statistical distribution characteristics of the data can be reserved, the test data with the same distribution can be generated for the test environment, the authenticity of the test data is ensured, and meanwhile, the security of the data is improved.

Description

Technical Field

[0001] This application relates to the field of data migration technology, and in particular to a data processing method, apparatus, device and medium. Background Technology

[0002] With the development of technology, data migration tools are widely used in enterprise information management, database management, software testing, and other fields. In particular, when conducting software functional or performance testing, the test environment often needs to simulate production environment data to obtain accurate test results. Therefore, migration tools can be used to migrate data to the software under test for testing purposes.

[0003] However, migration tools directly copy data to the software under test. This data may contain sensitive information. If testing is conducted directly based on this data, it may lead to the leakage of sensitive information and seriously affect data security.

[0004] Therefore, generating test data with the same distribution characteristics without revealing the real data has become a problem to be solved. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and medium to solve the problem that existing technologies, which directly use real data for software testing, can lead to the leakage of sensitive returned information and poor security.

[0006] This application provides a data processing method, the method comprising: Based on the data type of each field and the corresponding first processing rule, each piece of data to be migrated in the database is processed to obtain the feature vector corresponding to each piece of data; wherein, each piece of data contains multiple sub-data, and each sub-data corresponds to a field; A variational autoencoder (VAE) model is used to reconstruct the feature matrix composed of the feature vectors corresponding to each data point in the database, resulting in a reconstruction matrix. Each row in the feature matrix corresponds to the feature vector of one data point, and each row in the reconstruction matrix corresponds to the reconstructed feature vector of one data point. Based on the data type of each saved field and the corresponding second processing rule, each reconstruction feature vector in the reconstruction matrix is ​​processed to obtain the reconstruction data corresponding to each reconstruction feature vector, and the reconstruction data is used as test data for testing.

[0007] Furthermore, the step of processing each piece of data to be migrated in the database according to the data type of each saved field and the corresponding first processing rule to obtain the feature vector corresponding to each piece of data includes: For each sub-data in each data entry, if the data type of the field containing the sub-data is numeric, then determine the first mean and first standard deviation of the field in the database, determine the difference between the sub-data and the first mean, and determine the first ratio of the difference to the first standard deviation as the feature value of the sub-data; if the data type of the field containing the sub-data is string, then determine the string length of the sub-data as the feature value of the sub-data; if the data type of the field containing the sub-data is date, then convert the sub-data to a timestamp, determine the second mean and second standard deviation of the timestamp corresponding to the field in the database, determine the second difference between the timestamp and the second mean, and determine the second ratio of the second difference to the second standard deviation as the feature value of the sub-data. Arrange the feature values ​​corresponding to each sub-data of each data according to the set order, and determine the sequence of arrangement as the feature vector corresponding to each data.

[0008] Furthermore, the VAE model includes an encoder and a decoder; The variational autoencoder (VAE) model is used to reconstruct the feature matrix composed of the feature vectors corresponding to each data entry in the database, resulting in a reconstructed matrix including: The feature matrix is ​​encoded using an encoder to output the latent variables corresponding to the feature matrix; The latent variables are decoded using the decoder to obtain the reconstruction matrix.

[0009] Furthermore, the encoder includes a hidden layer and a latent space layer; The process of encoding the feature matrix using an encoder to output the latent variables corresponding to the feature matrix includes: The feature matrix is ​​reduced in dimensionality using the hidden layer to obtain the first candidate matrix; The latent variables are calculated using the latent space layer on the first candidate matrix to determine the latent variables.

[0010] Furthermore, the decoder includes a hidden layer, a raw output layer, and a structured constraint mapping layer; The step of decoding the latent variables using the decoder and outputting an initial reconstruction matrix based on the latent variables includes: The hidden layer is used to process the latent variables to obtain a second candidate matrix; The second candidate matrix is ​​increased in dimension using the original output layer to obtain a third candidate matrix; wherein the dimension of the third candidate matrix is ​​the same as the dimension of the feature matrix. The structured constraint mapping layer updates the candidate feature values ​​corresponding to each field in the third candidate matrix according to preset constraint rules to obtain the reconstruction matrix.

[0011] Further, the structured constraint mapping layer updates the candidate feature values ​​corresponding to each field in the third candidate matrix according to preset constraint rules to obtain the reconstructed matrix, including: For each field, if the field is an integer type field, the Hard-Sigmoid function and the first eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues; if the field is a floating-point type field, the Sigmoid function and the second eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues.

[0012] Furthermore, the training process of the VAE model: Obtain the sample feature matrix; The encoder of the VAE model to be trained is used to encode the sample feature matrix and output the sample latent variables corresponding to the sample feature matrix. The decoder of the VAE model to be trained is used to decode the latent variables of the sample, and an initial reconstruction matrix is ​​output based on the latent variables of the sample. The initial reconstruction matrix is ​​corrected using an inter-field coupling constraint model to obtain the sample reconstruction matrix; Based on the sample reconstruction matrix and the sample feature matrix, the loss value is determined, and the parameters of the VAE model are adjusted according to the loss value.

[0013] Furthermore, the initial reconstruction matrix is ​​corrected using an inter-field coupling constraint model to obtain the sample reconstruction matrix, which includes: The initial reconstruction matrix is ​​input into the inter-field coupling constraint model to obtain the residual matrix output by the inter-field coupling constraint model for correcting the initial reconstruction matrix; The initial reconstruction matrix is ​​corrected based on the residual matrix to obtain the sample reconstruction matrix.

[0014] Furthermore, the inter-field coupling constraint model includes a first fully connected layer, a second fully connected layer, and an output layer; The process of determining the residual matrix includes: The first fully connected layer processes the initial reconstruction matrix using a preset first weight matrix and activation function to obtain an intermediate matrix; The second fully connected layer processes the intermediate matrix using a preset second weight matrix to obtain the initial residual matrix; The output layer determines the elements in the initial residual matrix whose values ​​exceed a preset range, modifies the values ​​corresponding to the elements to preset values, and outputs the modified initial residual matrix as the residual matrix.

[0015] This application embodiment also provides a data processing apparatus, the apparatus comprising: The processing module is used to process each piece of data to be migrated in the database according to the data type of each field and the corresponding first processing rule, and to obtain the feature vector corresponding to each piece of data; wherein, each piece of data contains multiple sub-data, and each sub-data corresponds to a field; The reconstruction module is used to reconstruct the feature matrix composed of the feature vectors corresponding to each data point in the database using a variational autoencoder (VAE) model, thereby obtaining a reconstruction matrix; wherein each row in the feature matrix corresponds to the feature vector of a data point, and each row in the reconstruction matrix corresponds to the reconstructed feature vector of a data point. The processing module is further configured to process each reconstruction feature vector in the reconstruction matrix according to the data type of each field and the corresponding second processing rule, to obtain the reconstruction data corresponding to each reconstruction feature vector, and to use the reconstruction data as test data for testing.

[0016] Further, the processing module is specifically configured to, for each sub-data in each data entry, if the data type corresponding to the field containing the sub-data is numerical data, determine the first mean and first standard deviation corresponding to the field in the database, determine the difference between the sub-data and the first mean, and determine the first ratio of the difference to the first standard deviation as the feature value of the sub-data; if the data type corresponding to the field containing the sub-data is string data, determine the string length of the sub-data as the feature value of the sub-data; if the data type corresponding to the field containing the sub-data is date data, convert the sub-data to a timestamp, determine the second mean and second standard deviation of the timestamp corresponding to the field in the database, determine the second difference between the timestamp and the second mean, and determine the second ratio of the second difference to the second standard deviation as the feature value of the sub-data; arrange the feature values ​​corresponding to each sub-data of each data entry according to a set order, and determine the sequence obtained by the arrangement as the feature vector corresponding to each data entry.

[0017] Furthermore, the VAE model includes an encoder and a decoder; The reconstruction module is specifically used to encode the feature matrix using an encoder, output the latent variables corresponding to the feature matrix, and decode the latent variables using the decoder to obtain the reconstruction matrix.

[0018] Furthermore, the encoder includes a hidden layer and a latent space layer; The reconstruction module is specifically used to perform dimensionality reduction processing on the feature matrix using the hidden layer to obtain a first candidate matrix; and to perform latent variable calculation on the first candidate matrix using the latent space layer to determine the latent variables.

[0019] Furthermore, the decoder includes a hidden layer, a raw output layer, and a structured constraint mapping layer; The reconstruction module is specifically used to process the latent variables using the hidden layer to obtain a second candidate matrix; to perform dimensionality increase processing on the second candidate matrix using the original output layer to obtain a third candidate matrix; wherein the dimension of the third candidate matrix is ​​consistent with the dimension of the feature matrix; and to update the candidate feature values ​​corresponding to each field in the third candidate matrix using the structured constraint mapping layer according to preset constraint rules to obtain the reconstruction matrix.

[0020] Furthermore, the reconstruction module is specifically used to perform the following for each field: if the field is an integer type field, the Hard-Sigmoid function and the first eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues; if the field is a floating-point type field, the Sigmoid function and the second eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues.

[0021] Furthermore, the device also includes: The training module is used to acquire a sample feature matrix; the encoder of the VAE model to be trained encodes the sample feature matrix and outputs the sample latent variables corresponding to the sample feature matrix; the decoder of the VAE model to be trained decodes the sample latent variables and outputs an initial reconstruction matrix constructed based on the sample latent variables; the initial reconstruction matrix is ​​corrected using an inter-field coupling constraint model to obtain a sample reconstruction matrix; based on the sample reconstruction matrix and the sample feature matrix, a loss value is determined, and the parameters of the VAE model are adjusted according to the loss value.

[0022] Furthermore, the training module is specifically used to input the initial reconstruction matrix into the inter-field coupling constraint model, obtain the residual matrix output by the inter-field coupling constraint model for correcting the initial reconstruction matrix, and correct the initial reconstruction matrix according to the residual matrix to obtain the sample reconstruction matrix.

[0023] Furthermore, the inter-field coupling constraint model includes a first fully connected layer, a second fully connected layer, and an output layer; The training module is specifically used to process the initial reconstruction matrix using a preset first weight matrix and activation function in the first fully connected layer to obtain an intermediate matrix; the second fully connected layer processes the intermediate matrix using a preset second weight matrix to obtain an initial residual matrix; the output layer determines the elements in the initial residual matrix whose corresponding values ​​exceed a preset range, modifies the values ​​corresponding to the elements to preset values, and outputs the modified initial residual matrix as the residual matrix.

[0024] This application also provides an electronic device, which includes a processor for executing a computer program stored in a memory to implement the steps of any of the data processing methods described above.

[0025] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the data processing methods described above.

[0026] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the steps of any of the data processing methods described above.

[0027] In this embodiment, each piece of data to be migrated in the database is processed according to the data type of each field and the corresponding first processing rule to obtain a feature vector corresponding to each piece of data. Each piece of data contains multiple sub-data, and each sub-data corresponds to a field. A variational autoencoder (VAE) model is used to reconstruct the feature matrix composed of the feature vectors corresponding to each piece of data in the database, resulting in a reconstruction matrix. Each row in the feature matrix corresponds to a feature vector of one piece of data, and each row in the reconstruction matrix corresponds to a reconstructed feature vector of one piece of data. According to the data type of each field and the corresponding second processing rule, each reconstructed feature vector in the reconstruction matrix is ​​processed to obtain reconstructed data corresponding to each reconstructed feature vector. This reconstructed data is then used as test data for testing. By using the VAE model to generate the reconstruction matrix, the electronic device can retain the statistical distribution characteristics of the data without revealing the true content of the data in the database, generating test data with the same distribution for the test environment, ensuring the authenticity of the test data, and improving data security. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a schematic diagram of a data processing procedure provided in an embodiment of this application; Figure 2 A data processing flowchart provided for embodiments of this application; Figure 3 This application provides a schematic diagram of a data processing system structure. Figure 4 This is a schematic diagram of a data processing device structure provided in an embodiment of this application; Figure 5 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0031] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.

[0032] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that in the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solutions of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0033] Data migration tools are widely used in enterprise information management, database management, software testing, and other fields. When conducting software functional or performance testing, the test environment often needs to simulate production environment data to obtain accurate test results. However, directly copying production environment data may pose privacy and security issues, especially when the data contains sensitive information (such as user names, contact information, financial data, etc.). Therefore, generating test data with the same distribution characteristics without disclosing real data has become an important requirement in enterprise data migration.

[0034] Currently, data migration typically employs two methods: one is to directly generate random data, but this method struggles to preserve the distribution characteristics of production data; the other is to perform a certain degree of anonymization on existing production environment data, however, this method usually cannot preserve the overall distribution characteristics of the data while protecting privacy, especially when dealing with multiple types of data (such as numerical values, dates, and strings), where data consistency and correlation are difficult to guarantee.

[0035] In Method 1, test data can be created using a random number generator. This tool generates different types of data, such as random strings, dates, or numbers, based on a specified data format. However, while the data generated using Method 1 can guarantee privacy, it cannot maintain consistency in distribution with production data, making it difficult to simulate a real production environment and resulting in inaccurate test results.

[0036] Method two utilizes data anonymization techniques to partially modify or obfuscate data in the production environment to protect data privacy. Common anonymization methods include character replacement, partial masking, and data encryption. However, method two is typically only suitable for certain sensitive fields (such as names and ID numbers) and struggles to handle diverse data types while preserving the statistical distribution characteristics of the data. For numeric or date-based data, the anonymization process may disrupt the logical relationships between data points, leading to distorted test results.

[0037] While the two methods mentioned above can meet basic testing needs in certain scenarios, they still have the following drawbacks in terms of privacy protection and data distribution consistency in production environment data: Disadvantages of Method 1: 1. Data distribution does not match real data: Random data generation tools usually generate test data through preset rules, but the distribution of these generated data differs greatly from that of real production environment data. Especially when it is necessary to simulate real user behavior or business data, the randomly generated data cannot accurately reflect the statistical characteristics of the production environment, resulting in distorted test results.

[0038] 2. Difficulty in handling the complexity of multiple data types: Random data generation tools can generate different types of data, such as strings, numbers, and dates, but these tools can usually only generate data that conforms to a specific format and cannot simulate the complex relationships between data. For example, randomly generated date data may not have a logical chronological relationship, and numerical data may lack the patterns of change in the real environment.

[0039] 3. Lack of correlation between data: In a real production environment, the correlation between multiple fields is a crucial factor affecting business logic. Randomly generated test data often only conforms to the expected format within a single field, failing to reflect the correlation between fields. For example, randomly generated user registration dates and last login dates may not reflect real-world logic, leading to problems in business logic validation.

[0040] Disadvantages of Method Two: 1. Incomplete Privacy Protection: While data anonymization techniques can partially mask sensitive information (such as names and contact information), they cannot completely guarantee privacy when processing numerical or date-based data. For example, some algorithms may still infer the original data content based on certain features or relationships, leading to the risk of privacy leaks.

[0041] 2. Data distribution distortion: Data anonymization is usually achieved through replacement or partial encryption, but this process may destroy the distribution characteristics of the original data, especially when dealing with large-scale numerical data. The distribution of the anonymized data may differ significantly from the distribution of the production environment data, thus leading to a decrease in the accuracy of the test results.

[0042] 3. Insufficient ability to process multiple data types: While anonymization techniques perform well when processing string-based data, they can easily disrupt the logical relationships between numerical and date-based data. For example, date anonymization often involves randomizing dates, which may result in data that fails to reflect the true time-series characteristics of the production environment.

[0043] 4. Inability to guarantee data correlation: In production data, different data fields usually have certain correlations (such as the logical correlation between user registration time and account balance). Data anonymization technology has difficulty processing multiple fields simultaneously and maintaining their correlations; the anonymized data is usually processed only on a single field.

[0044] In summary, both method one and method two face the following common problems: 1. Inability to preserve the statistical distribution characteristics of production data: Existing technologies often compromise data distribution when dealing with data privacy, which makes it impossible for test data to truly simulate business scenarios in the production environment. This is especially true when conducting performance testing, where the distribution characteristics of test data are crucial to the accuracy of test results.

[0045] 2. The generated data cannot maintain the correlation between fields: In a real production environment, there are complex logical and data relationships between different fields, and existing technologies usually cannot effectively maintain this correlation, resulting in the generated data not being able to accurately reflect the real situation of the production environment.

[0046] 3. Insufficient adaptability: Existing technologies have difficulty processing multiple types of data (such as numeric, date, and string data) simultaneously, and cannot cope with complex data structures and diverse data needs.

[0047] Based on this, this application proposes a data processing method that utilizes a Variational Auto Encoder (VAE) model to combine data privacy protection with the preservation of data distribution, providing high-quality test data for the testing environment. Specifically, the method includes: processing each piece of data to be migrated in the database according to the data type of each saved field and the corresponding first processing rule to obtain a feature vector corresponding to each piece of data; wherein each piece of data contains multiple sub-data, and each sub-data corresponds to a field; using the Variational Auto Encoder (VAE) model, performing data reconstruction on the feature matrix composed of the feature vectors corresponding to each piece of data in the database to obtain a reconstruction matrix; wherein each row in the feature matrix corresponds to the feature vector of one piece of data, and each row in the reconstruction matrix corresponds to the reconstructed feature vector of one piece of data; processing each reconstructed feature vector in the reconstruction matrix according to the data type of each saved field and the corresponding second processing rule to obtain the reconstructed data corresponding to each reconstructed feature vector, and using the reconstructed data as test data for testing.

[0048] Example 1: Figure 1 This application provides a schematic diagram of a data processing process, which includes: S101: Based on the data type of each field and the corresponding first processing rule, process each piece of data to be migrated in the database to obtain the feature vector corresponding to each piece of data; wherein, each piece of data contains multiple sub-data, and each sub-data corresponds to a field.

[0049] The data processing method provided in this application is applied to an electronic device, which may be a PC, a server, etc.

[0050] In this embodiment of the application, when there is a testing requirement, the electronic device can migrate each piece of data from the database, wherein the database can be any database, such as a MySQL database used to store data generated in the production environment.

[0051] Specifically, technicians can send test commands to electronic devices via external devices or other means. These test commands may carry a first identifier for the database, a second identifier for the data to be migrated, and a third identifier for the application to be tested. Alternatively, the test command may only carry the third identifier of the application to be tested. The data to be migrated can be all the data in the database or only a portion of it. Furthermore, the different data to be migrated can be stored in the same database or in different databases.

[0052] That is, if the test instruction carries a first identifier for the database and a second identifier for the data to be migrated, the number of first identifiers can be one or more, and the number of second identifiers can be one or more. If the number of first identifiers is multiple, the test instruction can also carry a correspondence between the second identifiers and the first identifiers, or it can not carry a correspondence between the second identifiers and the first identifiers.

[0053] Based on this, after receiving the test command, the electronic device obtains the information carried in the test command and retrieves the data to be tested from the corresponding database.

[0054] The following describes several possible ways to obtain the data to be tested: Method 1: If the test instruction carries a first identifier of the database and a second identifier of the data to be migrated, and the number of first identifiers is one, then the electronic device retrieves the data of the second identifier from the database of the first identifier and uses the data of the second identifier as the data to be migrated. Method 2: If the test instruction carries a first identifier of the database and a second identifier of the data to be migrated, and there are multiple first identifiers, and the test instruction does not carry a correspondence between the second identifier and the first identifier, then the electronic device obtains the data of the second identifier from the database of the first identifier and uses the data of the second identifier as the data to be migrated.

[0055] Method 3: If the test instruction carries a first identifier of the database and a second identifier of the data to be migrated, and there are multiple first identifiers, and the test instruction carries a correspondence between the second identifier and the first identifier, then the electronic device retrieves the corresponding second identifier data from the database of each first identifier according to the correspondence between the second identifier and the first identifier, and uses the data of the second identifier as the data to be migrated.

[0056] Method 4: If the test instruction does not carry the first identifier of the database and the second identifier of the data to be migrated, the electronic device will obtain the default data from the default database and use the default data as the data to be migrated.

[0057] To enhance data security and prevent privacy breaches, electronic devices are pre-configured with the data type and corresponding primary processing rule for each field. This primary processing rule is used to convert sensitive information into other data, thereby ensuring data security. The data types of these fields include, but are not limited to: numeric data, string data, and date data.

[0058] Specifically, the data to be migrated obtained by the electronic device includes multiple sub-data. Each sub-data corresponds to a field. The electronic device can process each sub-data of each data according to the word type of the field corresponding to each sub-data and the first processing rule corresponding to each data type that is pre-configured, to obtain the feature vector corresponding to each data.

[0059] For example, for each sub-data of each data entry, the electronic device determines the target data type of the field corresponding to the sub-data; the electronic device searches for the target first processing rule corresponding to the target data type in the stored data type and the corresponding first processing rule, and uses the target first processing rule to process the sub-data to obtain the feature value corresponding to the sub-data.

[0060] The electronic device sorts the feature values ​​corresponding to each sub-data in each data entry and determines the resulting sequence as the feature vector corresponding to each data entry.

[0061] It should be noted that in the embodiments of this application, the fields that are the same in each dimension of the feature vector corresponding to different data are not identical. For example, the first dimension of the feature vector corresponds to the name, and the second dimension corresponds to the identity ID, etc.

[0062] S102: Using a variational autoencoder (VAE) model, data reconstruction is performed on the feature matrix composed of the feature vectors corresponding to each data point in the database to obtain a reconstruction matrix; wherein, each row in the feature matrix corresponds to the feature vector of a data point, and each row in the reconstruction matrix corresponds to the reconstructed feature vector of a data point.

[0063] In this embodiment of the application, after obtaining the feature vector corresponding to each piece of data, the electronic device can control the VAE model to reconstruct the data based on the feature vector corresponding to each piece of data, and obtain the reconstructed feature vector corresponding to each feature vector. The reconstructed feature vector is different from its corresponding feature vector. Therefore, the data obtained based on each reconstructed feature vector is different from the data to be migrated obtained by the electronic device, thereby avoiding the leakage of sensitive information of the data to be migrated.

[0064] Specifically, the electronic device constructs a feature matrix based on the feature vector corresponding to each data point. Each row in this feature matrix represents a feature vector corresponding to one data point. The electronic device then inputs this feature matrix into a VAE model, which reconstructs the data based on this feature matrix to obtain a reconstructed matrix.

[0065] S103: Based on the data type of each field and the corresponding second processing rule, process each reconstruction feature vector in the reconstruction matrix to obtain the reconstruction data corresponding to each reconstruction feature vector, and use the reconstruction data as test data for testing.

[0066] In this embodiment of the application, the electronic device is further configured with a second processing rule, which is used to indicate the method of assigning values ​​to each field based on the reconstructed feature vector.

[0067] Based on this, after obtaining the reconstruction matrix output by the VAE model, the electronic device determines each reconstruction feature vector contained in the reconstruction matrix. The electronic device obtains the data type of each saved field and the corresponding second processing rule, and processes each reconstruction feature vector according to the data type of each field and the corresponding second processing rule to obtain the reconstruction data corresponding to each reconstruction feature vector.

[0068] Specifically, for each dimension of each reconstructed feature vector, the electronic device determines the target field corresponding to that dimension; based on the data type of each field and the corresponding second processing rule, the electronic device determines the target second processing rule corresponding to that field, and based on the target second processing rule and the reconstructed feature value corresponding to that dimension, determines the reconstructed sub-data corresponding to that dimension.

[0069] In this embodiment, if the data type corresponding to the target field is numerical data, the electronic device uses the mean and standard deviation of the target field in the data to be migrated to perform inverse standardization on the reconstructed feature value of the corresponding dimension of the target field in the reconstruction matrix. The electronic device can use the following formula for processing:

[0070] in, To reconstruct the sub-data, The reconstructed feature value of the target field in the reconstruction matrix. The standard deviation of the target field in the data to be migrated. This is the mean of the target field in the data to be migrated.

[0071] If the target field corresponds to date data, the electronic device uses the mean and standard deviation of the timestamp corresponding to the target field in the data to be migrated to perform inverse standardization on the reconstructed feature values ​​of the corresponding dimension of the target field in the reconstruction matrix. The electronic device can use the following formula for processing:

[0072] in, To reconstruct the sub-data, The reconstructed feature value of the target field in the reconstruction matrix. The standard deviation of the timestamps corresponding to the target field in the data to be migrated. This is the average of the timestamps corresponding to the target field in the data to be migrated.

[0073] If the data type corresponding to the target field is string data, the electronic device rounds the reconstruction feature value of the corresponding dimension in the reconstruction matrix and randomly generates a string based on the rounded reconstruction feature value. The electronic device can use the following formula for processing: For example, for the field "name", the electronic device reconstructs the feature value of the corresponding dimension of the field "name" in the reconstruction matrix. Generate a random Chinese character string of the appropriate length. Example: If ( If )=2.3, then round it down to 2 to generate "Li Si"; for the email field, the electronic device uses the reconstructed feature value of the corresponding dimension of the email field in the reconstruction matrix. Generate a random string of the appropriate length, in the format username@domain.com. Example: If ( If )=19.7, then round it down to 20 to generate "user12345@example.com"; for the field phone_number, the electronic device uses the reconstructed feature value of the dimension corresponding to the field phone_number in the reconstruction matrix. Generate a random string of the appropriate length. Example: If ( If )=10.9, then round it down to 11, generating "13987654321".

[0074] The following example illustrates how to generate specific test data: For the field balance, if the field balance corresponds to the reconstructed eigenvalue of the dimension in the reconstruction matrix... Based on this reconstruction feature value, the generated reconstruction sub-data is: .

[0075] For the field registration_date, if the reconstruction feature value of the corresponding dimension in the reconstruction matrix is... Based on this reconstruction feature value, the reconstruction timestamp corresponding to the generated reconstruction sub-data is: The reconstruction timestamp was converted to reconstruction sub-data as 2021-02-02.

[0076] For the field 'name', if the reconstructed feature value of the corresponding dimension in the reconstruction matrix is... Based on this reconstruction feature value, the string length corresponding to the generated reconstructed sub-data is... A reconstructed sub-data is randomly generated based on the string length, such as "Zhao Liu".

[0077] For the field email, if the reconstructed feature value of the corresponding dimension in the reconstruction matrix is... Based on this reconstruction feature value, the string length corresponding to the generated reconstructed sub-data is... A reconstructed sub-data is randomly generated based on the string length, such as "userabc@example.com".

[0078] For the field phone_number, if the reconstruction feature value of the corresponding dimension in the reconstruction matrix is... Based on this reconstruction feature value, the string length corresponding to the generated reconstructed sub-data is... A reconstructed sub-data is randomly generated based on the string length, such as "13876543210".

[0079] In this embodiment of the application, after the electronic device generates reconstruction data based on the reconstruction matrix, the electronic device writes the generated reconstruction data into the test database.

[0080] The field correspondence between the reconstructed data and the test database can be as follows: user_id: can retain the original value or be regenerated; name: A randomly generated string of Chinese characters; email: A randomly generated email address; phone_number: The generated random phone number; registration_date: The date the application was generated; balance: The balance generated.

[0081] Electronic devices can also perform data validation on test data to ensure that the data type of each field in the test data is correct and to determine that the test data meets the database constraints, such as non-null and uniqueness.

[0082] The electronic device concatenates the reconstructed sub-data corresponding to each reconstructed feature vector in a set order to obtain the reconstructed data corresponding to each reconstructed feature vector.

[0083] In this embodiment, the electronic device generates a reconstruction matrix using a VAE model, which allows the statistical distribution characteristics of the data to be preserved without revealing the true content of the data in the database. This generates test data with the same distribution for the test environment, ensuring the authenticity of the test data and improving data security.

[0084] Example 2: To avoid the leakage of sensitive information and improve the security of testing, based on the above embodiments, in this embodiment, the step of processing each piece of data to be migrated in the database according to the data type of each stored field and the corresponding first processing rule to obtain the feature vector corresponding to each piece of data includes: For each sub-data in each data entry, if the data type of the field containing the sub-data is numeric, then determine the first mean and first standard deviation of the field in the database, determine the difference between the sub-data and the first mean, and determine the first ratio of the difference to the first standard deviation as the feature value of the sub-data; if the data type of the field containing the sub-data is string, then determine the string length of the sub-data as the feature value of the sub-data; if the data type of the field containing the sub-data is date, then convert the sub-data to a timestamp, determine the second mean and second standard deviation of the timestamp corresponding to the field in the database, determine the second difference between the timestamp and the second mean, and determine the second ratio of the second difference to the second standard deviation as the feature value of the sub-data. Arrange the feature values ​​corresponding to each sub-data of each data according to the set order, and determine the sequence of arrangement as the feature vector corresponding to each data.

[0085] To enhance data security and prevent privacy breaches, electronic devices are pre-configured with the data type and corresponding primary processing rule for each field. This primary processing rule is used to convert sensitive information into other data, thereby ensuring data security. The data types of these fields include, but are not limited to: numeric data, string data, and date data.

[0086] In this application embodiment, different data types correspond to different first processing rules. Several possible first processing rules are described in detail below: Method 1: The first processing rule for numerical data is to standardize each numerical field (such as balance).

[0087] Specifically, if the data type of the field containing the sub-data is numerical, the electronic device determines the first mean and first standard deviation of that field in the database, and determines the difference between the sub-data and the first mean. The electronic device then determines the first ratio of this difference to the first standard deviation as the characteristic value of the sub-data.

[0088] Method 2: The first processing rule for string data is: record the string length: for each string field (such as name, email, phone_number), record its string length, replace the string content with placeholders or leave it blank, and avoid processing the actual string content.

[0089] Specifically, if the data type of the field containing the sub-data is string data, then the string length of the sub-data is determined as the feature value of the sub-data.

[0090] Method 3: The first processing rule for date data is to convert the date field (such as registration_date) into a timestamp and then perform standardization processing.

[0091] Specifically, if the data type of the field containing the sub-data is date data, the electronic device converts the sub-data into a timestamp according to the first processing rule, determines the second mean and second standard deviation of the timestamp corresponding to the field in the database, determines the second difference between the sub-data and the second mean, and determines the second ratio of the second difference to the second standard deviation as the feature value of the sub-data.

[0092] After determining the feature value corresponding to each sub-data of each data point, the electronic device combines the feature values ​​obtained from processing each sub-data of each data point into a fixed-length vector, which is then used as the input to the VAE model.

[0093] Specifically, the electronic device arranges the feature values ​​corresponding to each sub-data of each data according to a set order, and determines the resulting sequence as the feature vector corresponding to each data.

[0094] For example, assuming the data to be transferred consists of the following three data points, the process of constructing the feature vector for each data point and the process of constructing the feature matrix are as follows: Data 1: Zhang San, zhangsan@example.com, 13800138000, 2021-01-15, 1025.50; Data 2: Li Si, lisi@example.com, 13800138001, 2021-02-20, 2048.75; Data 3: Wang Wu, wangwu@example.com, 13800138002, 2021-03-10, 512.00; For each sub-data in the above three pieces of data, the corresponding fields are name, email, phone_number, registration_date, and balance; Among them, the data types corresponding to the fields name, email, and phone_number are string data, the data type corresponding to the field registration_dater is date data, and the data type corresponding to the field balance is numeric data.

[0095] For the field name, the electronic device determines the string lengths of the sub-data of Data 1, Data 2, and Data 3 in this field. Among them, the sub-data of Data 1 in this field is "Zhang San", and its string length is 2 characters; the sub-data of Data 2 in this field is "Li Si", and its string length is 2 characters; the sub-data of Data 3 in this field is "Wang Wu", and its string length is 2 characters.

[0096] For the field email, the electronic device determines the string lengths of the sub-data of Data 1, Data 2, and Data 3 in this field. Among them, the sub-data of Data 1 in this field is "zhangsan@example.com", and its string length is 20 characters; the sub-data of Data 2 in this field is "lisi@example.com", and its string length is 16 characters; the sub-data of Data 3 in this field is "wangwu@example.com", and its string length is 18 characters.

[0097] For the field registration_dater, the sub-data of Data 1 in this field is "2021-01-15", and the electronic device converts "2021-01-15" to the timestamp "1610668800"; the sub-data of Data 2 in this field is "2021-02-20", and the electronic device converts "2021-02-20" to the timestamp "1613779200"; the sub-data of Data 3 in this field is "2021-03-10", and the electronic device converts "2021-03-10" to the timestamp "1615334400".

[0098] The electronic device calculates the second mean of the field registration_dater based on the following formula The value is 1613260800:

[0099] After determining the second mean Based on 1613260800, the electronic device calculates the second standard deviation of the registration_dater field using the following formula. For 1904020:

[0100] After determining the second mean The value is 1613260800, and the second standard deviation is... Based on 1904020, the electronic device uses the following formulas to calculate the feature value d1 corresponding to timestamp 1610668800 as -1.36, the feature value d2 corresponding to timestamp 1613779200 as 0.27, and the feature value d3 corresponding to timestamp 1615334400 as 1.09:

[0101] For the field "balance", data 1 has a sub-data value of "1025.5", data 2 has a sub-data value of "2048.75", and data 3 has a sub-data value of "512.00". The electronic device calculates the first mean of the "balancer" field using the following formula. The value is 1195.42.

[0102] After determining the first mean Based on 1195.42, the electronic device calculates the first standard deviation of the balancer field using the following formula. The value is 622.35.

[0103] After determining the first mean The value is 1195.42, and the first standard deviation is... Based on 622.35, the electronic device uses the following formulas to calculate the eigenvalues ​​b1 and b2 for sub-data 1025.5 as -0.27, b2 and b3 for sub-data 2048.75 as 1.37, and b3 and b3 for sub-data 512.00 as -1.10 respectively:

[0104] The electronic device arranges the feature values ​​corresponding to each sub-data of each data according to the set order [normalized balance, normalized registration_date, name_length, email_length, phone_number_length], and determines the sequence of arrangement as the feature vector corresponding to each data.

[0105] Wherein, the feature vector corresponding to data 1 is The feature vector corresponding to data 2 is The feature vector corresponding to data 3 is

[0106] The electronic device can construct a feature matrix based on the feature vectors corresponding to data 1, data 2, and data 3. This feature matrix can be represented as:

[0107] Example 3: To avoid the leakage of sensitive information and improve the security of testing, based on the above embodiments, in this application embodiment, the VAE model includes an encoder and a decoder; The variational autoencoder (VAE) model is used to reconstruct the feature matrix composed of the feature vectors corresponding to each data entry in the database, resulting in a reconstructed matrix including: The feature matrix is ​​encoded using an encoder to output the latent variables corresponding to the feature matrix; The latent variables are decoded using the decoder to obtain the reconstruction matrix.

[0108] In this embodiment, the VAE model includes at least an encoder and a decoder. The encoder compresses a high-dimensional, structured feature matrix into a low-dimensional latent space to obtain latent variables. The decoder reconstructs a reconstruction matrix from these latent variables that has the same dimensions as the original input feature matrix and outputs it.

[0109] Specifically, after the encoder obtains the input feature matrix, it progressively extracts features from the feature matrix and finally outputs latent variables. The decoder obtains the latent variables output by the encoder, maps them to the feature space corresponding to the feature matrix, and outputs a reconstruction matrix with the same dimensions as the feature matrix. Each row of this reconstruction matrix is ​​a reconstruction vector.

[0110] Example 4: To avoid the leakage of sensitive information and improve the security of testing, based on the above embodiments, in this application embodiment, the encoder includes a hidden layer and a latent space layer; The process of encoding the feature matrix using an encoder to output the latent variables corresponding to the feature matrix includes: The feature matrix is ​​reduced in dimensionality using the hidden layer to obtain the first candidate matrix; The latent variables are calculated using the latent space layer on the first candidate matrix to determine the latent variables.

[0111] In this embodiment, the encoder is designed as a multilayer perceptron. The input of the encoder is a feature vector matrix, which includes the feature vector corresponding to each data point. For example, the dimension of the feature matrix is ​​(batch_size, 5).

[0112] The encoder consists of hidden layers and a latent space layer. The hidden layer can be composed of one or more fully connected layers, each followed by a non-linear activation function (such as ReLU or Tanh). This hidden layer performs multiple dimensionality transformations on the feature matrix based on one or more fully connected layers, achieving dimensionality reduction of the feature matrix to obtain the first candidate matrix. For example, the hidden layer can be a Dense layer with 4 nodes, and each node is followed by a ReLU activation function.

[0113] The latent space layer is responsible for outputting the distribution parameters of the latent variables. Specifically, the latent space layer is used to calculate the mean of the latent variables. and standard deviation Then according to the formula Calculate latent variables ,in, Let μ be a random number sampled from a standard normal distribution. The encoder aims to map the input feature matrix to a low-dimensional latent variable space; therefore, the dimensions of the mean and standard deviation should be smaller than the dimension of the input feature vector. For example, the dimensions of the mean μ and standard deviation σ can be 2-dimensional.

[0114] Example 5: To avoid the leakage of sensitive information and improve the security of testing, based on the above embodiments, in this application embodiment, the decoder includes a hidden layer, a raw output layer, and a structured constraint mapping layer; The step of decoding the latent variables using the decoder and outputting an initial reconstruction matrix based on the latent variables includes: The hidden layer is used to process the latent variables to obtain a second candidate matrix; The second candidate matrix is ​​increased in dimension using the original output layer to obtain a third candidate matrix; wherein the dimension of the third candidate matrix is ​​the same as the dimension of the feature matrix. The structured constraint mapping layer updates the candidate feature values ​​corresponding to each field in the third candidate matrix according to preset constraint rules to obtain the reconstruction matrix.

[0115] In practical applications, when generating data, the VAE model only ensures that the generated data is similar to the original data from a statistical distribution perspective, but it cannot guarantee that the generated data meets business constraints. This may lead to the following problems: generating negative balances, generating future dates, and generating strings that exceed the database field length limit. These violations will cause the data to fail when loaded into the test database, requiring a lot of post-processing work to correct the data, as well as a decrease in the data quality of the test environment, affecting the effectiveness of the test.

[0116] Based on this, the encoder structure has been updated in this embodiment by adding a structured constraint mapping layer to constrain the eigenvalues ​​of the reconstructed matrix. Therefore, the decoder in this embodiment includes at least a hidden layer, a raw output layer, and a structured constraint mapping layer.

[0117] Specifically, the hidden layer of the decoder can consist of multiple fully connected layers, each followed by an activation function (such as ReLU) to perform dimensionality upscaling on the latent variables, resulting in a second candidate matrix whose dimension is smaller than that of the feature matrix.

[0118] The original output layer is a linear transformation layer used to further increase the dimensionality of the second candidate matrix to obtain a third candidate matrix, whose dimension is the same as that of the feature matrix. If the reconstructed data is generated directly based on this third candidate matrix, some sub-data in the reconstructed data may violate business rules: for example, the field balance may generate a negative value, etc.

[0119] Based on this, the embodiments of this application add a structured constraint mapping layer after the original output layer. This structured constraint mapping layer can dynamically adjust each candidate feature value according to preset field constraints.

[0120] Specifically, the structured constraint mapping layer is configured with constraint rules. Based on the preset constraint rules, the structured constraint mapping layer can update the candidate feature values ​​corresponding to each field in the third candidate matrix to obtain the reconstructed matrix.

[0121] To avoid the leakage of sensitive information and improve the security of testing, based on the above embodiments, in this embodiment, the step of using the structured constraint mapping layer to update the candidate feature values ​​corresponding to each field in the third candidate matrix according to preset constraint rules to obtain the reconstructed matrix includes: For each field, if the field is an integer type field, the Hard-Sigmoid function and the first eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues; if the field is a floating-point type field, the Sigmoid function and the second eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues.

[0122] In this embodiment, the structured constraint mapping layer includes constraint mapping sub-layers corresponding to the output fields, including but not limited to numeric mapping sub-layers and integer shooting sub-layers. The numeric mapping sub-layer transforms the first initial eigenvalue corresponding to the floating-point type field in the third candidate matrix, ensuring that the first initial eigenvalue falls within a corresponding range. The integer shooting sub-layer transforms the second initial eigenvalue corresponding to the integer type field in the third candidate matrix, ensuring that the second initial eigenvalue falls within a corresponding range.

[0123] Integer type fields include, but are not limited to, name, email, and mobile phone number; floating-point type fields include, but are not limited to, date and balance.

[0124] For example, the numerical mapping sublayer uses the Sigmoid function to linearly transform the first initial eigenvalues ​​corresponding to date and balance in the third candidate matrix, such as: balance_out = -1.180 + (1.70 + 1.18) × Sigmoid(\hat{y}_0) to ensure that the generated balance ∈ [-1.18, 1, 7]; the integer mapping sublayer uses the Hard-Sigmoid function to linearly transform the second initial eigenvalues ​​corresponding to name, email, and phone number in the third candidate matrix, such as: rounding the second eigenvalue name_len_out corresponding to name to an integer and making name_len_out ∈ [1, 20].

[0125] Specifically, for each field, if the field is an integer type field, the Hard-Sigmoid function and the first eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues; if the field is a floating-point type field, the Sigmoid function and the second eigenvalue range corresponding to the field are used to transform the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues.

[0126] In this embodiment, for integer type fields, the electronic device uses the Hard-Sigmoid function as the processing tool. Specifically, for the first initial eigenvalue in the third candidate matrix... The Hard-Sigmoid function is mapped to [0,1]. Then, based on the minimum eigenvalue L and the maximum eigenvalue U of the first eigenvalue range corresponding to this field, the updated eigenvalue is calculated. Finally, the result is rounded to the nearest integer to ensure that the output is a valid integer. For example, if the first initial feature value corresponding to the email field in the third candidate feature matrix is ​​18.7, then after processing with the Hard-Sigmoid function, the first initial feature value can be updated to 19. The processed first initial feature value conforms to the actual distribution of email address lengths.

[0127] In this embodiment of the application, when mapping the first initial eigenvalue x to the interval [0,1] using the Hard-Sigmoid function, it can be achieved using the following formula: , where α and These are preset parameters; typically, α is 0.2 and β is 0.5.

[0128] For example, if the range of the first eigenvalue corresponding to the first initial eigenvalue x is [1, 20], then ,in, This represents the updated eigenvalues, where 1 represents the smallest eigenvalue in the first eigenvalue range, and 19 represents the number of possible eigenvalues ​​in the first eigenvalue range. This represents the first initial eigenvalue.

[0129] For example: if x=0, then , If x=5, then , .

[0130] In this embodiment, after performing a linear transformation on the eigenvalues ​​corresponding to the field using the Hard-Sigmoid function and the range of the first eigenvalues ​​corresponding to the field, the resulting value may not be an integer. Therefore, it can be processed in the following way: Method 1: Forward Propagation: Directly round n' to obtain an integer output.

[0131] in, This represents the feature value of the final output. This indicates the eigenvalues ​​obtained by processing the field using the Hard-Sigmoid function and the corresponding first eigenvalue range. This indicates the rounding operation.

[0132] Method 2: Backpropagation: The gradient bypasses the rounding operation and is directly propagated to n'.

[0133] Specifically, the gradient calculation ignores the discrepancy caused by rounding, and the gradient of len_out is directly assigned to n', that is:

[0134] Wherein, the minimum eigenvalue of the first eigenvalue range corresponding to the L field is... This represents the feature value of the final output. This indicates the feature value obtained by processing the field using the Hard-Sigmoid function and the range of the first feature value corresponding to that field.

[0135] The following is a specific example to illustrate this: Assume the first initial eigenvalue corresponding to the integer type field in the third candidate feature matrix. =0.8, and the range of the first eigenvalue is [1,20]. The processing flow is as follows: 1. Calculate the Hard-Sigmoid output: Hard-Sigmoid(0.8)=max(0,min(1,0.2×0.8+0.5))=max(0,min(1,0.66))=0.66 2. Scale to length range: n' = 1 + 19 × 0.66 = 13.54 3. Round to the nearest integer (forward propagation): name_len_out = round(13.54) = 14 That is, the first initial feature value after processing is an integer 14, which satisfies the constraint of the first feature value range [1,20].

[0136] For floating-point type fields, electronic devices use the standard Sigmoid function to compress the second initial eigenvalue corresponding to the floating-point type field into the (0,1) interval, and then use a linear transformation to map the compressed second initial eigenvalue to the range of the second eigenvalue corresponding to the field.

[0137] The electronic device can update the second initial feature value using the following formula:

[0138] in, Let U be the minimum eigenvalue of the second eigenvalue range corresponding to the field, and U be the maximum eigenvalue of the second eigenvalue range corresponding to the field. For the Sigmoid function, This is the updated second initial feature value.

[0139] For example, if the range of the second feature value corresponding to the field 'balance' is [-1.180, 1.700], then the updated initial second feature value calculated based on the above formula is: .

[0140] Based on this, regardless of whether the eigenvalues ​​corresponding to each field in the third candidate matrix meet the requirements, the eigenvalues ​​corresponding to each field in the reconstructed matrix obtained after processing by the structured constraint mapping layer will be safely clamped within the legal range.

[0141] In this embodiment, integer fields such as name and phone number use the Hard-Sigmoid function, while floating-point fields such as amount use the Sigmoid function. This is mainly based on a comprehensive consideration of the field's data type, computational efficiency requirements, and gradient processing characteristics. Integer fields typically require discretized output and efficient computation, making the linear approximation of the Hard-Sigmoid function more suitable; while floating-point fields require high-precision continuous mapping, making the Sigmoid function's smoothness superior.

[0142] In this embodiment of the application, the value range of the field is related to the sub-data corresponding to each field in the data to be migrated.

[0143] Specifically, in this embodiment of the application, for each field in the data to be migrated, if the data type corresponding to the field is numerical data, the electronic device calculates the maximum and minimum sub-data corresponding to the field, and calculates the minimum and maximum values ​​corresponding to each field based on the maximum and minimum sub-data and the preset safety margin ratio; then, based on the first mean, the first standard deviation, and the minimum value corresponding to the field, the minimum feature value corresponding to the field is determined; and based on the first mean, the first standard deviation, and the maximum value corresponding to the field, the maximum feature value corresponding to the field is determined.

[0144] Specifically, electronic devices use the following formula to calculate the minimum value of a field containing numeric data:

[0145] in, Indicates the minimum value. This indicates the smallest sub-data item corresponding to this field in the data to be migrated. This indicates the preset safety margin ratio.

[0146] For fields where the value cannot be negative, the corresponding minimum value should be max(0, ).

[0147] Electronic devices use the following formula to calculate the maximum value of a field in numeric data:

[0148] in, Indicates the maximum value. This indicates the largest sub-data item corresponding to this field in the data to be migrated. This indicates the preset safety margin ratio.

[0149] For example, if the value corresponding to the field 'balance' in the data to be migrated is [1025.50, 2048.75, 512.00], then the electronic device determines the minimum sub-data min_observed = 512.00 and the maximum sub-data max_observed = 2048.75. The electronic device's preset safety margin ratio is 10%. Based on this, the electronic device determines the minimum value corresponding to the field 'balance'. Maximum value .

[0150] Since the value of the field balance cannot be negative, the minimum value corresponding to the field balance should be max(0, 460.8) = 460.8.

[0151] In this embodiment, after the electronic device determines the minimum and maximum values ​​of a field whose data type is numerical data, the electronic device determines the minimum feature value corresponding to the field based on the first mean, the first standard deviation, and the minimum value; and determines the maximum feature value corresponding to the field based on the first mean, the first standard deviation, and the maximum value.

[0152] Specifically, the electronic device determines the third difference between the minimum value and the first mean, and determines the third ratio of the third difference to the first standard deviation as the minimum characteristic value corresponding to the field; the electronic device determines the fourth difference between the maximum value and the first mean, and determines the fourth ratio of the fourth difference to the first standard deviation as the maximum characteristic value corresponding to the field.

[0153] For example, the minimum value for the field "balance" is 460.8, and the maximum value is 2253.63. The electronic device then determines the minimum characteristic value corresponding to this field "balance". The largest eigenvalue That is, the range of the first feature value corresponding to the field balance is [ , ].

[0154] In this embodiment of the application, for each field in the data to be migrated, if the data type corresponding to the field is string data, the electronic device determines the range of the first feature value corresponding to the field according to the set rules.

[0155] For example, for the field 'name', if the rule stipulates that a name must be no less than 1 character and no more than 15 characters, then the range of the first characteristic value corresponding to the 'name' field is []. , ].

[0156] In this embodiment of the application, for each field in the data to be migrated, if the data type corresponding to the field is date data, the electronic device calculates the maximum and minimum timestamps corresponding to the field, and calculates the minimum and maximum values ​​corresponding to each field based on the maximum and minimum timestamps and the preset safety margin ratio; then, based on the second mean, the second standard deviation, and the minimum value corresponding to the field, the minimum feature value corresponding to the field is determined; and based on the second mean, the second standard deviation, and the maximum value corresponding to the field, the maximum feature value corresponding to the field is determined.

[0157] The process of calculating the minimum and maximum values ​​for each field, as well as the maximum and minimum eigenvalues, is the same as the calculation process for fields of numerical data type, and will not be repeated here.

[0158] For example, if the sub-data corresponding to the registration_date field in the data to be migrated, after being converted to timestamps, is [1610668800, 1613779200, 1615334400], then the electronic device determines the minimum value corresponding to the registration_date field to be... The maximum value is Based on this, the minimum feature value corresponding to this field determined by the electronic device is... The largest eigenvalue is .

[0159] The electronic device stores the range of feature values ​​corresponding to each extracted field as constraint information as fixed parameters of the VAE model, which will not be adjusted during model training.

[0160] For example, this constraint information can be stored in the following format: constraint_bounds={ 'min_values':[-1.18,-4.320,1,5,8], 'max_values':[1.70,4.325,20,50,15] In this context, the dimension in which the feature value range of each field is located is consistent with the dimension corresponding to that field in the feature vector.

[0161] Example 6: To avoid the leakage of sensitive information and improve the security of testing, based on the above embodiments, the training process of the VAE model in this application embodiment is as follows: Obtain the sample feature matrix; The encoder of the VAE model to be trained is used to encode the sample feature matrix and output the sample latent variables corresponding to the sample feature matrix. The decoder of the VAE model to be trained is used to decode the latent variables of the sample, and an initial reconstruction matrix is ​​output based on the latent variables of the sample. The initial reconstruction matrix is ​​corrected using an inter-field coupling constraint model to obtain the sample reconstruction matrix; Based on the sample reconstruction matrix and the sample feature matrix, the loss value is determined, and the parameters of the VAE model are adjusted according to the loss value.

[0162] In this embodiment, before training the VAE model to be trained, the electronic device extracts a large number of sample records from the database and forms a sample feature matrix after preprocessing. The dimension of the sample feature matrix is ​​the same as the dimension of the feature matrix, and the number of sample feature matrices is not limited.

[0163] The electronic device inputs the sample feature matrix into the VAE model to be trained. The encoder of the VAE model encodes the sample feature matrix, outputting the mean μ and variance σ of the sample latent variables, and then obtains the sample latent variables through reparameterization. Next, the decoder of the VAE model to be trained generates an initial reconstruction matrix based on the sample latent variables. Although the initial reconstruction matrix is ​​close to the true value in a single field, the complex relationships between fields may not yet be fully modeled. This complex relationship could indicate that users with high balances tend to register early.

[0164] Based on this, during the model training phase, the electronic device introduces an inter-field coupling constraint model. This model is used to correct the initial reconstruction matrix to obtain the sample reconstruction matrix. The inter-field coupling constraint model is a lightweight sub-network that ensures the underlying business rules between fields, typically consisting of two fully connected (FC) layers with Tanh activation. This model takes the initial reconstruction matrix as input and outputs a residual matrix of the same dimension as the initial reconstruction matrix. The electronic device then corrects the initial reconstruction matrix based on this residual matrix to obtain the final sample reconstruction matrix. The goal of this residual matrix is ​​to compensate for biases in the joint distribution between fields. For example, if the model generates an unreasonable combination of high balance but late dates, the ICCN will adjust the date values ​​through negative residuals to make them more consistent with historical patterns.

[0165] In this embodiment of the application, the electronic device determines the loss value based on the sample reconstruction matrix and the sample feature matrix, and adjusts the parameters of the VAE model to be trained based on the loss value.

[0166] The loss function consists of three parts: Reconstruction Loss, Kullback-Leibler Divergence (KL) Loss, and Covariance Matching Loss. The Reconstruction Loss can be the Mean Squared Error (MSE), which measures the element-wise difference between the sample reconstruction matrix and the sample feature matrix. The KL loss constrains the latent distribution to be close to standard normal. The Covariance Matching Loss calculates the Frobenius norm of the difference between the field covariance matrices of the sample reconstruction matrix and the sample feature matrix.

[0167] Specifically, for fields with numeric and date data types, the reconstruction loss of the field is determined using MSE. For fields with string data types, the reconstruction loss of the field can be determined using MSE or a custom loss function.

[0168] The electronic device calculates the difference between the latent distribution of the sample reconstruction matrix and the prior distribution of the sample feature matrix, and uses this difference as the KL divergence loss.

[0169] The covariance matching loss for electronic devices can be determined using the following formula:

[0170] in, Represents the true covariance matrix. Represents the sample covariance matrix. This represents the Frobenius norm. The sample covariance matrix is ​​calculated based on the sample reconstruction matrix, while the true covariance matrix is ​​calculated based on the sample feature matrix. This sample covariance matrix is ​​pre-configured.

[0171] For example, if the sample reconstruction matrix is Then, the sample covariance matrix is ​​calculated based on the sample reconstruction matrix. The calculation process is as follows: (1) Calculate the mean of each field:

[0172] (2) Solve for the sample covariance matrix based on the covariance formula:

[0173] Substituting into the calculation, we get: Cov(b,b)=0.323 Cov(d,d)=1.063 Cov(b,d)=Cov(d,b)=-0.425 Based on this, the sample covariance matrix is .

[0174] Given the true covariance matrix is Then the covariance matching loss is:

[0175] In this embodiment of the application, the loss value determined by the electronic device based on the sample reconstruction matrix and the sample feature matrix is ​​the sum of reconstruction loss, KL divergence loss and covariance matching loss.

[0176] Specifically, the loss value satisfies the following formula: in, in, and For preset weights, generally, and The value of is 0.5.

[0177] In this embodiment, after the electronic device determines the loss value, it can backpropagate the loss value using the Adam optimizer, iteratively updating all parameters of the encoder and decoder of the VAE model. After sufficient training, the VAE model can not only accurately reconstruct single fields, but also implicitly learn and reproduce complex multi-field coupling relationships, laying a solid foundation for the generation of high-quality test data.

[0178] The training phase can be summarized into the following steps: Forward propagation: The input data passes through the encoder, reparameterization, and decoder to obtain the reconstructed data; Calculate the loss: Calculate the loss value based on the reconstructed data and the original input data; Backpropagation: Calculate the gradient and update the model parameters.

[0179] To avoid the leakage of sensitive information and improve the security of testing, based on the above embodiments, in this embodiment, the step of using an inter-field coupling constraint model to correct the initial reconstruction matrix to obtain the sample reconstruction matrix includes: The initial reconstruction matrix is ​​input into the inter-field coupling constraint model to obtain the residual matrix output by the inter-field coupling constraint model for correcting the initial reconstruction matrix; The initial reconstruction matrix is ​​corrected based on the residual matrix to obtain the sample reconstruction matrix.

[0180] During the training of a VAE model, although the initial reconstruction matrix output by the decoder performs well in univariate statistics, standard VAE models often ignore higher-order dependencies between variables because they only optimize pointwise reconstruction errors. For example, a VAE model might correctly generate a high balance (balance=5000) and a recent registration date (2024), but these two rarely occur simultaneously in real-world business scenarios. In other words, users with high balances are mostly early registered users, and this logical contradiction can severely impact the validity of the test data.

[0181] Based on this, the embodiments of this application add an inter-field coupling constraint model in the training phase. The inter-field coupling constraint model takes the initial reconstruction matrix output by the decoder as input, predicts a residual matrix based on the initial reconstruction matrix, and then corrects the initial reconstruction matrix according to the residual matrix to obtain the sample reconstruction matrix, so that the sample reconstruction matrix is ​​closer to the joint distribution of the real data.

[0182] To avoid the leakage of sensitive information and improve the security of testing, based on the above embodiments, in this application embodiment, the field coupling constraint model includes a first fully connected layer, a second fully connected layer, and an output layer; The process of determining the residual matrix includes: The first fully connected layer processes the initial reconstruction matrix using a preset first weight matrix and activation function to obtain an intermediate matrix; The second fully connected layer processes the intermediate matrix using a preset second weight matrix to obtain the initial residual matrix; The output layer determines the elements in the initial residual matrix whose values ​​exceed a preset range, modifies the values ​​corresponding to the elements to preset values, and outputs the modified initial residual matrix as the residual matrix.

[0183] Specifically, in this embodiment, the field coupling constraint model includes at least a first fully connected layer, a second fully connected layer, and an output layer. First, the initial reconstruction matrix is ​​input into the first fully connected layer. This first fully connected layer uses a first weight matrix and an activation function to map the initial reconstruction matrix into an intermediate matrix, which captures the preliminary interaction information between fields.

[0184] For example, the first fully connected layer is processed using the following formula:

[0185] in, Represents the intermediate matrix. Represents the initial reconstruction matrix. This represents the first weight matrix. Represents a random number.

[0186] The second fully connected layer uses a second weight matrix to process the intermediate matrix to obtain an initial residual matrix. Each element of the initial residual matrix corresponds to the correction amount of a field.

[0187] For example, the second fully connected layer is processed using the following formula:

[0188] in, Represents the intermediate matrix. Represents the initial reconstruction matrix. This represents the second weight matrix. Represents a random number.

[0189] To prevent excessive residuals from causing data distortion, electronic devices employ a threshold mechanism at the output layer: if the absolute value of an element in the initial residual matrix exceeds a preset range, that element is modified to the preset value. For example, if the value of the element corresponding to the field 'balance' in the residual matrix is ​​+1.2, but the maximum allowable correction is +0.8, then that element is modified to +0.8.

[0190] The following example illustrates this, with the initial reconstruction matrix being: =[balance_out, date_out, name_len_out, email_len_out, phone_len_out], the residual matrix output by the inter-field coupling constraint model is: The final corrected sample reconstruction matrix Among them, This is a learnable scaling factor, with an initial value of 0.1.

[0191] The network objective of the field coupling constraint model is to keep each element in the sample reconstruction matrix within the original boundary and to make the covariance matrix of the sample reconstruction matrix as close as possible to the covariance matrix of the true sample.

[0192] The input to the field coupling constraint model is the same as the decoder output, and can be [balance_z, date_z, name_len, email_len, phone_len].

[0193] Example: If the initial reconstruction matrix of the inter-field coupling constraint model input is Then, the first fully connected layer of the inter-field coupling constraint model processes the initial reconstruction matrix to obtain the intermediate matrix. The second fully connected layer processes this intermediate matrix to obtain the initial residual matrix. .

[0194] To avoid compromising the validity of the original fields, the electronic device performs amplitude clipping on each element in the residual matrix. Specifically, the width of the valid interval for the i-th field is defined as... Set the maximum allowable adjustment amount to For continuous fields: For integer fields: round to the nearest integer after cropping.

[0195] Based on this, the initial residual matrix The residual matrix after adjustment by the output layer is The electronic device adjusts the initial reconstruction matrix based on the residual matrix to obtain the sample reconstruction matrix. .

[0196] In this embodiment of the application, after determining the sample reconstruction matrix, the electronic device will also perform a secondary check on the feature values ​​of the fields with numerical data type in the sample reconstruction matrix to determine that the feature values ​​of the fields in the sample reconstruction matrix do not exceed the corresponding feature value range.

[0197] Figure 2 The data processing flowchart provided in this application embodiment is as follows: Figure 2 As shown, the process includes: S201: Data extraction.

[0198] Connect to the production database: Use secure authentication to connect to the MySQL database in the production environment.

[0199] Data extraction: Extract the data that needs to be migrated, ensuring data integrity.

[0200] S202: Data preprocessing and encoding.

[0201] Numerical data processing: Standardize each numeric field (such as balance).

[0202] Date data processing: Convert date fields (such as registration_date) into timestamps and perform standardization processing.

[0203] String data processing: Record string length: For each string field (such as name, email, phone_number), record its string length, replace the string content with placeholders or leave it blank, and avoid processing the actual string content.

[0204] Combined input vector: Combine the feature values ​​(standardized numeric and date data, string length) of all sub-data of each data point into a fixed-length vector, which is used as the feature vector.

[0205] Constraint information calculation and storage: Numerical boundaries: Record the minimum and maximum values ​​of each field, with a safety margin of 10%-20% depending on the numerical values; Data type constraints: integer fields, floating-point precision, date range, etc.

[0206] S203: Data Obfuscation Handling - Improved VAE Model Training.

[0207] VAE model input: a feature matrix composed of the feature vectors of each data point.

[0208] The encoder of the VAE model is used to: receive the input feature matrix, process it through the hidden layer, and output the mean and standard deviation of the latent variables; and use the mean and standard deviation of the latent variables to sample random noise from the standard normal distribution to calculate the latent variables. The decoder of the VAE model is used to generate a reconstructed matrix from the latent variables, which is consistent with the format and range of the feature matrix. The field coupling constraint layer of the VAE model is used to: ensure the implicit business rules between fields, add a lightweight "coupled sub-network" (2 layers FC + Tanh) after the decoder, input all mapped fields, output residual Δ and add it to the original value; use adversarial loss or covariance loss to make the residual approach 0, so as to implicitly satisfy the joint distribution.

[0209] Loss Calculation: Reconstruction loss: For numerical and date data, use mean squared error (MSE); for string length, use MSE or a custom loss function.

[0210] KL divergence loss (: measures the difference between the latent distribution of the encoder output and the prior distribution (standard normal distribution).

[0211] Covariance matching loss: measures the difference in covariance between training data and real data; Total loss = Reconstruction loss + β × KL divergence + r × Covariance matching loss, where weights β and r are trade-off coefficients; Model optimization: Use the Adam optimizer to iteratively update the model parameters.

[0212] S204: Data generation.

[0213] Sampling latent variables: Sample latent variables z from a standard normal distribution.

[0214] Data generation: Input z into the decoder to generate a new combined vector (\hat{x}).

[0215] S205: Data post-processing.

[0216] Numeric and date data: Denormalization: Denormalize the generated normalized data to restore it to the original numerical scale.

[0217] Date format conversion: Converts the denormalized timestamp back to date format.

[0218] String data: Generate random string: Generate a random string of the same length as the generated string.

[0219] Character set selection: You can limit the character set (such as letters, numbers, Chinese characters) to meet data format and business requirements.

[0220] Formatting requirements: For strings with specific formats (such as email, phone_number), the generated string should conform to the formatting specifications.

[0221] Business rule validation: Ensure that the generated data conforms to business logic and constraints.

[0222] S206: Data loading.

[0223] Connect to the test database: Use secure authentication to connect to the MySQL database in the test environment.

[0224] Data writing: Write the generated data into the test database according to the original table structure.

[0225] Data validation: Data type validation: Ensure that the data type of each field conforms to the table structure definition.

[0226] Constraint validation: Check database constraints such as primary keys, foreign keys, and uniqueness.

[0227] Compared with the prior art, this application has the following inventive points: Invention Point 1: Using the modified VAE model to uniformly process different types of data.

[0228] This application innovatively utilizes a VAE (Variable Object Model) to process multi-type data. During data migration in the production environment, different types of data (including strings, numeric, and date types) are unified into high-dimensional vectors and input into the VAE model. The core logic of VAE is to learn the latent structure of numerical data through latent variable models and neural networks, thereby achieving effective modeling and generation of data distribution.

[0229] Preserve data distribution characteristics: The generated data is consistent with the original data in terms of statistical characteristics, meeting the testing requirements.

[0230] Protecting data privacy: The generated data cannot be reverse-engineered to the original data, effectively preventing data leakage.

[0231] Compared to directly assuming a normal distribution for generated data, VAE does not directly assume a data distribution. Instead, it learns the complex distribution of data through neural networks, without restricting the shape of the generated data distribution. This allows it to capture the complex characteristics of the data, and the generated fake data more closely approximates the true distribution of the original data. Since VAE only accepts numerical input, this application uses the following processing methods to enable it to uniformly process all types of data: String data processing: For string data, an innovative approach is proposed that only the string length is retained, generating a random string with the same length as the original string. This method avoids privacy leaks while ensuring that the format of the generated data is consistent with the original data.

[0232] Processing of numerical and date data: For numerical and date data, this application adopts standardization and destandardization methods to maintain the distribution characteristics of the data during the model generation process, so that the generated test data can conform to the statistical regularity and logic in the production environment.

[0233] Invention Point 2: Processing multi-row data through a unified VAE model without the need for column IDs.

[0234] Existing VAE models typically require introducing a column ID for each data column to distinguish the characteristics of different fields. This application improves the input method of the VAE model, enabling the model to directly process multiple rows of data in a data table, thus avoiding the complexity of introducing column IDs.

[0235] This innovation is reflected in the following aspects: Batch processing of multiple rows of data: This application can process multiple rows of data from a data table simultaneously in a batch and input them into the model by combining all field data into a fixed-length high-dimensional vector. This not only reduces the complexity of the model but also improves the efficiency of training and generation.

[0236] Preserving inter-field relationships: By processing the entire row of data simultaneously, the model can learn and preserve the logical relationships between multiple fields, ensuring that the generated data is logically consistent with the data in the production environment.

[0237] The significance of preserving field associations: The patterns in the raw data are as follows: users with high balances are usually early registrants with shorter email addresses (corporate emails); users with low balances are usually recent registrants with longer email addresses (personal emails).

[0238] This application enables the VAE model to learn the correlation between [high balance, early date, short email_len], and the final reconstructed data can be [4800, 2020-03, 18], which is logical.

[0239] Invention Point 3: A processing method that combines privacy protection and data distribution consistency.

[0240] Another important innovation of this application lies in simultaneously solving the problems of data privacy protection and data distribution consistency. Current technologies often compromise the original data distribution characteristics when handling privacy protection, while this application overcomes this problem by: Generate test data consistent with production data distribution: By learning the potential distribution of production data through the VAE model, this application can generate test data with the same statistical characteristics, which is especially suitable for performance testing and load testing.

[0241] Privacy protection: The generated data will not reveal the actual content of the production data. In particular, when processing string data, a string of the same length is randomly generated to replace the original data, ensuring that sensitive information is not exposed.

[0242] This application proposes a constraint-aware conditional VAE architecture to ensure that the generated data meets predefined constraints.

[0243] Invention Point 4: Differentiated processing strategy for multiple types of data.

[0244] The tool employs differentiated processing strategies based on different data types to ensure a balance between privacy protection and data quality for each data type. Specifically, this includes: String data: By recording the length and generating random strings, privacy protection and format consistency are ensured.

[0245] Numerical data: Through standardization and destandardization, we ensure that the generated data conforms to the statistical distribution in the production environment.

[0246] Date-based data: By converting it to a timestamp and standardizing it, the generated data is guaranteed to conform to the true time series pattern.

[0247] Invention Point 5: Unified data migration process design.

[0248] This application automates the entire data migration process—from extraction and processing of production data to generation and loading—through a seamless data migration design. Each step is supported by specific technical means, ensuring the efficiency of the entire process and the accuracy of the generated data.

[0249] Invention Point 6: Applicable to complex and diverse data scenarios.

[0250] The design of this application is capable of handling complex and diverse business scenarios, including but not limited to: Large-scale performance testing: Simulate real-world business stress by generating test data that is consistent with the data distribution in the production environment.

[0251] Functional testing: Generate test data with the same logical relationship without disclosing real data to ensure the coverage and accuracy of functional testing.

[0252] This application's embodiments innovatively address many shortcomings of existing technologies by introducing a VAE model. The following are the main advantages of this application compared to existing technologies: 1. Balancing privacy protection with data distribution consistency: Existing technologies often disrupt the statistical distribution characteristics of the original data during the data anonymization process, resulting in test data failing to accurately simulate business scenarios in the production environment. This application, through a VAE model, can simultaneously achieve privacy protection and data distribution consistency during data migration. Privacy protection: String data is protected by generating random strings of the same length to ensure that the actual data content is not disclosed.

[0253] Data distribution consistency: The model learns the potential distribution of the production data, and the generated test data is consistent with the production data in terms of statistical properties.

[0254] 2. Efficient data processing and generation: This application utilizes a unified VAE model to process multi-row data without introducing column IDs, thus improving the efficiency of model training and data generation. Batch processing of multiple rows of data: Process multiple rows of data in a table simultaneously within a batch, avoiding the complexity of processing column by column and greatly improving the speed of data processing and generation.

[0255] Eliminate column ID complexity: There is no need to introduce a column ID for each data column, which reduces the complexity of the model and allows the model to maintain the logical relationship between data.

[0256] 3. Flexible processing capability for multiple data types: This application designs a flexible processing mechanism for different data types, adapting to the different processing needs of string, numeric, and date data: String data processing: By recording the string length and generating random strings, data format consistency is maintained while effectively protecting privacy.

[0257] Numerical and date data processing: Through standardization and destandardization, we ensure that the generated data conforms to the statistical characteristics of the production environment, and that the time series and numerical logic of the data are effectively preserved.

[0258] 4. Maintain logical connections between data: Existing data anonymization or random generation tools often fail to maintain the logical relationships between different fields. This application, however, processes and generates data from the entire row, ensuring that the generated data is logically consistent with the data in the production environment. Especially in data processing involving multiple fields such as dates and numbers, this application ensures that the generated test data is logically sound and suitable for complex business testing.

[0259] 5. Suitable for complex business and testing scenarios: Through its innovative VAE model, this application can be widely applied in various business and testing scenarios, and has significant advantages, especially in the following aspects: Large-scale performance testing: By generating test data that is consistent with the data distribution in the production environment, it can accurately simulate business pressure in the production environment, helping enterprises to conduct large-scale performance testing and stress testing.

[0260] Functional testing: Generate test data with the same logical relationship without disclosing real data, ensuring the accuracy of functional testing. It is particularly suitable for functional verification scenarios involving complex data logic.

[0261] Data integration and automation: Through modular design, this application supports automatic data extraction, processing, generation and loading, reducing manual intervention and improving the efficiency of test data management.

[0262] 6. Comprehensive data privacy protection mechanism: By generating test data with the same statistical properties as the original data but without any actual correlation, this application can maximize the protection of data privacy while ensuring the quality of the test data. Especially when processing sensitive information (such as names and contact information), the generated data cannot be traced or reversed, thus effectively reducing the risk of data leakage.

[0263] Based on the same inventive concept, embodiments of this application provide a data processing system. Figure 3 Please refer to the schematic diagram of a data processing system structure provided in this application embodiment. Figure 3 The system includes: a data extraction module, a data processing module, and a data loading module.

[0264] The data extraction module is specifically used for: Connect to the production database: Connect to the MySQL database in the production environment via secure authentication.

[0265] Data extraction: Extract all data that needs to be migrated based on the database table structure.

[0266] Data validation: Ensure the integrity and accuracy of the extracted data.

[0267] The data processing module is specifically used for: Data classification: Based on the data type of the fields, data is classified into string (str), numeric (numeric), date (date), etc.

[0268] Extract constraint information: Numerical boundaries: Record the minimum and maximum values ​​for each field, with a safety margin of 10%-20% depending on the numerical values. Data type constraints: integer fields, floating-point precision, date range, etc. Data obfuscation: Different methods of generating fake data are used for different data types.

[0269] String data processing: Calculate the length of the string data and use the length as input to the improved VAE.

[0270] Date data processing: Convert date data into timestamps, maintain the consistency of date distribution, and input them into the VAE.

[0271] Numerical data processing: Modeling and generating numerical data using variational autoencoders (VAEs) preserves the statistical distribution characteristics of the data while protecting data privacy.

[0272] The data loading module is specifically used for: Connect to the test database: Connect to the MySQL database in the test environment.

[0273] Data writing: Write the processed data into the test database according to the original table structure.

[0274] Data validation: Ensure data integrity, correct constraints, and correct relationships.

[0275] Example 7: Based on the same inventive concept, embodiments of this application provide a data processing apparatus. Figure 4 This is a schematic diagram of a data processing device structure provided in an embodiment of this application. Please refer to... Figure 4 The device includes: The processing module 401 is used to process each piece of data to be migrated in the database according to the data type of each field and the corresponding first processing rule, and obtain the feature vector corresponding to each piece of data; wherein, each piece of data contains multiple sub-data, and each sub-data corresponds to a field; The reconstruction module 402 is used to reconstruct the feature matrix composed of the feature vectors corresponding to each data in the database using a variational autoencoder (VAE) model, and obtain a reconstruction matrix; wherein, each row in the feature matrix corresponds to the feature vector of a data, and each row in the reconstruction matrix corresponds to the reconstructed feature vector of a data. The processing module 401 is further configured to process each reconstruction feature vector in the reconstruction matrix according to the data type of each field and the corresponding second processing rule, to obtain the reconstruction data corresponding to each reconstruction feature vector, and to use the reconstruction data as test data for testing.

[0276] In one possible implementation, the processing module 401 is specifically configured to, for each sub-data in each data entry, if the data type corresponding to the field containing the sub-data is numerical data, determine the first mean and first standard deviation corresponding to the field in the database, determine the difference between the sub-data and the first mean, and determine the first ratio of the difference to the first standard deviation as the feature value of the sub-data; if the data type corresponding to the field containing the sub-data is string data, determine the string length of the sub-data as the feature value of the sub-data; if the data type corresponding to the field containing the sub-data is date data, convert the sub-data to a timestamp, determine the second mean and second standard deviation of the timestamp corresponding to the field in the database, determine the second difference between the timestamp and the second mean, and determine the second ratio of the second difference to the second standard deviation as the feature value of the sub-data; arrange the feature values ​​corresponding to each sub-data of each data entry according to a set order, and determine the sequence obtained by the arrangement as the feature vector corresponding to each data entry.

[0277] In one possible implementation, the VAE model includes an encoder and a decoder; The reconstruction module 402 is specifically used to encode the feature matrix using an encoder, output the latent variables corresponding to the feature matrix, and decode the latent variables using the decoder to obtain the reconstruction matrix.

[0278] In one possible implementation, the encoder includes a hidden layer and a latent space layer; The reconstruction module 402 is specifically used to perform dimensionality reduction processing on the feature matrix using the hidden layer to obtain a first candidate matrix; and to perform latent variable calculation on the first candidate matrix using the latent space layer to determine the latent variables.

[0279] In one possible implementation, the decoder includes a hidden layer, a raw output layer, and a structured constraint mapping layer; The reconstruction module 402 is specifically used to process the latent variables using the hidden layer to obtain a second candidate matrix; to perform dimensionality increase processing on the second candidate matrix using the original output layer to obtain a third candidate matrix; wherein the dimension of the third candidate matrix is ​​consistent with the dimension of the feature matrix; and to update the candidate feature values ​​corresponding to each field in the third candidate matrix using the structured constraint mapping layer according to preset constraint rules to obtain the reconstruction matrix.

[0280] In one possible implementation, the reconstruction module 402 is specifically used to perform the following for each field: if the field is an integer type field, then the Hard-Sigmoid function and the first eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain updated eigenvalues; if the field is a floating-point type field, then the Sigmoid function and the second eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain updated eigenvalues.

[0281] In one possible implementation, the device further includes: Training module 403 is used to obtain a sample feature matrix; encode the sample feature matrix using the encoder of the VAE model to be trained, and output the sample latent variables corresponding to the sample feature matrix; decode the sample latent variables using the decoder of the VAE model to be trained, and output an initial reconstruction matrix constructed based on the sample latent variables; correct the initial reconstruction matrix using an inter-field coupling constraint model to obtain a sample reconstruction matrix; determine the loss value based on the sample reconstruction matrix and the sample feature matrix, and adjust the parameters of the VAE model based on the loss value.

[0282] In one possible implementation, the training module 403 is specifically used to input the initial reconstruction matrix into the inter-field coupling constraint model, obtain the residual matrix output by the inter-field coupling constraint model for correcting the initial reconstruction matrix, and correct the initial reconstruction matrix according to the residual matrix to obtain the sample reconstruction matrix.

[0283] In one possible implementation, the inter-field coupling constraint model includes a first fully connected layer, a second fully connected layer, and an output layer; The training module 403 is specifically used to process the initial reconstruction matrix using a preset first weight matrix and activation function in the first fully connected layer to obtain an intermediate matrix; the second fully connected layer processes the intermediate matrix using a preset second weight matrix to obtain an initial residual matrix; the output layer determines the elements in the initial residual matrix whose corresponding values ​​exceed a preset range, modifies the values ​​corresponding to the elements to preset values, and outputs the modified initial residual matrix as the residual matrix.

[0284] Example 8: Based on the same inventive concept, embodiments of this application provide an electronic device that can implement the steps of the data manipulation method described above. Figure 5 This application provides a schematic diagram of an electronic device structure, such as... Figure 5As shown, it includes: processor 501, communication interface 502, memory 503 and communication bus 504, wherein processor 501, communication interface 502 and memory 503 communicate with each other through communication bus 504. The memory 503 stores a computer program. When the program is executed by the processor 501, the processor 501 performs the following steps: Based on the data type of each field and the corresponding first processing rule, each piece of data to be migrated in the database is processed to obtain the feature vector corresponding to each piece of data; wherein, each piece of data contains multiple sub-data, and each sub-data corresponds to a field; A variational autoencoder (VAE) model is used to reconstruct the feature matrix composed of the feature vectors corresponding to each data point in the database, resulting in a reconstruction matrix. Each row in the feature matrix corresponds to the feature vector of one data point, and each row in the reconstruction matrix corresponds to the reconstructed feature vector of one data point. Based on the data type of each saved field and the corresponding second processing rule, each reconstruction feature vector in the reconstruction matrix is ​​processed to obtain the reconstruction data corresponding to each reconstruction feature vector, and the reconstruction data is used as test data for testing.

[0285] In one possible implementation, the step of processing each piece of data to be migrated in the database according to the data type of each stored field and the corresponding first processing rule to obtain the feature vector corresponding to each piece of data includes: For each sub-data in each data entry, if the data type of the field containing the sub-data is numeric, then determine the first mean and first standard deviation of the field in the database, determine the difference between the sub-data and the first mean, and determine the first ratio of the difference to the first standard deviation as the feature value of the sub-data; if the data type of the field containing the sub-data is string, then determine the string length of the sub-data as the feature value of the sub-data; if the data type of the field containing the sub-data is date, then convert the sub-data to a timestamp, determine the second mean and second standard deviation of the timestamp corresponding to the field in the database, determine the second difference between the timestamp and the second mean, and determine the second ratio of the second difference to the second standard deviation as the feature value of the sub-data. Arrange the feature values ​​corresponding to each sub-data of each data according to the set order, and determine the sequence of arrangement as the feature vector corresponding to each data.

[0286] In one possible implementation, the VAE model includes an encoder and a decoder; The variational autoencoder (VAE) model is used to reconstruct the feature matrix composed of the feature vectors corresponding to each data entry in the database, resulting in a reconstructed matrix including: The feature matrix is ​​encoded using an encoder to output the latent variables corresponding to the feature matrix; The latent variables are decoded using the decoder to obtain the reconstruction matrix.

[0287] In one possible implementation, the encoder includes a hidden layer and a latent space layer; The process of encoding the feature matrix using an encoder to output the latent variables corresponding to the feature matrix includes: The feature matrix is ​​reduced in dimensionality using the hidden layer to obtain the first candidate matrix; The latent variables are calculated using the latent space layer on the first candidate matrix to determine the latent variables.

[0288] In one possible implementation, the decoder includes a hidden layer, a raw output layer, and a structured constraint mapping layer; The step of decoding the latent variables using the decoder and outputting an initial reconstruction matrix based on the latent variables includes: The hidden layer is used to process the latent variables to obtain a second candidate matrix; The second candidate matrix is ​​increased in dimension using the original output layer to obtain a third candidate matrix; wherein the dimension of the third candidate matrix is ​​the same as the dimension of the feature matrix. The structured constraint mapping layer updates the candidate feature values ​​corresponding to each field in the third candidate matrix according to preset constraint rules to obtain the reconstruction matrix.

[0289] In one possible implementation, the step of using the structured constraint mapping layer to update the candidate feature values ​​corresponding to each field in the third candidate matrix according to preset constraint rules to obtain the reconstructed matrix includes: For each field, if the field is an integer type field, the Hard-Sigmoid function and the first eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues; if the field is a floating-point type field, the Sigmoid function and the second eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues.

[0290] In one possible implementation, the training process of the VAE model is as follows: Obtain the sample feature matrix; The encoder of the VAE model to be trained is used to encode the sample feature matrix and output the sample latent variables corresponding to the sample feature matrix. The decoder of the VAE model to be trained is used to decode the latent variables of the sample, and an initial reconstruction matrix is ​​output based on the latent variables of the sample. The initial reconstruction matrix is ​​corrected using an inter-field coupling constraint model to obtain the sample reconstruction matrix; Based on the sample reconstruction matrix and the sample feature matrix, the loss value is determined, and the parameters of the VAE model are adjusted according to the loss value.

[0291] In one possible implementation, the step of correcting the initial reconstruction matrix using an inter-field coupling constraint model to obtain the sample reconstruction matrix includes: The initial reconstruction matrix is ​​input into the inter-field coupling constraint model to obtain the residual matrix output by the inter-field coupling constraint model for correcting the initial reconstruction matrix; The initial reconstruction matrix is ​​corrected based on the residual matrix to obtain the sample reconstruction matrix.

[0292] In one possible implementation, the inter-field coupling constraint model includes a first fully connected layer, a second fully connected layer, and an output layer; The process of determining the residual matrix includes: The first fully connected layer processes the initial reconstruction matrix using a preset first weight matrix and activation function to obtain an intermediate matrix; The second fully connected layer processes the intermediate matrix using a preset second weight matrix to obtain the initial residual matrix; The output layer determines the elements in the initial residual matrix whose values ​​exceed a preset range, modifies the values ​​corresponding to the elements to preset values, and outputs the modified initial residual matrix as the residual matrix.

[0293] Since the principle of the above-mentioned electronic device in solving the problem is similar to that of the data processing method, the implementation of the above-mentioned electronic device can be referred to the embodiments of the method, and repeated parts will not be described again.

[0294] The communication bus mentioned in the aforementioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. Communication interface 502 is used for communication between the aforementioned electronic device and other devices. The memory can include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0295] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0296] Example 9: Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a processor. When the program runs on the processor, it causes the processor to perform the following steps: Based on the data type of each field and the corresponding first processing rule, each piece of data to be migrated in the database is processed to obtain the feature vector corresponding to each piece of data; wherein, each piece of data contains multiple sub-data, and each sub-data corresponds to a field; A variational autoencoder (VAE) model is used to reconstruct the feature matrix composed of the feature vectors corresponding to each data point in the database, resulting in a reconstruction matrix. Each row in the feature matrix corresponds to the feature vector of one data point, and each row in the reconstruction matrix corresponds to the reconstructed feature vector of one data point. Based on the data type of each saved field and the corresponding second processing rule, each reconstruction feature vector in the reconstruction matrix is ​​processed to obtain the reconstruction data corresponding to each reconstruction feature vector, and the reconstruction data is used as test data for testing.

[0297] In one possible implementation, the step of processing each piece of data to be migrated in the database according to the data type of each stored field and the corresponding first processing rule to obtain the feature vector corresponding to each piece of data includes: For each sub-data in each data entry, if the data type of the field containing the sub-data is numeric, then determine the first mean and first standard deviation of the field in the database, determine the difference between the sub-data and the first mean, and determine the first ratio of the difference to the first standard deviation as the feature value of the sub-data; if the data type of the field containing the sub-data is string, then determine the string length of the sub-data as the feature value of the sub-data; if the data type of the field containing the sub-data is date, then convert the sub-data to a timestamp, determine the second mean and second standard deviation of the timestamp corresponding to the field in the database, determine the second difference between the timestamp and the second mean, and determine the second ratio of the second difference to the second standard deviation as the feature value of the sub-data. Arrange the feature values ​​corresponding to each sub-data of each data according to the set order, and determine the sequence of arrangement as the feature vector corresponding to each data.

[0298] In one possible implementation, the VAE model includes an encoder and a decoder; The variational autoencoder (VAE) model is used to reconstruct the feature matrix composed of the feature vectors corresponding to each data entry in the database, resulting in a reconstructed matrix including: The feature matrix is ​​encoded using an encoder to output the latent variables corresponding to the feature matrix; The latent variables are decoded using the decoder to obtain the reconstruction matrix.

[0299] In one possible implementation, the encoder includes a hidden layer and a latent space layer; The process of encoding the feature matrix using an encoder to output the latent variables corresponding to the feature matrix includes: The feature matrix is ​​reduced in dimensionality using the hidden layer to obtain the first candidate matrix; The latent variables are calculated using the latent space layer on the first candidate matrix to determine the latent variables.

[0300] In one possible implementation, the decoder includes a hidden layer, a raw output layer, and a structured constraint mapping layer; The step of decoding the latent variables using the decoder and outputting an initial reconstruction matrix based on the latent variables includes: The hidden layer is used to process the latent variables to obtain a second candidate matrix; The second candidate matrix is ​​increased in dimension using the original output layer to obtain a third candidate matrix; wherein the dimension of the third candidate matrix is ​​the same as the dimension of the feature matrix. The structured constraint mapping layer updates the candidate feature values ​​corresponding to each field in the third candidate matrix according to preset constraint rules to obtain the reconstruction matrix.

[0301] In one possible implementation, the step of using the structured constraint mapping layer to update the candidate feature values ​​corresponding to each field in the third candidate matrix according to preset constraint rules to obtain the reconstructed matrix includes: For each field, if the field is an integer type field, the Hard-Sigmoid function and the first eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues; if the field is a floating-point type field, the Sigmoid function and the second eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues.

[0302] In one possible implementation, the training process of the VAE model is as follows: Obtain the sample feature matrix; The encoder of the VAE model to be trained is used to encode the sample feature matrix and output the sample latent variables corresponding to the sample feature matrix. The decoder of the VAE model to be trained is used to decode the latent variables of the sample, and an initial reconstruction matrix is ​​output based on the latent variables of the sample. The initial reconstruction matrix is ​​corrected using an inter-field coupling constraint model to obtain the sample reconstruction matrix; Based on the sample reconstruction matrix and the sample feature matrix, the loss value is determined, and the parameters of the VAE model are adjusted according to the loss value.

[0303] In one possible implementation, the step of correcting the initial reconstruction matrix using an inter-field coupling constraint model to obtain the sample reconstruction matrix includes: The initial reconstruction matrix is ​​input into the inter-field coupling constraint model to obtain the residual matrix output by the inter-field coupling constraint model for correcting the initial reconstruction matrix; The initial reconstruction matrix is ​​corrected based on the residual matrix to obtain the sample reconstruction matrix.

[0304] In one possible implementation, the inter-field coupling constraint model includes a first fully connected layer, a second fully connected layer, and an output layer; The process of determining the residual matrix includes: The first fully connected layer processes the initial reconstruction matrix using a preset first weight matrix and activation function to obtain an intermediate matrix; The second fully connected layer processes the intermediate matrix using a preset second weight matrix to obtain the initial residual matrix; The output layer determines the elements in the initial residual matrix whose values ​​exceed a preset range, modifies the values ​​corresponding to the elements to preset values, and outputs the modified initial residual matrix as the residual matrix.

[0305] Since the principle of the computer-readable storage medium in solving the problem is similar to that of the data processing method, the implementation of the computer-readable storage medium can be found in the implementation of the method, and the repeated parts will not be described again.

[0306] Example 10: Based on the same inventive concept, this application also provides a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to execute any of the data processing methods described above. Since the principle by which the above-described computer program product solves the problem is similar to that of the data processing method, the implementation of the above-described computer program product can be referred to the implementation of the method, and repeated details will not be described again.

[0307] In this embodiment, each piece of data to be migrated in the database is processed according to the data type of each field and the corresponding first processing rule to obtain a feature vector corresponding to each piece of data. Each piece of data contains multiple sub-data, and each sub-data corresponds to a field. A variational autoencoder (VAE) model is used to reconstruct the feature matrix composed of the feature vectors corresponding to each piece of data in the database, resulting in a reconstruction matrix. Each row in the feature matrix corresponds to a feature vector of one piece of data, and each row in the reconstruction matrix corresponds to a reconstructed feature vector of one piece of data. According to the data type of each field and the corresponding second processing rule, each reconstructed feature vector in the reconstruction matrix is ​​processed to obtain reconstructed data corresponding to each reconstructed feature vector. This reconstructed data is then used as test data for testing. By using the VAE model to generate the reconstruction matrix, the electronic device can retain the statistical distribution characteristics of the data without revealing the true content of the data in the database, generating test data with the same distribution for the test environment, ensuring the authenticity of the test data, and improving data security.

[0308] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0309] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0310] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0311] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0312] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method, characterized in that, The method includes: Based on the data type of each field and the corresponding first processing rule, each piece of data to be migrated in the database is processed to obtain the feature vector corresponding to each piece of data; wherein, each piece of data contains multiple sub-data, and each sub-data corresponds to a field; A variational autoencoder (VAE) model is used to reconstruct the feature matrix composed of the feature vectors corresponding to each data point in the database, resulting in a reconstruction matrix. Each row in the feature matrix corresponds to the feature vector of one data point, and each row in the reconstruction matrix corresponds to the reconstructed feature vector of one data point. Based on the data type of each saved field and the corresponding second processing rule, each reconstruction feature vector in the reconstruction matrix is ​​processed to obtain the reconstruction data corresponding to each reconstruction feature vector, and the reconstruction data is used as test data for testing.

2. The method according to claim 1, characterized in that, The process of processing each piece of data to be migrated in the database according to the data type of each saved field and the corresponding first processing rule to obtain the feature vector corresponding to each piece of data includes: For each sub-data in each data entry, if the data type of the field containing the sub-data is numeric, then determine the first mean and first standard deviation of the field in the database, determine the difference between the sub-data and the first mean, and determine the first ratio of the difference to the first standard deviation as the feature value of the sub-data; if the data type of the field containing the sub-data is string, then determine the string length of the sub-data as the feature value of the sub-data; if the data type of the field containing the sub-data is date, then convert the sub-data to a timestamp, determine the second mean and second standard deviation of the timestamp corresponding to the field in the database, determine the second difference between the timestamp and the second mean, and determine the second ratio of the second difference to the second standard deviation as the feature value of the sub-data. Arrange the feature values ​​corresponding to each sub-data of each data according to the set order, and determine the sequence of arrangement as the feature vector corresponding to each data.

3. The method according to claim 1, characterized in that, The VAE model includes an encoder and a decoder; The variational autoencoder (VAE) model is used to reconstruct the feature matrix composed of the feature vectors corresponding to each data entry in the database, resulting in a reconstructed matrix including: The feature matrix is ​​encoded using an encoder to output the latent variables corresponding to the feature matrix; The latent variables are decoded using the decoder to obtain the reconstruction matrix.

4. The method according to claim 3, characterized in that, The encoder includes a hidden layer and a potential space layer; The process of encoding the feature matrix using an encoder to output the latent variables corresponding to the feature matrix includes: The feature matrix is ​​reduced in dimensionality using the hidden layer to obtain the first candidate matrix; The latent variables are calculated using the latent space layer on the first candidate matrix to determine the latent variables.

5. The method according to claim 3, characterized in that, The decoder includes a hidden layer, a raw output layer, and a structured constraint mapping layer; The step of decoding the latent variables using the decoder and outputting an initial reconstruction matrix based on the latent variables includes: The hidden layer is used to process the latent variables to obtain a second candidate matrix; The second candidate matrix is ​​increased in dimension using the original output layer to obtain a third candidate matrix; wherein the dimension of the third candidate matrix is ​​the same as the dimension of the feature matrix. The structured constraint mapping layer updates the candidate feature values ​​corresponding to each field in the third candidate matrix according to preset constraint rules to obtain the reconstruction matrix.

6. The method according to claim 5, characterized in that, The structured constraint mapping layer updates the candidate feature values ​​corresponding to each field in the third candidate matrix according to preset constraint rules to obtain the reconstructed matrix, including: For each field, if the field is an integer type field, the Hard-Sigmoid function and the first eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues; if the field is a floating-point type field, the Sigmoid function and the second eigenvalue range corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field to obtain the updated eigenvalues.

7. The method according to any one of claims 1-6, characterized in that, The training process of the VAE model: Obtain the sample feature matrix; The encoder of the VAE model to be trained is used to encode the sample feature matrix and output the sample latent variables corresponding to the sample feature matrix. The decoder of the VAE model to be trained is used to decode the latent variables of the sample, and an initial reconstruction matrix is ​​output based on the latent variables of the sample. The initial reconstruction matrix is ​​corrected using an inter-field coupling constraint model to obtain the sample reconstruction matrix; Based on the sample reconstruction matrix and the sample feature matrix, the loss value is determined, and the parameters of the VAE model are adjusted according to the loss value.

8. The method according to claim 7, characterized in that, The process of refining the initial reconstruction matrix using an inter-field coupling constraint model to obtain the sample reconstruction matrix includes: The initial reconstruction matrix is ​​input into the inter-field coupling constraint model to obtain the residual matrix output by the inter-field coupling constraint model for correcting the initial reconstruction matrix; The initial reconstruction matrix is ​​corrected based on the residual matrix to obtain the sample reconstruction matrix.

9. The method according to claim 8, characterized in that, The inter-field coupling constraint model includes a first fully connected layer, a second fully connected layer, and an output layer; The process of determining the residual matrix includes: The first fully connected layer processes the initial reconstruction matrix using a preset first weight matrix and activation function to obtain an intermediate matrix; The second fully connected layer processes the intermediate matrix using a preset second weight matrix to obtain the initial residual matrix; The output layer determines the elements in the initial residual matrix whose values ​​exceed a preset range, modifies the values ​​corresponding to the elements to preset values, and outputs the modified initial residual matrix as the residual matrix.

10. A data processing apparatus, characterized in that, The device includes: The processing module is used to process each piece of data to be migrated in the database according to the data type of each field and the corresponding first processing rule, and to obtain the feature vector corresponding to each piece of data; wherein, each piece of data contains multiple sub-data, and each sub-data corresponds to a field; The reconstruction module is used to reconstruct the feature matrix composed of the feature vectors corresponding to each data point in the database using a variational autoencoder (VAE) model, thereby obtaining a reconstruction matrix; wherein each row in the feature matrix corresponds to the feature vector of a data point, and each row in the reconstruction matrix corresponds to the reconstructed feature vector of a data point. The processing module is further configured to process each reconstruction feature vector in the reconstruction matrix according to the data type of each field and the corresponding second processing rule, to obtain the reconstruction data corresponding to each reconstruction feature vector, and to use the reconstruction data as test data for testing.

11. The apparatus according to claim 10, characterized in that, The processing module is specifically used to, for each sub-data in each data, if the data type corresponding to the field where the sub-data is located is numerical data, determine the first mean and the first standard deviation corresponding to the field in the database, determine the difference between the sub-data and the first mean, and determine the first ratio of the difference to the first standard deviation as the feature value of the sub-data. If the data type of the field containing the sub-data is string data, then the string length of the sub-data is determined as the characteristic value of the sub-data. If the data type of the field containing the sub-data is date data, then the sub-data is converted into a timestamp, the second mean and the second standard deviation of the timestamp corresponding to the field in the database are determined, the second difference between the timestamp and the second mean is determined, and the second ratio of the second difference to the second standard deviation is determined as the feature value of the sub-data. Arrange the feature values ​​corresponding to each sub-data of each data according to the set order, and determine the sequence of arrangement as the feature vector corresponding to each data.

12. The apparatus according to claim 10, characterized in that, The VAE model includes an encoder and a decoder; The reconstruction module is specifically used to encode the feature matrix using an encoder, output the latent variables corresponding to the feature matrix, and decode the latent variables using the decoder to obtain the reconstruction matrix.

13. The apparatus according to claim 12, characterized in that, The encoder includes a hidden layer and a potential space layer; The reconstruction module is specifically used to perform dimensionality reduction processing on the feature matrix using the hidden layer to obtain a first candidate matrix; and to perform latent variable calculation on the first candidate matrix using the latent space layer to determine the latent variables.

14. The apparatus according to claim 12, characterized in that, The decoder includes a hidden layer, a raw output layer, and a structured constraint mapping layer; The reconstruction module is specifically used to process the latent variables using the hidden layer to obtain a second candidate matrix; to perform dimensionality increase processing on the second candidate matrix using the original output layer to obtain a third candidate matrix; wherein the dimension of the third candidate matrix is ​​consistent with the dimension of the feature matrix; and to update the candidate feature values ​​corresponding to each field in the third candidate matrix using the structured constraint mapping layer according to preset constraint rules to obtain the reconstruction matrix.

15. The apparatus according to claim 14, characterized in that, The reconstruction module is specifically used to perform a linear transformation on the feature values ​​corresponding to each field. If the field is an integer type field, the Hard-Sigmoid function and the first feature value range corresponding to the field are used to obtain the updated feature values. If the field is a floating-point number, the Sigmoid function and the range of the second eigenvalues ​​corresponding to the field are used to perform a linear transformation on the eigenvalues ​​corresponding to the field, resulting in updated eigenvalues.

16. The apparatus according to any one of claims 10-15, characterized in that, The device further includes: The training module is used to acquire a sample feature matrix; the encoder of the VAE model to be trained encodes the sample feature matrix and outputs the sample latent variables corresponding to the sample feature matrix; the decoder of the VAE model to be trained decodes the sample latent variables and outputs an initial reconstruction matrix constructed based on the sample latent variables; the initial reconstruction matrix is ​​corrected using an inter-field coupling constraint model to obtain a sample reconstruction matrix; based on the sample reconstruction matrix and the sample feature matrix, a loss value is determined, and the parameters of the VAE model are adjusted according to the loss value.

17. The apparatus according to claim 16, characterized in that, The training module is specifically used to input the initial reconstruction matrix into the inter-field coupling constraint model, obtain the residual matrix output by the inter-field coupling constraint model for correcting the initial reconstruction matrix, and correct the initial reconstruction matrix according to the residual matrix to obtain the sample reconstruction matrix.

18. The apparatus according to claim 17, characterized in that, The inter-field coupling constraint model includes a first fully connected layer, a second fully connected layer, and an output layer; The training module is specifically used to process the initial reconstruction matrix using a preset first weight matrix and activation function in the first fully connected layer to obtain an intermediate matrix; and to process the intermediate matrix using a preset second weight matrix in the second fully connected layer to obtain an initial residual matrix. The output layer determines the elements in the initial residual matrix whose values ​​exceed a preset range, modifies the values ​​corresponding to the elements to preset values, and outputs the modified initial residual matrix as the residual matrix.

19. An electronic device, characterized in that, The electronic device includes a processor that executes a computer program stored in a memory to implement the steps of the data processing method as described in any one of claims 1-9.

20. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1-9.

21. A computer program product, characterized in that, The computer program product includes: computer program code, which, when run on a computer, causes the computer to perform the steps of the data processing method as described in any one of claims 1-9.

Images