Test data generation method, device, equipment, medium and program product

Abstract

This application provides a method, apparatus, device, medium, and program product for generating test data, which can be applied to the fields of big data, information security, and artificial intelligence. The method includes: sampling and querying a production database based on a target data volume to generate n original data entries; wherein each original data entry includes m fields, where m and n are integers greater than 1; based on each field in each original data entry, extracting sensitive information from each field to obtain sensitive information boundaries and sensitive types; performing data simulation on each field according to the sensitive types to generate simulation data; replacing each field based on the sensitive information boundaries and the simulation data to obtain test data corresponding to each field; and generating a test dataset based on the test data corresponding to the m fields in the n original data entries.

Description

Technical Field

[0001] This application relates to the fields of big data, information security, and artificial intelligence, and more specifically to a method, apparatus, device, medium, and program product for generating test data. Background Technology

[0002] In application testing, the quality of test data directly impacts the reliability of test results. From a data quality perspective, test data should reproduce the characteristics of production environment data to the greatest extent possible. However, due to data privacy and security requirements, production data must be anonymized and modified. How to make test data as close to production data as possible while ensuring data privacy and security is a key challenge in the field of application testing.

[0003] Existing test data generation methods rely on manual labeling of sensitive data types during data anonymization and use preset rules to transform sensitive data. This makes the labeling results susceptible to subjective factors, and the preset rules destroy the distribution characteristics of the original production data. As a result, the test data restoration is low, the data processing scope is small, and the effect is unstable. Summary of the Invention

[0004] In view of the above problems, this application provides a method, apparatus, device, medium and program product for generating test data to improve the fidelity of test data.

[0005] According to a first aspect of this application, a method for generating test data is provided, comprising: sampling and querying a production database based on a target data volume to generate n original data entries; wherein each original data entry includes m field data, where m and n are integers greater than 1; extracting sensitive information from each field data entry to obtain sensitive information boundaries and sensitive types; performing data simulation on each field data entry according to the sensitive types to generate simulation data; replacing each field data entry based on the sensitive information boundaries and the simulation data to obtain test data corresponding to each field data entry; and generating a test dataset based on the test data corresponding to the m field data entries in the n original data entries.

[0006] According to an embodiment of this application, the step of performing data simulation on each field data according to the sensitivity type to generate simulation data includes: vector encoding the sensitivity type to obtain a condition vector; generating diffusion noise data through diffusion model operation based on each field data and the condition vector; discretizing the diffusion noise data to obtain a diffusion embedding vector; reconstructing the diffusion embedding vector into text to obtain a text string; and verifying the text string to obtain the simulation data.

[0007] According to an embodiment of this application, the step of generating diffusion noise data based on each field data and the condition vector through diffusion model calculation includes: initializing noisy data based on each field data and Gaussian random noise; and performing iterative calculations through the diffusion model according to the condition vector, the noisy data, and the diffusion step number to update the noisy data until the number of iterations reaches a preset step value, and using the updated noisy data as the diffusion noise data.

[0008] According to an embodiment of this application, the iterative operation through the diffusion model includes: embedding the conditional vector and the diffusion step number to generate a conditional embedding vector and a step embedding vector; fusing the conditional embedding vector, the step embedding vector, and the current noisy data to obtain fused data; inputting the fused data into a noise prediction network to obtain predicted noise data; and performing noise reduction operation on the current noisy data based on the predicted noise data.

[0009] According to an embodiment of this application, discretizing the diffuse noise data to obtain a diffuse embedding vector includes: searching the diffuse noise data in the full word embedding vector space to obtain k neighboring vectors; k is an integer greater than 1; and performing distance-weighted selection on the k neighboring vectors to obtain the diffuse embedding vector.

[0010] According to an embodiment of this application, verifying the text string to obtain the simulation data includes: verifying the text string based on the verification rules corresponding to the sensitive type; and if the text string passes the verification, using the text string as the simulation data.

[0011] According to an embodiment of this application, the step of extracting sensitive information from each field data to obtain sensitive information boundaries and sensitive types includes: segmenting each field data into words based on a sub-word segmentation strategy to obtain word metadata; encoding and processing the word metadata through a bidirectional encoder representation transformation model to obtain semantic vectors; classifying the semantic vectors through a boundary-aware classification model; wherein the boundary-aware classification model is obtained by training a multilayer perceptron on a historical dataset labeled by a boundary annotation strategy; and integrating the classification results to obtain the sensitive information boundaries and sensitive types.

[0012] According to an embodiment of this application, the step of integrating the classification results to obtain the sensitive information boundary and the sensitive type includes: identifying continuous entity segments of the classification results; determining the sensitive type based on the continuous entity segments; and statistically analyzing the boundaries of the continuous entity segments to obtain the sensitive information boundary.

[0013] According to an embodiment of this application, the training process of the multilayer perceptron includes: labeling historical data with sensitive information based on the boundary labeling strategy to obtain labeled historical data; generating the historical dataset based on the labeled historical data; and training the multilayer perceptron through supervised learning based on the historical dataset.

[0014] A second aspect of this application provides a test data generation apparatus, comprising: a data sampling module for sampling and querying a production database based on a target data volume to generate n original data entries; wherein each original data entry includes m field data, where m and n are integers greater than 1; a sensitive information extraction module for extracting sensitive information from each field data entry to obtain sensitive information boundaries and sensitive types; a simulation data generation module for performing data simulation on each field data entry according to the sensitive types to generate simulation data; a data replacement module for replacing each field data entry based on the sensitive information boundaries and the simulation data to obtain test data corresponding to each field data entry; and a dataset generation module for generating a test dataset based on the test data corresponding to the m field data entries in the n original data entries.

[0015] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0016] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0017] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0018] In the embodiments of this application, a sampling query of the production database is used to quickly obtain the original data. Then, through a data transformation method combining sensitive information extraction, data simulation, and data replacement, each field in the original data is processed to achieve field-level desensitization of the original data, generating desensitized test data that conforms to the characteristics of sensitive data, thus improving the fidelity of the test data. Data replacement based on sensitive information boundaries and simulated data ensures precise processing, achieves automated test data generation without manual intervention, provides stable results, and is cost-effective. Attached Figure Description

[0019] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 The illustration schematically depicts application scenarios of the test data generation method, apparatus, device, medium, and program product according to embodiments of this application;

[0021] Figure 2 A flowchart illustrating a method for generating test data according to an embodiment of this application is shown schematically.

[0022] Figure 3 A simplified flowchart illustrating a method for generating test data according to an embodiment of this application is shown schematically;

[0023] Figure 4 This illustration schematically shows a flowchart of the sensitive information extraction method for generating test data according to an embodiment of this application;

[0024] Figure 5 This illustration schematically shows a data simulation flowchart of a test data generation method according to an embodiment of this application;

[0025] Figure 6 A simplified simulation flowchart of a test data generation method according to an embodiment of this application is illustrated schematically;

[0026] Figure 7 A schematic diagram illustrating the structure of a test data generation apparatus according to an embodiment of this application is shown; and

[0027] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a method for generating test data according to an embodiment of this application. Detailed Implementation

[0028] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0032] In the fintech field, the rigor of application testing directly impacts transaction security, risk control effectiveness, and user asset protection, with the quality of test data being a core prerequisite. Production data in financial scenarios often contains massive amounts of sensitive information, such as user identity information, mobile phone numbers, bank card information, transaction records, and credit history. This data must accurately reproduce real business characteristics during testing, such as simulating credit limit distribution, transaction frequency patterns, and risk level correlations for different customer groups. This ensures that the test results of core modules like risk control models and payment systems accurately reflect actual business scenarios.

[0033] However, financial data also needs to meet data security requirements. Using unprocessed production data directly for testing could lead to data breaches and compliance risks. Therefore, testing in the financial sector requires the use of anonymization and distortion techniques. While preserving the original distribution characteristics of the data (such as amount ranges and transaction frequency distributions), sensitive information is eliminated through masking and distortion. This satisfies the testing requirements for data authenticity while upholding data security and compliance standards.

[0034] To meet data anonymization requirements, rule-based processing and model-based processing are commonly used to generate test data. Rule-based processing: During the data anonymization phase, database fields are manually labeled with privacy attributes to clarify the sensitive data types of each field. In the test data generation phase, the system executes corresponding data transformation strategies based on the labeled sensitive types and preset string processing rules. Model-based processing: Training data is collected from the production environment to build a data generation model, learn data distribution characteristics, deploy the trained model, and output anonymized and highly realistic test data according to the required data types.

[0035] The drawbacks of the rule-based processing method are: 1) It relies entirely on manual labeling of sensitive data types, which is not only costly in terms of manpower, but also easily affected by subjective factors; 2) Using preset rules to transform data will destroy the distribution characteristics of the original production data, resulting in poor performance in some test scenarios related to data distribution, such as the inability to accurately simulate the actual traffic ratio of each process in the production environment during performance testing.

[0036] The drawbacks of the model processing method are: 1) It lacks an effective identification and processing mechanism for embedded sensitive information (such as mobile phone number fragments, address fragments, etc. in long texts); 2) The generative model has inherent defects, including unstable convergence during training, insufficient diversity of generated samples, and poor training and generation effects after adding conditional information.

[0037] Therefore, to address the shortcomings of existing test data generation methods, embodiments of this application provide a test data generation method. This method involves sampling and querying a production database based on a target data volume to generate n original data entries. Each original data entry includes m fields, where m and n are integers greater than 1. Based on each field in each original data entry, sensitive information is extracted to obtain sensitive information boundaries and sensitive types. According to the sensitive types, data simulation is performed on each field to generate simulated data. Based on the sensitive information boundaries and simulated data, data replacement is performed on each field to obtain test data corresponding to each field. Finally, a test dataset is generated based on the test data corresponding to the m fields in the n original data entries. In the embodiments of this application, sampling and querying a production database is used to quickly obtain original data. A data transformation method combining sensitive information extraction, data simulation, and data replacement is used to process each field in the original data, achieving field-level desensitization of the original data and generating desensitized test data that conforms to the characteristics of sensitive data, thereby improving the fidelity of the test data. Data replacement is performed based on sensitive information boundaries and simulation data, with precise processing range. It realizes automated generation of test data without human intervention, with stable results and low cost.

[0038] In the technical solution of this application, the user information (including but not limited to user personal information, user identity information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0039] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0040] Figure 1 The illustration shows an application scenario diagram of the test data generation method, apparatus, device, medium, and program product according to embodiments of this application.

[0041] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0042] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0043] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0044] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0045] It should be noted that the test data generation method provided in this application embodiment can generally be executed by server 105. Correspondingly, the test data generation apparatus provided in this application embodiment can generally be located in server 105. The test data generation method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the test data generation apparatus provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0046] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0047] The following will be based on Figure 1 The described scene, through Figures 2-6 The method for generating test data according to embodiments of this application will be described in detail.

[0048] Figure 2 A flowchart illustrating a method for generating test data according to an embodiment of this application is shown schematically.

[0049] Figure 3 A simplified flowchart illustrating a method for generating test data according to an embodiment of this application is shown.

[0050] like Figure 2 , 3 As shown, the test data generation method of this embodiment includes operations S210 to S250. This test data generation method is not limited to a specific execution subject. The execution subject can be any electronic device, such as a terminal device or a server device, etc. The execution subject can also be any software application or client.

[0051] In operation S210, the production database is sampled and queried based on the target data volume to generate n original data entries; each original data entry includes m fields, where m and n are integers greater than 1.

[0052] Based on the test requirements, configure the target data volume and set the number of data entries to be generated, n. Perform a sampling query from the production database to obtain n original data entries, which will serve as the basis for generating the test data. The production database is a structured database within the production environment, and the original data is structured data stored by entry (such as a user record, a transaction record, etc.).

[0053] In operation S220, based on each field of each piece of original data, sensitive information is extracted from each field to obtain the sensitive information boundaries and sensitive types.

[0054] Since each piece of original data may include field data from multiple fields, a field-level desensitization pipeline processing mechanism can be adopted. Operations S220 to S240 are executed sequentially on each field data in each piece of original data to achieve field-level sensitive information extraction, data simulation, and data replacement, generating test data corresponding to each field data in each piece of original data.

[0055] By extracting sensitive information boundaries and sensitive types from field data, sensitive information boundaries include sensitive information locations (such as start and end positions), and sensitive types include sensitive type information (such as address, bank card number, mobile phone number, etc.).

[0056] In operation S230, data simulation is performed on each field of data according to the sensitivity type to generate simulation data.

[0057] During data simulation, different simulation models can be trained and deployed independently based on different sensitivity types. Simulation models include, but are not limited to, generative adversarial networks, variational autoencoders, autoregressive models, and diffusion models.

[0058] Differentiated sensitive data simulation models can be constructed and independently trained and deployed to address different types of data sensitivity. For example, for identifier-type sensitive data such as user identity information and bank card information, rule-based masking and format preservation can be used, with a focus on learning data structure features during model training to ensure that the simulated data conforms to the verification rules. For numerical sensitive data such as transaction amounts and credit scores, the model needs to be trained using statistical distributions to accurately fit the distribution characteristics of the original data, such as the mean and variance, while perturbation algorithms are used to prevent the leakage of true values.

[0059] Independent training of simulation models can reduce interference from cross-sensitive data types and ensure simulation accuracy. Independent deployment facilitates flexible optimization for different scenarios, which can not only meet the data authenticity requirements of the test scenario, but also strengthen the security management of sensitive data through classification protection.

[0060] In operating S240, based on the sensitive information boundary and simulation data, the data of each field is replaced to obtain the test data corresponding to each field.

[0061] By combining the location index of sensitive information boundaries with simulation data, the corresponding data content in the original field data is precisely replaced. For example, if the original field data is "Zhang San's...", the entity object location index is "{"start position":0," end position":1}", and the text to be replaced is "Li Si", then the processed field data will be "Li Si's...". After all entity objects in the field have been replaced, the final field data is generated and used as test data.

[0062] In operation S250, a test dataset is generated based on the test data corresponding to m fields in n original data.

[0063] After processing each field of data using a field-level desensitization pipeline mechanism, test data corresponding to each field of each original data is generated. Test data corresponding to multiple fields of each original data is then statistically analyzed to obtain the test data for each original data. Finally, the test dataset is obtained by statistically analyzing the test data for multiple original data.

[0064] It is worth noting that the generation of test data relies on the structured storage of data in the database and the processing of massive amounts of data using big data technology. Synthesis tools can be used to align the production database, effectively verifying the performance and consistency of the application system.

[0065] In the embodiments of this application, a sampling query of the production database is used to quickly obtain the original data. Then, through a data transformation method combining sensitive information extraction, data simulation, and data replacement, each field in the original data is processed to achieve field-level desensitization of the original data, generating desensitized test data that conforms to the characteristics of sensitive data, thus improving the fidelity of the test data. Data replacement based on sensitive information boundaries and simulated data ensures precise processing, achieves automated test data generation without manual intervention, provides stable results, and is cost-effective.

[0066] Figure 4 A flowchart illustrating the sensitive information extraction process of a test data generation method according to an embodiment of this application is shown.

[0067] like Figure 4 As shown, in operation S220, based on each field of each piece of original data, sensitive information is extracted from each field to obtain the sensitive information boundary and sensitive type, including operations S410 to S440.

[0068] When operating S410, based on the sub-word segmentation strategy, each field of data is segmented to obtain word metadata.

[0069] The WordPiece strategy first constructs an initial vocabulary from individual characters, then iteratively finds the most frequent adjacent sub-word pairs, merges them into new sub-words, and adds them to the vocabulary, until a preset vocabulary size is reached. Words are split into the longest possible sub-word sequence existing in the vocabulary; unrecorded parts are marked as prefix sub-words with special symbols. The WordPiece strategy offers fine-grained segmentation, facilitating precise identification of sensitive information boundaries in subsequent steps.

[0070] A WordPiece word segmenter is built based on the WordPiece strategy. After the field data is input into the WordPiece word segmenter for word segmentation, it outputs several tokens, including Chinese characters, English words, numbers, and punctuation marks.

[0071] In operation S420, the word metadata is processed by the bidirectional encoder representation transformation model to obtain the semantic vector.

[0072] To balance contextual semantic understanding capabilities with model training costs, a Bidirectional Encoder Representations from Transformers (BERT) model is used as the encoder. Each token is input into the pre-trained Chinese BERT model, which outputs a semantic vector.

[0073] In operation of S430, semantic vectors are classified using a boundary-aware classification model; the boundary-aware classification model is obtained by training a multilayer perceptron on a historical dataset labeled with a boundary labeling strategy.

[0074] Based on historical datasets labeled with boundary annotation strategies, a multilayer perceptron (MLP) is trained for classification. The trained MLP is then used as a boundary-aware classification model, and semantic vectors are input into the boundary-aware classification model to output classification results.

[0075] Boundary annotation strategies include, but are not limited to, the BIO annotation method and the BIOES annotation method. BIO annotation method: The positional relationship of entities in the text is marked by three tags, mainly including three tags: B (Begin, start): indicating the starting position of an entity; I (Inside, inside): indicating the inside of an entity (non-starting position); O (Outside, outside): indicating not belonging to any entity. Through the BIO annotation method, the boundaries, inside, and non-entity parts of entities can be clearly distinguished, providing clear boundary information for subsequent data replacement. For example, the text data is "Zhang's card number is 123". After word segmentation, it is "Zhang, mou, de, ka, hao, shi, 1, 2, 3", and the corresponding BIO category annotation is "B-name, I-name, O, O, O, O, B-card number, I-card number, I-card number".

[0076] BIOES annotation method: New tags are added based on the BIO annotation method to more precisely define the entity boundaries, mainly including five tags: B (Begin, start): the starting position of the entity; I (Inside, inside): the middle part of the entity; O (Outside, outside): the non-entity part; E (End, end): the end position of the entity; S (Single, single): an entity composed of a single character. For example, the text data "Zhang's card number is 123". After word segmentation, it is "Zhang, mou, de, ka, hao, shi, 1, 2, 3", and the corresponding BIOES category annotation is "B-name, E-name, O, O, O, O, B-card number, I-card number, E-card number".

[0077] According to an embodiment of the present application, the training process of the multi-layer perceptron includes: based on the boundary annotation strategy, performing sensitive information annotation on historical data to obtain annotated historical data; generating a historical data set according to the annotated historical data; and training the multi-layer perceptron in a supervised learning manner based on the historical data set.

[0078] For the training of the MLP, a semi-automated data set establishment scheme can be adopted. First, based on rules such as regular expressions, following the boundary annotation strategy, automatic pre-annotation is performed on the historical data of existing services (such as structured sensitive information like phone numbers and address information). Secondly, through manual correction, a historical data set can be constructed to generate a high-quality corpus that complies with data security specifications. Finally, the MLP is trained through supervised learning.

[0079] In an embodiment of the present application, sensitive information is accurately annotated through the boundary annotation strategy, a high-quality annotated historical data set is constructed, and the multi-layer perceptron is trained in a supervised learning manner, improving the accuracy and generalization ability of the multi-layer perceptron training and enhancing the recognition effect and reliability of sensitive information.

[0080] In operation S440, the classification results are integrated to obtain the boundaries of sensitive information and sensitive types.

[0081] In the embodiments of this application, semantic features are accurately captured by word segmentation strategy and BERT model encoding, and classified by boundary-aware classification model, which significantly improves the accuracy of sensitive information boundary identification and the precision of sensitive type judgment, accurately extracts sensitive information, and improves the restoration of test data while ensuring the desensitization effect.

[0082] According to an embodiment of this application, in operation S440, the classification results are integrated to obtain sensitive information boundaries and sensitive types, including identifying continuous entity segments of the classification results; determining sensitive types based on continuous entity segments; and statistically analyzing the boundaries of continuous entity segments to obtain sensitive information boundaries.

[0083] A field's data may contain multiple entity objects. The classification results output by the boundary-aware classification model are parsed to identify continuous entity segments and output an array of entity objects. Each entity object contains sensitive boundary information (start and end position information) and sensitive type information. For example, if the classification result is "B-name, I-name, O, B-address, I-address, I-address, O, O", after parsing, the entity object information output is "[{"type":"name","start position":0,"end position":1},{"type":"address","start position":3,"end position":5}]".

[0084] In the embodiments of this application, by identifying continuous entity fragments and statistically analyzing their boundaries, the boundary locations and sensitive categories of embedded sensitive information (such as bank card number fragments, address fragments, etc. in long texts) are accurately identified, the range of sensitive information is precisely located, the accuracy and efficiency of sensitive information identification are improved, and the compliance and security of test data generation are enhanced.

[0085] Figure 5 A schematic diagram illustrating a data simulation flowchart of a test data generation method according to an embodiment of this application is provided.

[0086] Figure 6 A simplified simulation flowchart of a test data generation method according to an embodiment of this application is illustrated.

[0087] like Figure 5 , 6 As shown, in operation S230, data simulation is performed on each field of data according to the sensitivity type, and the simulation data includes operations S510 to S550.

[0088] When operating S510, sensitive types are vector-encoded to obtain condition vectors.

[0089] The vector encoding method is a 16-bit one-hot encoding of the type number. For example, a name is type 1 and the encoding is 0000000000000001; a mobile phone number is type 3 and the encoding is 0000000000000100. The sensitive information type of the entity object (the sensitive type of the field) is used as a conditional vector input into the diffusion model.

[0090] When operating the S520, diffusion noise data is generated through diffusion model calculations based on each field data and condition vector.

[0091] According to an embodiment of this application, in operation S520, which generates diffused noise data based on each field data and a condition vector through a diffusion model, the following steps are included: initializing noisy data based on each field data and Gaussian random noise; and performing iterative calculations through a diffusion model based on the condition vector, the noisy data, and the number of diffusion steps to update the noisy data until the number of iterations reaches a preset step value, and using the updated noisy data as diffused noise data.

[0092] The diffusion model can employ a denoised diffusion probability model. In the initial iteration, the noisy data is treated as Gaussian random noise with the same shape as the embedding vector corresponding to the entity object. Iterative calculations are performed using the denoised diffusion probability model to update the noisy data. If the preset number of iterations has not been reached, the next iteration begins. Once the preset number of iterations is reached, the iteration stops, and the updated noisy data, i.e., the diffused noise data, is output, thus entering the subsequent discretization process.

[0093] In the embodiments of this application, noisy data is initialized based on Gaussian random noise, and highly realistic diffused noise data is generated by iterative diffusion using a diffusion model, enhancing data diversity and realism, and making the diffused noise data approximate the distribution of real field data. Compared with traditional rule-based data generation methods, generating diffused noise data through a diffusion model has advantages in stability, diversity, and fidelity, resulting in better data generation effects.

[0094] According to an embodiment of this application, iterative computation using a diffusion model includes: embedding the conditional vector and the diffusion step count to generate a conditional embedding vector and a step embedding vector; fusing the conditional embedding vector, the step embedding vector, and the current noisy data to obtain fused data; inputting the fused data into a noise prediction network to obtain predicted noise data; and performing noise reduction computation on the current noisy data based on the predicted noise data.

[0095] In each iteration, firstly, the conditional vector and the number of diffusion steps are embedded to generate conditional embedding vectors and step embedding vectors. Embedding can be achieved by using embedding functions (such as fully connected layers, lookup table operations, etc.) to map discrete diffusion steps or high-dimensional sparse conditional vectors into continuous vector representations of fixed dimensions. Secondly, the conditional embedding vectors and step embedding vectors are summed with the noisy data and input together into a noise prediction network with a U-shaped network structure. Finally, based on the predicted noise data output by the noise prediction network, noise reduction is performed on the noisy data.

[0096] In the embodiments of this application, conditional embedding vectors and step embedding vectors are generated by embedding the conditional vectors and diffusion steps corresponding to sensitive types. Noisy data is then fused with these vectors and combined with a noise prediction network and denoising operations to generate highly realistic simulation data. At the same time, the controllability of sensitive information features is ensured, thereby improving the privacy security and accuracy of the generated simulation data.

[0097] In the S530 operation, the diffuse noise data is discretized to obtain the diffuse embedding vector.

[0098] According to an embodiment of this application, the process of discretizing the diffused noise data in operation S530 to obtain the diffused embedding vector includes: searching the diffused noise data in the full word embedding vector space to obtain k neighboring vectors; k is an integer greater than 1; and performing distance-weighted selection on the k neighboring vectors to obtain the diffused embedding vector.

[0099] The diffusion noise data output by the diffusion model is a continuous embedding vector. In the full word embedding vector space, a k-nearest neighbor search is performed on the current embedding vector output by the diffusion model to obtain the k nearest word embedding vectors, i.e., neighbor vectors. Then, one of these neighbor vectors is randomly selected as the diffusion embedding vector corresponding to the current embedding vector based on a distance-weighted method. The distance-weighted selection process involves calculating the cosine distance between the current embedding vector and each neighbor vector, generating a selection probability distribution based on the cosine distance, and then randomly selecting a neighbor vector as the diffusion embedding vector corresponding to the current embedding vector based on the probability distribution. The continuous embedding vectors of the diffusion noise data are discretized to obtain multiple diffusion embedding vectors. The full word embedding vector space is the value space of the embedding vectors corresponding to the tokens output by the WordPiece segmenter, containing the set of embedding vectors corresponding to all tokens output by the WordPiece segmenter during historical processing.

[0100] In the embodiments of this application, by introducing the randomness of the domain vectors in the full word embedding vector space, the neighboring vectors are selected as diffusion embedding vectors through distance-weighted selection, and the robustness of the data simulation algorithm is improved by selecting from multiple neighboring vectors.

[0101] In the S540 operation, the diffuse embedding vector is used to reconstruct the text, resulting in a text string.

[0102] A pre-defined mapping table (dictionary) is used to establish a correspondence between embedding vectors and text units (such as subwords and characters). The diffusion embedding vectors are then decoded and restored to text strings by looking up the table. The decoding and restoration process is as follows: the text unit that best matches a given diffusion embedding vector is found in the mapping table (this can be done by calculating vector similarity, such as cosine similarity), and then all the matched text units are concatenated in order to form a complete text string.

[0103] By operating the S550, verifying the text string, and obtaining simulation data.

[0104] According to an embodiment of this application, in the process of verifying the text string and obtaining simulation data in operation S530, the text string is verified based on the verification rules corresponding to the sensitive type; and if the text string passes the verification, the text string is used as simulation data.

[0105] For certain categories of structured sensitive information, a verification mechanism is implemented based on verification rules, including address rule verification, bank card number rule verification, and mobile phone number rule verification.

[0106] The generation of simulation data has a certain degree of randomness, and may output data that fails validation. Data that fails validation can be discarded, and that field's data can be used as test data. This is because, when generating test data based on massive amounts of raw data, discarding a single field will not significantly affect the overall validity of the test data results. Alternatively, the simulation of that field's data can be repeated, or the number of retries can be set according to the scenario requirements, or the simulation can terminate after a certain number of test data results are obtained.

[0107] It should be noted that the massive amount of raw data itself contains rich feature distributions. The absence of a single field can be compensated for by the correlation features of other related fields. The generated test data can still retain the core distribution patterns of the original data and meet the requirements for data authenticity and correlation in most scenarios. However, it should be noted that if the field is a core key feature (such as transaction amount in financial data), it may affect the accuracy of specific test scenarios and needs to be judged in conjunction with business requirements.

[0108] In the embodiments of this application, verification is performed based on the verification rules corresponding to the sensitive types to ensure that the generated simulation data conforms to the sensitive type specifications, thereby improving the security and compliance of the simulation data and avoiding the generation of invalid simulation data.

[0109] In the embodiments of this application, based on the condition vector corresponding to the sensitive type, diffusion noise data is generated using a diffusion model to simulate the style of sensitive information categories. After discretization, text reconstruction and verification, realistic simulation data is generated, so that the simulation data restores the sensitive information characteristics of the field data as much as possible, which facilitates effective data desensitization.

[0110] Based on the above-described method for generating test data, this application also provides a device for generating test data. The following will be combined with... Figure 7 The device is described in detail.

[0111] Figure 7 A schematic block diagram of a test data generation apparatus according to an embodiment of this application is shown.

[0112] like Figure 7 As shown, the test data generation device 700 of this embodiment includes a data sampling module 710, a sensitive information extraction module 720, a simulation data generation module 730, a data replacement module 740, and a dataset generation module 750.

[0113] The data sampling module 710 is used to sample and query the production database based on the target data volume to generate n original data entries; wherein each of the above original data entries includes m fields, where m and n are integers greater than 1. In one embodiment, the data sampling module 710 can be used to perform the operation S210 described above, which will not be repeated here.

[0114] The sensitive information extraction module 720 is used to extract sensitive information from each field of each piece of the aforementioned original data, thereby obtaining the sensitive information boundaries and sensitive types. In one embodiment, the sensitive information extraction module 720 can be used to perform the operation S220 described above, which will not be repeated here.

[0115] The simulation data generation module 730 is used to perform data simulation on each of the above-mentioned field data according to the aforementioned sensitivity type, and generate simulation data. In one embodiment, the simulation data generation module 730 can be used to perform the operation S230 described above, which will not be repeated here.

[0116] The data replacement module 740 is used to replace the data in each field based on the aforementioned sensitive information boundary and the aforementioned simulation data, thereby obtaining the test data corresponding to each field. In one embodiment, the data replacement module 740 can be used to perform the operation S240 described above, which will not be repeated here.

[0117] The dataset generation module 750 is used to generate a test dataset based on the test data corresponding to the m fields in the above n original data. In one embodiment, the dataset generation module 750 can be used to perform the operation S250 described above, which will not be repeated here.

[0118] According to an embodiment of this application, the simulation data generation module 730 includes: a vector encoding submodule, used to perform vector encoding on the above-mentioned sensitive types to obtain a condition vector; a diffusion model submodule, used to generate diffusion noise data based on each field data and the above-mentioned condition vector through diffusion model operation; a discretization submodule, used to discretize the above-mentioned diffusion noise data to obtain a diffusion embedding vector; a text reconstruction submodule, used to perform text reconstruction on the above-mentioned diffusion embedding vector to obtain a text string; and a verification submodule, used to verify the above-mentioned text string to obtain the above-mentioned simulation data.

[0119] According to an embodiment of this application, the above-mentioned diffusion model submodule includes: an initialization unit, used to initialize noisy data based on each of the above-mentioned field data and Gaussian random noise; and an iterative calculation unit, used to perform iterative calculations through the above-mentioned diffusion model according to the above-mentioned condition vector, the above-mentioned noisy data and the diffusion step number, so as to update the above-mentioned noisy data until the number of iterations reaches a preset step value, and use the updated noisy data as the above-mentioned diffusion noise data.

[0120] According to an embodiment of this application, the iterative operation unit includes: an embedding processing subunit, used to embed the conditional vector and the diffusion step number respectively to generate a conditional embedding vector and a step embedding vector; a fusion subunit, used to fuse the conditional embedding vector, the step embedding vector and the current noisy data to obtain fused data; a noise prediction subunit, used to input the fused data into a noise prediction network to obtain predicted noise data; and a denoising operation subunit, used to perform denoising operation on the current noisy data based on the predicted noise data.

[0121] According to an embodiment of this application, the discretization submodule includes: a search unit, configured to search the diffused noise data in the full word embedding vector space to obtain k neighboring vectors; k is an integer greater than 1; and a selection unit, configured to perform distance-weighted selection on the k neighboring vectors to obtain the diffused embedding vector.

[0122] According to an embodiment of this application, the above-mentioned verification submodule includes: a rule verification unit, used to verify the above-mentioned text string based on the verification rules corresponding to the above-mentioned sensitive type; and a verification pass unit, used to use the above-mentioned text string as the above-mentioned simulation data if the above-mentioned text string passes the verification.

[0123] According to an embodiment of this application, the aforementioned sensitive information extraction module 720 includes: a word segmentation submodule, used to segment each field data into words based on a sub-word segmentation strategy to obtain word metadata; an encoding processing submodule, used to encode the aforementioned word metadata using a bidirectional encoder representation transformation model to obtain a semantic vector; a classification submodule, used to classify the aforementioned semantic vector using a boundary-aware classification model; wherein the aforementioned boundary-aware classification model is obtained by training a multilayer perceptron on a historical dataset labeled using a boundary labeling strategy; and a data integration submodule, used to integrate the classification results to obtain the aforementioned sensitive information boundary and the aforementioned sensitive type.

[0124] According to an embodiment of this application, the above-mentioned data integration submodule includes: an identification unit for identifying continuous entity segments of the above-mentioned classification results; a sensitive type determination unit for determining the above-mentioned sensitive type based on the above-mentioned continuous entity segments; and a sensitive information boundary determination unit for statistically analyzing the boundaries of the above-mentioned continuous entity segments to obtain the above-mentioned sensitive information boundaries.

[0125] According to an embodiment of this application, the classification submodule includes a model training unit, which is used to annotate historical data with sensitive information based on the boundary annotation strategy to obtain annotated historical data; generate the historical dataset based on the annotated historical data; and train the multilayer perceptron based on the historical dataset through supervised learning.

[0126] According to embodiments of this application, any multiple modules among the data sampling module 710, sensitive information extraction module 720, simulation data generation module 730, data replacement module 740, and dataset generation module 750 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the data sampling module 710, sensitive information extraction module 720, simulation data generation module 730, data replacement module 740, and dataset generation module 750 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the data sampling module 710, sensitive information extraction module 720, simulation data generation module 730, data replacement module 740, and dataset generation module 750 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0127] Figure 8 A block diagram schematically illustrates an electronic device suitable for implementing a method for generating test data according to an embodiment of this application.

[0128] like Figure 8 As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0129] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 902 and / or RAM 903. It should be noted that programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0130] According to embodiments of this application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0131] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0132] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0133] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the test data generation method provided in the embodiments of this application.

[0134] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0135] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0136] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0137] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0139] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A method for generating test data, characterized in that, The method includes: Based on the target data volume, a sample query is performed on the production database to generate n original data entries; each of the original data entries includes m fields, where m and n are integers greater than 1. Based on each field of the original data, sensitive information is extracted from each field to obtain the sensitive information boundary and sensitive type; Based on the sensitivity type, data simulation is performed on each field to generate simulation data; Based on the sensitive information boundaries and the simulation data, each field data is replaced to obtain the test data corresponding to each field data; and A test dataset is generated based on the test data corresponding to the m fields in the n original data.

2. The method according to claim 1, characterized in that, The step of performing data simulation on each field of data according to the sensitivity type to generate simulation data includes: The sensitive types are vector-encoded to obtain condition vectors; Based on each field data and the condition vector, diffusion noise data is generated through diffusion model calculations. Discretize the diffused noise data to obtain the diffused embedding vector; The text is reconstructed using the diffuse embedding vector to obtain a text string; and The text string is verified to obtain the simulation data.

3. The method according to claim 2, characterized in that, The step of generating diffused noise data based on each field data and the condition vector through a diffusion model includes: Initialize noisy data based on the data in each field and Gaussian random noise; and Based on the condition vector, the noisy data, and the number of diffusion steps, iterative calculations are performed using the diffusion model to update the noisy data until the number of iterations reaches a preset value. The updated noisy data is then used as the diffused noise data.

4. The method according to claim 3, characterized in that, The iterative calculation using the diffusion model includes: The conditional vector and the number of diffusion steps are respectively embedded to generate a conditional embedding vector and a step embedding vector; The conditional embedding vector, the step embedding vector, and the current noisy data are fused to obtain fused data; The fused data is input into a noise prediction network to obtain predicted noise data; and Based on the predicted noise data, noise reduction operations are performed on the current noisy data.

5. The method according to claim 2, characterized in that, Discretizing the diffused noise data to obtain the diffused embedding vector includes: In the full word embedding vector space, the diffused noise data is searched to obtain k neighbor vectors; k is an integer greater than 1; and The k nearest neighbor vectors are selected by distance weighting to obtain the diffusion embedding vector.

6. The method according to claim 2, characterized in that, The verification of the text string to obtain the simulation data includes: Based on the validation rules corresponding to the aforementioned sensitive types, the text string is validated; and If the text string passes the validation, the text string will be used as the simulation data.

7. The method according to claim 1, characterized in that, The step of extracting sensitive information from each field of data to obtain sensitive information boundaries and sensitive types includes: Based on the sub-word segmentation strategy, each field data is segmented to obtain word metadata; The word data is processed by a bidirectional encoder to represent the transformation model encoding, resulting in a semantic vector; The semantic vectors are classified using a boundary-aware classification model; wherein the boundary-aware classification model is obtained by training a multilayer perceptron on a historical dataset labeled with a boundary labeling strategy; and The classification results are integrated to obtain the sensitive information boundary and the sensitive type.

8. The method according to claim 7, characterized in that, The step of integrating the classification results to obtain the sensitive information boundary and the sensitive type includes: Identify continuous entity fragments from the classification results; Based on the continuous entity fragments, the sensitivity type is determined; and The boundaries of the continuous entity segments are statistically analyzed to obtain the boundaries of the sensitive information.

9. The method according to claim 7, characterized in that, The training process of the multilayer perceptron includes: Based on the boundary labeling strategy, sensitive information is labeled in historical data to obtain labeled historical data; Based on the labeled historical data, the historical dataset is generated; and The multilayer perceptron is trained using supervised learning based on the historical dataset.

10. A test data generation device, characterized in that, The device includes: The data sampling module is used to sample and query the production database based on the target data volume to generate n original data entries; wherein each original data entry includes m fields, where m and n are integers greater than 1; The sensitive information extraction module is used to extract sensitive information from each field of each piece of original data to obtain the sensitive information boundary and sensitive type. The simulation data generation module is used to perform data simulation on each field data according to the sensitivity type and generate simulation data; A data replacement module is used to replace the data in each field based on the sensitive information boundary and the simulation data to obtain the test data corresponding to each field; and The dataset generation module is used to generate a test dataset based on the test data corresponding to the m fields in the n original data.

11. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 9.

Images