The training of a model that corresponds to a data set and the generation of a synthetic data set.

Abstract

A method of producing a trained model (machine learning model) that indicates whether data field records of a dataset (real world data) contain sensitive (private) data. The dataset may be converted to synthetic data using the machine learning model. The model may be trained to compare statistical data of the dataset with the synthetic data to assess whether the synthetic data accurately corresponds to the sensitive data. Applying a hash function to the data set, to obtain a hash value, may identify the machine learning model as being associated with the data set. A map may be constructed which identifies relationships between data fields of more than one dataset. The real world dataset may contain Personally Identifiable Information (PII) and / or Material Non-Public Information (MNPI). Producing synthetic data provides a way to anonymise sensitive data and thus prevents leakage of client private data.

Description

This disclosure relates to the producing of a trained model that corresponds to a data set, and further relates to use of the trained model to generate a synthetic data set. Background A production environment is a segregated part of a computer network that is reserved for software systems and data that deal with real-world trades, rather than test data. Often engineers are not able to access the production environment, and instead are confined to work in a non-production environment. Personally Identifiable Information (PH) is the information used to identify an individual, and most institutions are responsible for ensuring that this data is handled and manipulated without breaching the privacy of their clients. Material Non-Public Information (MNPI) is corporate news or information that has not yet been made public and which could also have an impact on a share price. Often sensitive data (e.g., PH and / or MNPI) is not allowed to leave the production environment. Thus, there is a demand for a way for engineers to check whether software systems that work in the non-production environment will also work once installed in the production environment. Regulatory specifications mean that proposed changes (bug fixes or new features) to a software system should be tested before they are applied to the production environment. The testing is often automated and relies on substantial amounts of real-world data to ensure that what worked in the previous version of the software works the same way once the change has been applied - and that only the thing that you expect to change has changed. The data itself is often relational - i.e., there are typically linkages between different kinds of data elements. For example, consider the relationship between a trade and a trader. The trader has a name, location, etc., and the trade might have a ticker symbol, a side (buy / sell), an amount, etc. These are often stored in a database in two separate tables with a linkage between them - indicating that a single trader has made a single trade. This linkage itself is often complex - for example, there could be cardinality in the relationship - a trader may have multiple trades but a trade can only have one trader. Trading systems store their data in databases which have complex schemas which describe these entities and their relations. When this data is needed to perform analysis or for testing new software, the actual data set cannot be used as this breaches both local and international personal data regulations. The problem is that this testing happens in a non-production environment, and cannot use production data, because that will contain Pll and / or MNPI. Two solutions have been used in the past: (1) scrubbing production data of Pll and MNPI then exfiltrating it into a non-production environment. This leaves skeletal data with gaps which are to be filled with dummy data. (2) generating random garbage that has the right 'shape' but none of the real-world patterns in the production data. Neither of these solutions correctly maintains either the entity data or the trends in the relationships - for example, trading patterns in North America are very different to those in China. It is important to maintain these relationship trends as well as the distribution of data for a given entity. Summary Aspects of the present invention are set out by the claims. Further aspects are also described. As a first aspect, there is provided a method (S20) of producing a trained model (130) that corresponds to a data set. The method comprises: importing (S21; S112'a; S112"a) the data set, wherein the data set has a number of records, with each record specifying at least one data field; determining (S22) whether the data field contains sensitive data; training (S24; S112'b; S112"b) a model (130) to indicate whether the data field contains sensitive data; and exporting (S26; S112'c-d; S112"c) the trained model (130). Advantageously, the trained model allows the sensitive data to be replaced by generated data. This will result in a generated data set that has the same shape as the restricted data set. The generated data set can be used by a party that isn't permitted access to the restricted data set. The generated data set can be used to test software that is to be used with the restricted data set after testing of the software has been completed. This allows the software to be designed by engineers who are not permitted access to the restricted data set. Optionally, the method further comprises: training (S24; S112'b; S112"b) the model (130) to indicate that the data field contains a particular category of sensitive data. This ensures that the model identifies what type of sensitive data is to be generated according to the model. Optionally, the method is configured wherein: training (S24; S112'b; S112"b) the model (130) comprises comparing real data of the data set with synthetic data that is to be generated by the model (130) once it has been trained, to assess (S114) whether the synthetic data accurately corresponds to the sensitive data. This assessment allows the quality of the model to be assessed, so that a determination can be made of whether any further training of the model is to be performed. Optionally, the method further comprises: comparing (S112'; S114b) statistical data in a data field of the data set with generated data in a corresponding data field of the generated data set, to obtain (S114c) a proficiency value that determines whether the model (130) accurately captures a shape of the data set. The proficiency value quantifies the quality of the model, so that it can be established whether the model has been sufficiently trained to produce a synthetic data set that has characteristics that correspond to the restricted data set. Optionally, the method further comprises: constructing (S112") a map which identifies relationships between data fields of more than one data set. As a result, the model can be used to generate data that is to replace any sensitive data that is detected in a number of data sets which are related to one another. Optionally, the method further comprises: applying a hash function to the data set, to obtain (S113) a hash value that identifies the model (130) as being associated with the data set. The hash value can be used identify the model. This allows the trained model to be identified and used in the event that a further generated data set is to be created. This saves computer resources, by avoiding the training of another model. Optionally, the method further comprises: reading (S23; S113) the hash value, to determine whether the model (130) has already been trained on the data set. If a model has already be trained on the data set, then a number of replacement data sets can be generated without training another model. The models may be stored centrally, and used by several different parties who have been authorised to generate replacement data sets. Optionally, the method further comprises: determining (S22; S115) whether the data field contains sensitive data comprises scanning the data set and identifying whether a heading of the data set is likely to be sensitive data. As a result, a determination can be made of the probability that the data field contains sensitive data. This determination may establish a type of sensitive data that is contained in the data field, which allows the model to be trained that can be used to generate a replacement data set. As a second aspect, there is provided a method (S30) of generating a synthetic data set, the method comprising: loading (S31) a trained model (130) that has been produced by a method according to the first aspect; generating (S32) replacement data based on the trained model (130); determining (S32) whether the trained model (130) indicates that at least one data field of the data set contains sensitive data; and replacing (S32) any data field of the data set with the replacement data that has been generated, if it is determined that that data field contains sensitive data. As a result, a restricted data set can be replaced by a synthetic data set. As a result, a synthetic data set can be used which has characteristics of the restricted data set. A party that isn't permitted access to the restricted data set may instead by authorised to access the synthetic data set. Alternatively, access may be given to the model, in which case the party may generate their own cope of the synthetic data. Optionally, the method further comprises: reading (S117) the model (130) to identify a category of sensitive data that is to be generated as synthetic data. As a result, the model can be used to generate synthetic data of the type that is contained in the restricted data set. Optionally, the method further comprises: reading (S117) the model (130) to identify a type of distribution that is to be generated as synthetic data. As a result, the distribution of synthetic data has similar characteristics to the distribution of the restricted data set. For example, the 5synthetic data may have a mean and standard deviation that is consistent with the restricted data set. Optionally, the method further comprises: determining (S117b) whether there are any relationships between different categories of sensitive data that are to be generated. This allows different data fields of the data set to be generated so that they have a corresponding relationship. For example, there is often a link between a name data field and an email data field. Thus, the model ensures that the generated data set preserves any links between different data fields of the data set. As a third aspect, there is provided a program which, when executed by a computer, causes the computer to perform a method according to the first aspect or the second aspect. As a fourth aspect, there is provided a computer-readable storage medium storing a program according to the third aspect. A problem addressed by this innovation is to prevent the leakage of private client data. To achieve this, a solution is provided that doesn't leak client private data, by producing a model which can be exfiltrated from a production environment for data generation. An advantage of this solution is that synthetic data is generated randomly. This is achieved by providing a model that is trained on a source dataset in order to generate target datasets, such that the target dataset has a similar shape and correlates to the source dataset. Technical effects of this innovation include the generation of synthetic data that correlates to real data, while maintaining the privacy of the real data. A synergistic technical effect is that the resulting synthetic data correlates directly to the real data and has the same shape when evaluated. The technical effects of this innovation extend to processes that are external to a computer, by enhancing the capability of the computer to generate synthetic data. Thus, this innovation confers a technical effect of providing a way to anonymise sensitive data. This innovation discloses software that can take sensitive data within the production environment of a computer, and produces a machine learning model which can be exported out of the production environment to produce anonymous data which can be used anywhere on the computer. Thus, this software is designed with the specific partitions / environments of a computer in mind and thus produces a further technical effect. This provides a computer program that has been designed based on specific technical considerations of the internal functioning of the computer on which it is to be executed, such as by being adapted to the specific architecture of the computer. Brief Description of the Drawings Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: • FIGs. 1A-1D provide schematic diagrams of a system configured to execute an application, with: o FIG. 1A illustrating the system, o FIG. IB illustrating a data agent of the system, o FIG. IC illustrating a command line tool of the system, and o FIG. ID illustrating a user interface of the system; • FIGs. 2A-2C provide flow diagrams showing the execution of the application, with: o FIG. 2A illustrating a method implemented by the entire application, o FIG. 2B illustrating details of the method implemented by a data agent, and o FIG. 2C illustrating details of the method implemented by the command line tool and the user interface; • FIGs. 3A-3H provide flow diagrams showing how sub-systems of the system execute the application, with: o FIG. 3A illustrating a method of performing a data generation service, o FIG. 3B illustrating a method of performing a personally identifiable information (PH) generation service, o FIG. 3C illustrating a method of performing a base extraction service, o FIG. 3D illustrating a method of performing a model service for single table generation, o FIG. 3E illustrating a method of performing a model service for multi table generation, o FIG. 3F illustrating a method of performing a quality assessment service, o FIG. 3G illustrating a method of performing a hashing service, and o FIG. 3H illustrating a method of performing a preselection service; • FIGs. 4A-4G provide screen shots of the user interface of the application, with: o FIG. 4A illustrating an entry page of the application, o FIG. 4B illustrating a data set that includes a single table, o FIG. 4C illustrating a window that includes column matching results which have been extracted from the data set, o FIG. 4D illustrating the data set together with the window that includes column matching results which have been extracted from the data set, o FIG. 4E illustrating a window for exporting anonymised data that is based on the column matching results which have been extracted from the dataset, o FIG. 4F illustrating a window that includes column matching results which have been extracted from a data set that includes multiple tables, and o FIG. 4G illustrating a window that includes a Data Analysis tab; and. • FIG. 5 is a bar chart which illustrates a frequency comparison of gender column of real data versus synthetic data.. Detailed Description Various exemplary embodiments, features, and aspects are described in detail below with reference to the drawings. Disclosure is provided of technical specifications of a Synthetic Data Generator application. The Synthetic Data Generator is a machine-learning project that allows users to generate synthetic data that closely resembles actual user data. This means that both actual and synthesized data should have similar distributions when visualized. A goal of the disclosed Synthetic Data Generator (which may be referred to as the AnZ Synthetic Data Generator) is to use generative models to produce data that provides similar insights as would be derived if the original user data set is analyzed. After completion, this application will allow users to reduce privacy-related risks and meet internationally regulated data anonymization standards by anonymizing such sensitive data. FIGs. 1A-1D provide schematic diagrams of a system 100 configured to execute an application. FIGs. 2A-2C provide flow diagrams showing how the system 100 executes the application. FIGs. 3A-3H provide flow diagrams showing how sub-systems (116,117, 118,112', 112”, 114, 113,115) of the system 10 execute the application. FIGs. 4A-4D provide screen shots of the user interface 140 of the application. FIG. 5 provides an example of how a statistical evaluation is conducted to determine the model that is to be trained. A system 100 is described for an example in which personally identifiable information (Pll) is modelled, and replaced by generated data. However, it will be understood that this example can be generalised to the modelling of other type of sensitive data that is to be replaced by generated data. The term sensitive data may refer to any information that is to be kept confidential, such as private information, business sensitive data, financial records, material non-public information (MNPI), etc. Product FIG. 1A demonstrates that sub-systems of the system 100 include a data agent 100a (FIG. IB), a command line tool 100b (FIG. IC), and a user interface 100c (FIG. ID). These sub-systems (lOOa-c) each have a similar architecture, with the same reference numerals being used to illustrate corresponding features. The Synthetic Data Generator Application 100 has two separate components: 1. The Synthetic Data Generator data agent (100a, FIG. IB) for scanning the production data (112-114) and producing the model (130). 2. The Synthetic Data Generator creator (100b, 100c) which reads the model (130) and generatively produces new data (116-118). The Synthetic Data Generator creator may be implemented with a command line tool (100b FIG. IC), or with a user interface tool (100c, FIG. ID), or with a combination of both (100b &100c). The data agent 100a runs in a production environment 110a, whereas the creator (100b, 100c) runs in an non-production environment (110b, 110c). This makes it possible to restrict any PH centrally to the production environment 110a, while allowing the creation of data to be distributed to the non-production environment (110b, 110c). A generation service is provided that can generate Pll that is customised. The disclosed generators are reliable and generate data consistent with the real life PI Is The system 100 is provided as a desktop application. This is advantageous over web applications, which are exposed to the web drivers that run the application, and thus are capable of leaking private data to clients. Product Specification FIG. IB illustrates the data agent 100a as a sub-system that is executed by a processor 110a which makes use of data stored by one or more memory device 120 in order to output a machine learning model 130. FIG. IC illustrates the command line tool 100b as a sub-system that is executed by a processor 110b which makes use of data stored by one or more memory device 120, and the machine learning model 130, in order to export data to one or more memory device 120. FIG. ID illustrates the user interface 100c as a sub-system that is executed by a processor 110c which makes use of data stored by one or more memory device 120, the machine learning model 130, and user input provided by a user interface 140, in order to export data to one or more memory device 120. Each of these sub-systems (100a, 100b, 100c) is executed by a processor (110a, 110b, 110c). A single processor 110 may implement this innovation on a single server. However, it's also possible for the system 100 to be executed by separate processors (110a, 110b, 110c) that are hosted by different servers. An advantage of the production environment 110a of the data agent 100a being executed by an on premise server is that sensitive information can be kept physically secure by restricting physical access. Storing the Pll in a secure location reduces the risk of this data being stolen. With this arrangement, authorised users are restricted to accessing the Pll via desktop application, without any Pll being transmitted over the internet. The described innovation is not a Software As A Service (SaaS) solution, and hence would not be hosted or run on by a central provider, as is Gretel Al. Instead, the application is self-hosted on the servers of the client who is deploying the solutions. For example, application may be self-hosted on an on a server which is managed by the client (which could be an on premise server or part of a cloud service). The use of a desktop application that is hosted by an on premise server is preferable to a web driver, because web drivers are vulnerable to leaking private data to clients, and so if this solution is adopted for the production environment 100a, mitigation should be prioritised, including routine monitoring for cyber attacks. The non-production environment (110b, 110c) may be implemented by the same on premise server as the production environment 110a. However, the creation of the generated data set may be distributed to other servers, which may be on premise servers or virtual servers which are hosted by a cloud service. Authorised users of the non-production environment (110b, 11c) can access the generated data via a web application. Nevertheless, a desktop application is preferred, because web applications are exposed to the web drivers that run the application. The processors (110a, 110b, 110c) are configured to run software, which FIGs. 1B-1D illustrate as being written in Python, although could instead be created using any high-level programming language. The memory 120 includes a first memory 121 (e.g., storing executable files), and a second memory 122 (e.g., storing a database). The import service 111 is shown importing data from the first memory 121a and the second memory 122a. The export service 119 is shown exporting data from the first memory 121b and the second memory 122b. In practice, the first memory 121a and the second memory 122a that are accessed by the import service 111 to import the data are the same first memory 121b and second memory 122b that are accessed by the export service 119 to export the data. However, in principle, the data could be imported to and exported from different parts of the memory 120 (121a, 121b; 122a, 122b). The processors (110a, 110b, 110c) are shown in FIGs. 1B-1D to include an import service 111, a model service 112 (FIGs. 3D &3E), a hashing service 113 (FIG. 3G), a quality assessment service 114 (FIG. 3F), and a preselection service 115 (FIG. 3H). The processors (110b, 110c) of the command line tool 100b and the user interface 100c are shown in FIGs. 1C& ID to further include a data generation service 116 (FIG. 3A), a PH generation service 117 (FIG. 3B), a base extraction service 118 (FIG. 3C), and an export service 119. Agent (100a, FIG. IB) The data agent 100a runs in the production environment 110a and can read production data. It scans the data and using unsupervised learning, produces a model 130 which describes the shape and distribution of the data elements and their relationships but crucially contains none of the data itself. In the model training process, random values of the same data type are generated and the training algorithm possesses a mechanism (this mechanism varies across different models) that inspects this random data and provides feedback on its quality and authenticity. The feedback then guides the generation of new random data that is closer to the real data. This process is repeated until the model 130 has learned the shape of the data. The model 130 can then be exfiltrated from the production environment 110a. Thus, disclosure is provided of a way of training a machine learning model 130 on a given data set, and saving the trained model 130 locally. This allows later use of the trained model 130 without any need for retraining. Creator (100b &100c, FIGs. IC &ID) The creator (100b, 100c) is in the non-production environment (110b, 110c). It reads the model 130, exfiltrated from the production environment 110a, which contains the parameters needed to generate synthetic data such as the distribution types of numeric columns and the strings identified in categorical columns. These parameters correspond to a lower dimensional representation of the real data and to produce new synthetic data, the model 130 uses samples points from the latent space to then generate corresponding samples of synthetic data. Post-processing steps like normalization, noise addition, or any other transformations are applied to the synthetic data to align it with the characteristics of real data. Finally, linkages between the data are also recreated and Plls redacted using internal tooling and datastores of synthetic Pll data. The creator (100b, 100c) doesn't use a library of fake profiles. Instead, the creator (100b, 100c) is configured to generate synthetic data. Disclosure is provided of a non-production environment (110b, 110c) that uses a generative machine learning process to create new data for all entities and to create linkages between them that follow the correct distribution. Technical Details The synthetic data agent 110a solves the issue of data leaks from production environments 110a. Run as a command line application, it acts on production data to produce a serialized machine-learning model 130 that contains no traces of the production data, hence the model 130 is safe for export to other environments (110b, 110c) where it is used by the desktop application (100c, FIG. ID) or command line application (100b, FIG. IC) to produce synthetic data. A smart preselection service 115 scans the input data and identifies sensitive attributes based on configurations defined in the application. These configurations may be based on a variety of personal identifiers - e.g.: name, age (date of birth), job, country, address, phone number, personal national identification number, education, gender, and email address. Further personal identifiers may also be supported. Common Pll column names are matched against columns in the dataset using a rule-based approach to extract the personal identifiers that these columns represent. Also, a machine-learning algorithm is used to study the information in the columns and using Natural Language Processing match them to recognised Pll fields. For single-table data (S112', FIG. 3D), the TVAE (Triplet-based Variational Auto-Encoder) and CTGAN (Conditional Tabular Generative Adversarial Network) models 130 undergo training using the dataset. Using SDMetrics (Synthetic Data Metrics), a statistical evaluation is conducted to determine the model 130 that exhibits superior proficiency in accurately capturing the shape of the data (the mean and standard deviation of the numeric columns, and the distribution of the categorical variables). The determination of superior proficiency is done by comparing the statistical features of the data in the columns of the real data to the corresponding columns of synthetic data. For example, comparing the distributions of categorical data in these columns using a frequency comparison test (e.g., a chi-squared test). FIG. 5 illustrates an example of a statistical evaluation being performed in the case of a gender column - there are male, female and other present, hence the quality assessment service will compare the frequencies of these three gender categories in the real data and the synthetic data generated. The more similar the frequency distributions, the higher the quality of synthetic data. In the case of multi-tabular generation (S112", FIG. 3E), only the HMA1 (Hiro Metaverse Acquisitions I) model is used, hence there is no further testing to select the better performing model. In the case of relational data, the interconnections among tables are recursively traced and mapped to construct a relationship map. This map serves as a valuable resource for the HMA1 model, facilitating its comprehensive comprehension of the intricate dependencies present across all tables and the overall shape of the data. This process guarantees the generation of data that is consistent across all tables. For non-normalized tables, duplicate columns are identified by comparing overlaps across parent and child tables. After identification, the columns are constrained to make sure that values are consistent across parent and child tables. To expedite the training process, the dataset is divided into smaller, manageable chunks, which are treated as individual tables. This approach offers an additional advantage: by considering each chunk as a separate table, it becomes possible to determine which model between the CTGAN and TVAE performs better on different chunks. Consequently, when the results from these models are combined, the overall quality of the data is greatly improved as compared to training a single model on the entire dataset. To maintain coherence in generated data, Pll interdependencies are extracted and analyzed. These interdependencies serve as a guide for synthesizing PI Is in a way that maintains their inherent relationships. For example, when generating emails, the dependency between a person's name and their email address is considered. By considering such interdependencies, the process of Pll synthesis maintains the logical and meaningful connections among different pieces of information, resulting in coherent and realistic generated data. To uniquely identify serialized models and avoid unnecessary retraining for datasets that have already been processed, a dataset specific hashing technique is employed. A hash function is applied to the dataset, producing a unique hash value identifying the serialized models associated with the dataset under consideration. Before each data generation request, the application checks if a serialized model already exists corresponding to that dataset, otherwise a new model is trained and its hash is computed and stored. Relational Integrity Constraints are maintained by retrieving the database metadata, ensuring that data relational integrity is maintained during export operations, making the synthetic data highly substitutable with non-synthetic input data. Implementation of the Synthetic Data Generator Application FIGs. 2A-2C provide details of the method that is implemented by the application. FIG. 2A illustrates a method S10 of the application that is implemented by the system 100 as a whole (FIG. 1A). FIG. 2B illustrates details of the method S20 that is implemented by the data agent 100a (FIG. IB), and FIG. 2C illustrates details of the method S30 that is implemented by the command line tool 100b (FIG. IC) and / or the user interface 100c (FIG. ID). FIG. 2B is a flow chart that demonstrates how the model is trained by the data agent 110a. In step S21, the data agent 100a imports a data set that is to be modelled. This task is performed by the import service 111, with the data being imported from the memory 120 (121a; 121b). In step S22, the data agent 100a identifies sensitive data (e.g., Pll, MNPI, etc) and starts model generation. Sensitive Data includes Personally Identifiable Information (Pll), which is information that is used to identify an individual. Most institutions are responsible for ensuring that this data is handled and manipulated without breaching the privacy of their clients. Material Non-Public Information (MNPI) is corporate news or information that has not yet been made public and which could also have an impact on a share price. For each data field (column) of the data set, a determination is made of whether a data set contains sensitive data, such as PH. If it is determined that the data field does contain sensitive data (e.g., PH), then an assessment is made of the category of sensitive data that it contains (e.g., name, address, email address, date of birth, MNPI). Typically, all data fields of the data set that are found to contain sensitive data will be selected so that the data that is contained in these fields will be replaced by anonymous data. The data set is modified by replacing the sensitive data with anonymous data that has been generated according to S32. In step S23, the data agent 100a determines whether the a model has already been trained for the data set. This is achieved by the hashing service 113 applying a hash function to the data set, thus obtaining a hash value (S113, FIG. 3G). This makes it possible for a determination to be made of whether a model has already been trained that corresponds to the data set. If a model is not already trained on a dataset, then the method progresses to step S24. If a model is already trained on a dataset, then the method progresses to step S26. In step S24, the data agent 100a the data agent 100a trains a model on the dataset, so that the model can create a generated data set that contains synthetic data instead of PH. The model may be trained S24 by an unsupervised machine learning algorithm. Model Training Details: The script will load and analyse the dataset, and train the machine learning model. Training via an unsupervised machine learning algorithm can be achieved by generating random values of the same category of data (e.g, name, date of birth, location, etc.), and the quality assessment service 114 of the training algorithm inspecting this random data and provides feedback on its quality and authenticity. Feedback of the kind shown in FIG. 5 then guides the generation of new random data that is closer to the real data. The unsupervised training involves the data set being split into a training set and a testing set. The training set is used to train the model, and the testing set is used by the quality assessment service 114 to determine whether the model has been accurately trained. If the quality assessment service 114 determines that the model has been accurately trained, then the method progresses to step S25. However, if it is assessed that the model fails to provide a sufficient level of accuracy, then further training is to be performed. When the training has been completed, the model allows the shape of the data set to be inferred. The shape of the data set corresponds to way in which the different data fields of the data set are arranged. However, the model itself doesn't specify any of the PH of the data set. Thus, the model can be shared with another party (such as a non-production environment 100b, 100c), so that the shape of the data can be understood. The model 130 is trained using the data entries that are contained in the data set, and so the model has a shape that corresponds to the data set. The shape of the data set may include the records of the data set being arranged in rows, and the data fields being arranged in columns. The shape of the data set may also include the data fields being arranged in a particular order. The shape of the data set may further include the records being arranged in a particular order (e.g., in alphabetical order corresponding to one or more of the data fields). Examples are provided below of the model being trained S24 for single table generation (S112', FIG. 3D) or multi table generation (S112", FIG. 3E). As part of the training, the quality of the model is assessed by the quality assessment service 114 (S114, FIG. 3F). Any data of the dataset that has been found to be PH is to be kept confidential, and so a hashing service 113 applies a hash function to the data set, thus obtaining a hash value (5113, FIG. 3G). It isn't necessary for the hashing service to encrypt PHs. Instead, PHs to be kept confidential are identified by the preselection service 155, before training of the model begins. Once the model has been trained, the method progresses to step S25. In step S25, the data agent 100a saves the trained model in the memory 120. The method then progresses to step S26. In step S26, the data agent 100a exports the model to the command line interface tool 100b and the graphical user interface 100c. FIG. 2C is a flow chart that demonstrates how the model 130 is used to generated a generated data set, which is performed by one or both of the command line interface tool 100b and the graphical user interphase 100c. In step S31, the command line interface tool 100b and / or the graphical user interphase 100c loads the model that is received from the data agent 100a. In step S32, the command line interface tool 100b and / or the graphical user interphase 100c generates data from the model. As part of step S32, the command line interface tool 100b and / or the graphical user interphase 100c performs an assessment of whether anonymisation of the data has been requested by the data agent 100a in step S22. As part of step S32, if the anonymisation of the data has been requested, then the command line interface tool 100b and / or the graphical user interphase 100c anonymises the data set, by using the data generation service 116 to generate synthetic data (S116, FIG. 3A). This is achieved by the data generation service 116 calling (S116d, FIG. 3A) the Pll generation service 117, in order to replace the Pll (S117, FIG. 3B). The Pll generation service calls (S117b, FIG. 3B) the base extraction service 118 to determine whether there are relationships between the data fields that are to be replaced (S118, FIG. 3C). As an example, the base extraction service 118 determines (S118b) whether there is a link between a "name" data field and an "email address" data field (S118, FIG. C). In step S33, the command line interface tool 100b and / or the graphical user interphase 100c generated data is exported. Sub-Systems of the Data Generator Application FIGs. 3A-3H are flow charts that demonstrates execution of sub-systems (116,117,118,112', 112”, 114,113,115) of the system 100. FIG. 3A illustrates a method S116 of performing a data generation service 116. (As shown in FIGs. IC &ID, the data generation service 116 is executed by the command line tool 100b and the user interface 100c). In step S116a, a data generation request is received by the data generation service 116 from the user, with the data generation request being retrieved from a memory 120 (FIGs. IC &ID). For the user interface application 100c, the data generation request may be received by the user from the user interface 140 (FIG. ID). In step S116b, a path to the saved model is received from the model service 120 (e.g., a model service suitable for single table generation 120'; or a model service suitable for multi table generation 120"). In step S116c, the data from the model service 120 is sampled. In step S116d, the Pll generation service 117 is called, to replace the privacy sensitive fields that have been specified by the user. In step S116e, the newly generated data is saved in temporary .csv files. In step S116f, a mapping is returned for the temporary .csv file path and the name of the table. FIG. 3B illustrates a method S117 of performing a personally identifiable information (Pll) generation service 117. The Pll generation service 117 serves to generate sensitive data. In this example, synthetic data is generated to replace personally identifiable information (Pll). In a similar way, other types of synthetic data can be generated to replace other types of sensitive data, such as material non-public information (MNPI). (As shown in FIGs. IC &ID, the Pll generation service 117 is executed by the command line tool 100b and the user interface 100c). In step S117a, the Pll generation service 117 receives columns for data replacements and data from the data generation service 116. The Pll generation service 117 is configured to read the model 130 to identify a category of Pll that is to be generated as synthetic data, and also to identify a type of distribution that is to be generated as synthetic data (e.g., mean, standard deviation, frequency). In step S117b, the base extraction service 118 is called, to determine if there are any relationships between the columns to be data replaced (For example, email depending on name). In step S117c, new data generated from data stores is to replace data in columns that is requested to be data replaced. In step S117d, the Pll generation service 117 returns data with columns containing personal identifiable information (Pll) being redacted. FIG. 3C illustrates a method S118 of performing a base extraction service 118. (As shown in FIGs. IC &ID, the base extraction service 118 is executed by the command line tool 100b and the user interface 100c). In step S118a, columns for data replacements are received from the Pll generation service 117. In step S118b, the base extraction service 118 calls on its map of Pll and their dependent fields (e.g., address depends on country). In step S118c, the columns which depend on other fields for data replacement are returned. FIG. 3D illustrates a method S112' of performing a model service for single table generation 112'. (As shown in FIGs. 1B-1D, the model service 112 is executed by the data agent 100a, the command line tool 100b, and the user interface 100c). In step S112'a, tabular data is received from the import service 111. In step S112'b, TVAE and CTGAN models are trained with data. Steps S112'c-S112'd explain how the data is prepared so that it can be passed S26 to the data generation service 116. In step S112'c, the original data and the synthetic data that the TVAE and CTGAN models produce are analysed by the Quality Assessment Service 114. In step S112'd, the better performing model is selected and saved as a pickle file with a hash-code obtained from the hashing service 113. Accordingly, the model service 112 can train the model S24 by performing S112' a model service for single table generation 112'. This can be achieved by comparing S114b statistical data in a data field of the data set with generated data in a corresponding data field of the generated data set, to obtain S114c a proficiency value that determines whether the model accurately captures a shape of the data set. FIG. 3E illustrates a method S112" of performing a model service for multi table generation 112". (As shown in FIGs. 1B-1D, the model service 112 is executed by the data agent 100a, the command line tool 100b, and the user interface 100c). In step S112"a, a map is received of tubular data that is mapped to the name of the table from the import service 111, and information on inter tabular relationships from the import service 111. In step S112”b, the HMA1 model is trained on tabular map data, and the information on inter-tabular relationships. In step S112"c, the HMAl model is passed to the data generation service 116. Accordingly, the model service 112 can train the model S24 by performing S112" a model service for multi table generation 112". This can be achieved by constructing S112”b a map which identifies relationships between data fields of more than one data set. FIG. 3F illustrates a method S114 of performing a quality assessment service 114. (As shown in FIGs. 1B-1D, the quality assessment service 114 is executed by the data agent 100a, the command line tool 100b, and the user interface 100c). In step S114a, real data and synthetic data are received from the model service 112 (112'; 112"). In step S114b, sdmetrics are used to determine the quality of the synthetic data produced. In step S114c, a quality score is returned to the model service 112 (112'; 112"). Comparing the real data of the data set with synthetic data that is to be generated by the model once it has been trained, an assessment can be made of whether the synthetic data accurately corresponds to the PI I. Statistical data in a data field of the data set is compared (S114b) with generated data in a corresponding data field of the generated data set, in order to obtain (S114c) a proficiency value that determines whether the model accurately captures a shape of the data set. FIG. 3G illustrates a method S113 of performing a hashing service 113. (As shown in FIGs. IBID, the hashing service 113 is executed by the data agent 100a, the command line tool 100b, and the user interface 100c). In step S113a, the hashing service 113 receives from the model service 112 (112'; 112”) tabular data or a map of tabular data to the name of the table. In step S113b, uses a combination of the column names and values to produce a hash for the data. In step S113c, the hashing service 113 returns to the model service 112 (112'; 112”) a hash that identifies the dataset. The hashing service 113 serves to apply S113 a hash function to the data set, to obtain a hash value that identifies the model as being associated with the data set. As a result, this makes it possible for a determination S23 to be made of whether a model has already been trained on the data set. FIG. 3H illustrates a method 5115 of performing a preselection service 115. (As shown in FIGs. 1B-1D, the preselection service 115 is executed by the data agent 100a, the command line tool 100b and the user interface 100c). In step S115a, tabular data is received by the preselection service 115 from the import service 111. In step S115b, the preselection service 115 searches through the column headings in the data for headings that are likely to be privacy sensitive data (e.g., column with heading name). In step S115c, the preselection service 115 returns to the import service 111 a mapping of the columns mapped to the personal identifiable information that they are likely to contain. In step S22, the data agent 100a is configured to determine S115 whether the data field contains personally identifiable information by scanning the data set and identifying whether a heading of the data set is likely to be personally identifiable information. Thus, FIGs. 2A-2C and FIGs. 3D &3E describe how a trained model is provided by the model service 112: • The data set is imported (S21; S112'a; S112”a). The data set has a number of records. Each record of the data set specifies at least one data field. • A determination is made of whether the data field contains PH (S22). • The models are trained using the data set that has been imported (S24; S112'b; S112"b). This training involves determining whether each column of the data set contains PH, and if so, what category of Pll (e.g., name, email address, etc.). • The trained models are exported (S26; S112'c-d; S112"c). User Interface FIGs. 4A-4G provide screen shots of the user interface 140 of the application. The user interface 140 is a graphical user interface (GUI) application that supports data import, data generation, visualization and data export functionality. As shown in FIG. ID, for the described example this user interface 140 may be coded in PySide6, which is a Python module that can be used to create graphical user interfaces, to ensure compatibility with the Qt6 application framework. The user interface 140 may be installed by launching an installer file and following prompts to complete the installation process. When installed, users can run the application from the install location specified. FIG. 4A illustrates a landing page which serves as an entry point of the user interface application 140 for the AnZ Synthetic Data Generator. The user is encouraged to import data with the prompt "Welcome to AnonymiZer! Let's get startedl". Import options include the landing page supporting the importing of data into the application from files and databases. Furthermore, the user is presented with recently accessed datastores. The application supports the importing of data from files 121a and selected relational databases 122a, or the importing of pretrained models 130 that have been generated by a data agent 100a. File types that are supported include CSV. Database flavours that are supported include PostgreSQL, MSSQL, MySQL. A "Connect to Database" button 411 allows the user to select data from a database stored in the memory 122a, with a list of shortcuts 412 allowing the user to quickly navigate to any recently accessed databases. An "Upload tabular file(s)" button 413 allows the user to select data from one or more file stored in the memory 121a, with a list of shortcuts 414 allowing the user to quickly navigate to any recently accessed files. Once the data has been selected, it is retrieved from the memory 120 by the import service 111. An info icon 415 provides the user with information about the application. A "Contact Us" button 416 leads to a view that contains the contact details of the development team. This allows users to find out more about the Synthetic Data Generator, seek technical support, and provide feedback. Selecting the "Connect to Database" button 411 progresses the user interface to the data import window 420 shown in FIG. 4B. After successfully importing, the data synthesis stage is where the user can configure parameters that will determine the size and quality of the data. Some of these are: • Count, which specifies the number of rows to be generated. Generating without a specified count will generate the same number of data as the input data. • Scale factor of generation for multi-tabular relational data. • Pre-selected Plls to be verified using the application's column matching functionalities. FIG. 4B illustrates a data import window 420 for a data set 421 that includes a single table 422, corresponding to the example shown in FIG. 3D, for which the model service is shown for single table generation 112'. The data set 421 can be selected from a list of databases 423 which are presented by a Database tab 424. Once selected, the data set 421 is imported by the import service 111 from the memory 120 (step S21). The data set 421 is then presented as a table 422 in an Input Data tab 425. The Input Data tab 425 allows a number of data sets 421 to be opened via the Database tab 424. Each data set 421 includes a close button 426, which allows that data set 421 to be closed. A Generate Data button 427 can be selected to "AnonymiZe" the data of a selected data set 421, which if selected progresses the user interface 140 to the Anonymize window 430 shown in FIG. 4C. For the example shown, the "churn" data set 421 is selected in the Databases tab 424, and is presented in the Input Data tab 425. The table 422 of the data set 421 shows 5 data entries, each having the following fields: "Customerld", "name", "Job", "Credit Score", "Education", and "Marital Status". A horizontal scroll bar of the table 422 identifies that further columns of data are available to be viewed, and a vertical scroll bar of the table 422 identifies that further data entries are available to be viewed. Thus, a single data set 421 is shown for this example, with analysis of this single data set 421 being shown in FIGs. 4C-4E. A plurality of data sets (451, 452, 453) are shown for another example, with analysis being shown in FIG. 4F. FIG. 4B further shows the data import window 420 including an Analyse Table button 428, which if pressed will perform analysis of the data, and progress to a window as shown in FIG. 4G. FIG. 4C illustrates an Anonymize window 430 of the user interface 140, showing column matching results (431,432, 433) which have been extracted from the single data set 421. The column matching results include the identification of: the number of records 431 of the data set 421; Personally Identifiable Information (PH) 432 that has been detected within the data set 421; and the Column Name 433 that is has been used in the data set 421. An Anonymize button 434 allows the data of the data set 421 to be replaced by generated data instead of the personally identifiable information. Before the Anonymize button 434 is pressed, customisation buttons (435, 436, 437) allow the user to make changes to the data that will be generated. If the user doesn't wish to output a generated data set 441, they can instead press a "close" button 438, in which case the user interface returns to the data import window 420 shown in FIG. 4B. It's possible for columns to be added, so that the generated data set 441 contains additional data, which is achieved by pressing the "Add columns" button 435. Also, each column has a delete button 436, which can be used to delete a column so that it is not included in the generated data set 441. The number of records 431 by default will correspond to the number of records that are detected as being in the data set 421. However, the number of records can be increased or decreased, by pressing up or down arrow buttons 437, as appropriate. An alternative way of changing the number of records that is to be generated is to click the mouse cursor into the box showing the number of records, and then manually entering a number of records that is to be generated. For the example shown, results of the "churn" data set 421 are provided as a consequence of the Generate Data button 427 being selected. The results identify that the number of records 431 is "100". The Personally Identifiable Information (PH) 432 that has been detected are of the following categories: "NAME", "JOB", "EDUCATION", "COUNTRY", and "GENDER". The Column Names 433 that have been identified are: "name", "Job", "Education", "Geography", and "Gender". Pressing the Anonymize button 434 progresses the user interface 140 to output a generated data set 441, with the user interface 140 displaying the export window 440 shown in FIG. 4E. The quality assessment service 114 compares the real Pll data that is to be replaced with the synthetic data that is to be generated by the model once it has been trained. FIG. 5 provides an example of a comparison between the real data and the synthetic data, where a Pll of gender is assessed. Thus, the quality assessment service 114 ensures that training of the model 130 exceeds a threshold level of accuracy, guiding the generation of new random data so that it is as close as possible to the real Pll data that is to be replaced. The accuracy of the model is determined by comparing the statistical features of the data in the columns of the real data to the corresponding columns of synthetic data. For example, comparing the distributions of categorical data in these columns using a frequency comparison test. The accuracy determination described above is used by the quality assessment service 114 during the generation of a model for a single table dataset. This is to guide the selection of the most appropriate training algorithm(s) for chunks of the data. Another accuracy score that corresponds to the overall quality of generated data from the generated multi-tabular or single tabular model is included in a report along with other relevant generation details. FIG. 4D illustrates the data import window 420 of FIG. 4B, overlaid with the Anonymize window 430 of FIG. 4C. Pressing the Generate Data button 427 causes the Anonymize window 430 to pop up. Pressing the cancel button 438 will discard the Anonymize window, and return to the import window 420. Pressing the Anonymize button 434 results in the output of the generated data set 441, with the user interface 140 allowing a comparison to be made between the synthetic data and the original data. Presenting the data input window 420 and the Anonymize window 430 in this way ensures that the user interface 140 is intuitive, so that it can easily be navigated by the user. Customisation of the output allows the user to select PI Is that are to be anonymised, and then start generation of the generated data set (step S22). Once the Anonymize button 434 has been pressed, the data agent 100a generates the machine learning model 130 if it hasn't already been trained (steps S23-S25), and then exports the model (step S26). The command line tool 100b and the user interface tool 100c are configured to load the model (step S31), generate data from the model (S32), and export the data (S33). This results in the user interface 140 progressing to the export window 440 of FIG. 4E. FIG. 4E illustrates an export window 440 of the user interface 140, which can be used for exporting the generated data set 441. Once the data set 441 has been exported, the synthetic data that has been generated can then be used in third party applications. The generated data set 441 is based on the column matching results from the Anonymize window of FIG. 4C, which have been extracted from the data set 441 that was selected from the data import window 420 of FIG. 4B. The user can select a file name for the generated data set 441, which for the example shown is "my_data". By default, the generated data set 441 is stored as a comma-separated values (CSV) file format 442, with the file name being given a ".csv" file extension. The user can navigate a menu 443 to determine where in memory 420 (421b, 422b) the generated data set 441 file is to be stored. The user confirms that the generated data set 441 is to be stored by pressing the "Save" button 444. Alternatively, the user can press the "Cancel" button 445, which allows the user to return to Anonymize window 430 of FIG. 4C to make changes to the data that is to be generated. The export window 440 can be closed by pressing the "close" button 446. FIG. 4F illustrates another Anonymize window 450, which includes column matching results which have been extracted from a data set that includes multiple tables (451, 452, 453). In this case, the model service 112 functions in accordance with the example shown in FIG. 3E, for which the model service is shown for multi table generation 112”. For the example shown, the data set is formed from three tables, which are the "churn" data set 451; the "credit" data set 452; and the "marketing" data set 453. The "churn" data set 451 corresponds to the data set 451 that was described for the example shown in FIGs. 4B-4D for which the data set includes only a single table 421. For each of the tables (451, 452, 453), the Anonymize window 450 shows the extraction of: Personally Identifiable Information (PII) that has been detected within the corresponding data set; and the Column Name that is has been used in the corresponding data set. In the example shown: the "churn" data set 451 has a shape that is corresponds to the shape that was described for the Anonymise window 430 in FIG. 4C; the "credit" data set 452 has a shape that has PH that includes "NAME" and "JOB", with Column Names of "name" and "Job"; and the "marketing" data set 453 has a shape that doesn't identify any of the data as being PH. The Anonymize window includes an Anonymize button 534, which if pressed results in the output of a generated data set 441, with the user interface 140 progressing to display an export window 440 as shown in FIG. 4E. Before the Anonymize button 534 is pressed, customisation buttons (535, 536, 537) allow the user to make changes to the data that will be generated. If the user doesn't wish to output a generated data set 441, they can instead press a "close" button 438, in which case the user interface returns to the data import window 420 shown in FIG. 4B. It's possible for columns to be added, so that the generated data set 441 contains additional data, which is achieved by pressing an "Add columns" button 535. Also, each column has a delete button 536, which can be used to delete a column so that it is not included in the generated data set 441. A scale factor can be modified by pressing up or down arrow buttons 537, as appropriate. The scale factor determines how the number of records compares to the number of records that are contained in each of the tables (451,452, 453). A scale factor 539 determines the number of records that is to be generated. The number of records that is to be generated is the scale factor multiplied by the sum of records that are in each of the individual data sets (451,452, 453). By default, the scale factor 539 has a value of 1.0, in which case the number of records correspond to the sum of records that are in each of the individual data sets (451, 452, 453). An alternative way of changing the scale factor is to click the mouse cursor into the box showing the scale factor 539, and then manually entering the scale factor that is to be applied for the number of records that is to be generated. FIG. 4G illustrates a Data Analysis window 460, which is displayed if the user presses the Analyse Table button 428 in FIG. 4B. This enables the user to analyze the data selected data set. The Data Analysis window provides an Analyze tab 461, which helps visualise the similarities between the input and synthesized datasets using graphs generated. Graphs supported are: Distribution graphs; Principal Component Analysis; Correlation map; Log, mean, and standard deviation of numeric columns. Just like importing the data, users can also export the data by clicking an Export button. The data can be exported to a database (the same database as the input data; or a new database). Alternatively, the data can be exported to a file. The application gives access to the PC's file system so that the file can be saved in any supported file format (see FIG. 4E). The user can continue the anonymization process or close the application to end the process. DATA PRIVACY AND SECURITY CONSIDERATIONS 1. The application was built for desktop use only to ensure sensitive data that clients import into the application is not exposed to browser mechanisms of reporting data back to the browser proprietors. 2. The data agent prevents production data from being exposed in the synthetic data generation process Further Details The disclosed examples can be realised by a computer of a system or apparatus. These examples can be implemented by any device configured to execute instructions, or any dedicated hardware that is capable of carrying out all or a portion of the functionality. Disclosure is provided of hardware (e.g., a processor such as a central processing unit (CPU) or a microprocessor unit (MPU)) configured to read out and executes a program recorded on a memory device to perform the functions of the disclosed examples. For this purpose, the program is provided to the computer for example via a network or from a recording medium of various types serving as the memory device (e.g., a computer-readable medium such as a non-transitory computer-readable medium). The steps of the disclosed methods may be performed in any suitable order, or simultaneously where possible. Although the disclosed subject matter has been described using specific terminology relating to apparatus features and / or method features, it is to be understood that the claimed subject matter is not necessarily limited to the examples disclosed. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and 5 equivalent structures and functions. The advantages disclosed may relate to several of the examples that are disclosed.

Claims

1. A method (S20) of producing a trained model (130) that corresponds to a data set, the method comprising:importing (S21; S112'a; S112"a) the data set, wherein the data set has a number of records, with each record specifying at least one data field;determining (S22) whether the data field contains sensitive data;training (S24; S112'b; S112''b) a model (130) to indicate whether the data field contains sensitive data; andexporting (S26; S112'c-d; S112"c) the trained model (130).

2. The method (S20) of producing the trained model (130) according to claim 1, themethod further comprising:training (S24; S112'b; S112"b) the model (130) to indicate that the data field containsa particular category of sensitive data.

3. The method (S20) of producing the trained model (130) according to claim 1 or claim 2, wherein:training (S24; S112'b; S112"b) the model (130) comprises comparing real data of the data set with synthetic data that is to be generated by the model (130) once it has been trained, to assess (S114) whether the synthetic data accurately corresponds to the sensitive data.

4. The method (S20) of producing the trained model (130) according to any precedingclaim, the method further comprising:comparing (S112'; S114b) statistical data in a data field of the data set with generated data in a corresponding data field of the generated data set, to obtain (S114c) a proficiency value that determines whether the model (130) accurately captures a shape of the data set.5 5. The method (S20) of producing the trained model (130) according to any precedingclaim, the method further comprising:constructing (S112") a map which identifies relationships between data fields of more than one data set.10 6. The method (S20) of producing the trained model (130) according to any precedingclaim, the method further comprising:applying a hash function to the data set, to obtain (S113) a hash value that identifies the model (130) as being associated with the data set.15 7. The method (S20) of producing the trained model (130) according to claim 6, themethod further comprising:reading (S23; S113) the hash value, to determine whetherthe model (130) has already been trained on the data set.20 8. The method (S20) of producing the trained model (130) according to any precedingclaim, wherein:determining (S22; S115) whether the data field contains sensitive data comprises scanning the data set and identifying whether a heading of the data set is likely to be sensitive data.

259. A method (S30) of generating a synthetic data set, the method comprising:loading (S31) a trained model (130) that has been produced by a method according to any preceding claim;generating (S32) replacement data based on the trained model (130);determining (S32) whether the trained model (130) indicates that at least one data field of the data set contains sensitive data; andreplacing (S32) any data field of the data set with the replacement data that has been generated, if it is determined that that data field contains sensitive data.

10. The method (S30) of generating the synthetic data set according to claim 9, themethod further comprising:reading (S117) the model (130) to identify a category of sensitive data that is to begenerated as synthetic data.

11. The method (S30) of generating the synthetic data set according to claim 9 or claim 10, the method further comprising:reading (S117) the model (130) to identify a type of distribution that is to be generated as synthetic data.

12. The method (S30) of generating the synthetic data set according to any one of claims 9 to 11, the method further comprising:determining (S117b) whether there are any relationships between different categories of sensitive data that are to be generated.

13. A program which, when executed by a computer, causes the computer to perform a method according to any preceding claim.

14. A computer-readable storage medium storing a program according to claim 13.

Images