A method for data cleansing, a method for pre-processing healthcare patient data prior to a data cleansing, and a method for post-processing healthcare patient data after data cleansing

Abstract

The present invention relates to a computer implemented method for data cleansing. In the method, a data pre-processing module of a first computing instance (3) obtains an at least two-dimensional un-shuffled data structure (10). The data structure (10) comprises healthcare patient data. The data structure (10) comprises data on a plurality of patient records. Each patient record comprises one or more fields wherein each field has a field type and wherein the patient records have a pre-shuffled order. The data pre-processing module of the first computing device shuffles fields of a first field type by moving fields having the first field types from one patient record to another patient record. Fields with the first field type have a post-shuffled order after the shuffling thereby creating a shuffled data structure(lO). A communications module transmits (5) the shuffled data structure (10) comprising the healthcare patient data over a communications network (7) from the first computing instance to a second computing instance.

Description

[0001] A method for data cleansing, a method for pre-processing healthcare patient data prior to a data cleansing, and a method for post-processing healthcare patient data after data cleansing

[0002] The present invention relates to a method for data cleansing, a method for pre-processing healthcare patient data prior to a data cleansing, and a computer implemented method for post-processing healthcare patient data after data cleansing. Further, the present application also relates to data processing systems, computer programs and computer readable storage mediums comprising instructions for carrying out any one of these methods.

[0003] Healthcare providers, for example hospitals, doctors, or laboratories, collect patient data. The patient data collected by the healthcare providers often comprises health-related data that is typically considered sensitive. Further, the healthcare provider may collect the data in a non-standardized fashion. For example, measurements may include different units, or a diagnosis may be formulated differently while relating to the same condition. Other reasons that a same condition may be of represented by a different set of characters may include abbreviations, typos, and using different languages.

[0004] In order to be useful, e.g. for statistical analysis, machine learning, clinical studies or quality assurance, the data is preferably processed and recorded in a standardized fashion.

[0005] However, data holders, e.g. healthcare providers, often do not have the resources to cleanse, e.g. standardize the data sets, correct errors, etc.

[0006] In the art, it is known to anonymize or randomize data by encrypting the data itself. However, this renders the entire data set unreadable to anyone else and does not allow an external provider to cleanse the data.

[0007] The object of the present invention is to overcome the drawbacks of the prior art. A particular object of the present invention is providing a method of anonymizing and deanonymizing data while still allowing third parties to read and cleanse individual fields of a data set. A first aspect of the present invention relates to a computer implemented method for data cleansing. In the method, a two-dimensional un-shuffled data structure is obtained, e.g. by a data pre-processing module of a first computing instance. The data structure comprises healthcare patient data. An un-shuffled data structure as mentioned herein may refer to the data structure prior to the shuffling described herein. This does not exclude processing prior to the methods disclosed herein including other shuffling methods.

[0008] Data cleansing may refer to changing the content of individual fields of a data set and may involve changing the content of a field (e.g. altering the entries of a field) or even removing the content of a field. Data cleansing may comprise detecting and correcting (or removing) corrupt or inaccurate data from a data set (e.g. a data table). Data cleansing is sometimes also referred to as 'data cleaning'. Data cleansing may comprise to identifying incomplete, incorrect, inaccurate or irrelevant parts of the data and then replacing, modifying, or deleting the dirty or coarse data. Data cleansing may be performed manually, e.g. through a user of a computer, and / or or automatized, e.g. using standardized computer programmed rules.

[0009] Thus, the present invention may relate only to a first party sharing the data structure without receiving a data structure from another party.

[0010] A computing instance as mentioned herein may refer to any computing means. The method may be executed on any computing device e.g. comprising one or more microprocessors and volatile and / or nonvolatile memory. Example computing instances may be a cloud computing system, a personal computer, or a mobile computing device or similar. The computing means may have a communications interface across which data can be transmitted to other computing devices. The communication interface may be wireless or wired.

[0011] A module of the computing instance may refer to one or more software instance running on the computing instance and / or one or more hardware components being part of the computing instance.

[0012] A two-dimensional data structure may include multiple fields having a same (e.g. first, second, etc.) field type. Each field may have (e.g. be attributed to) a field type. Typically, a field type may include fields relating to the same or similar properties, e.g. a same type of information. Two-dimensional may refer to the data structure comprising a plurality of patient records and each patient record includes one or more fields. For example, a first dimension may be a number of healthcare patient data sets, e.g. patient 1, patient 2, etc. or lab results 1, lab results 2, etc. A second dimension may be the different field types, e.g. a diagnosis, a physical parameter of the patient such as weight or height, an in-vitro-diagnostic (IVD) test results, an in-vivo test result, a patient identifier (e.g. patient name or a patient ID number), a sex indication, an age indication, any other health related data, and / or any other personal data. The age indication may be a birthdate, an age, and / or a range thereof. Although the present disclosure is described with reference to healthcare patient data, it may be applied to any other data having a similar, i.e. two-dimensional, structure.

[0013] In the method, at least two of the above-mentioned field types may be shuffled independently.

[0014] The data structure may also be three or more dimensional and may be shuffled across a part or all dimensions.

[0015] After transmission, the third party is able to read the fields of any (preferably all) field types in clear form. The individual fields of the data structure may be unencrypted. Alternatively, the individual fields of the data structure may be encrypted. However, in this case, the third party must have (e.g. receive or generate) a key for decrypting the fields.

[0016] The data structure comprises data on a plurality of patient records. Each patient record comprises one or more fields wherein each field has a field type and wherein the patient records have a pre-shuffled order. The order may in particular refer to the order along a first dimension, i.e. the number of a healthcare patient data set, e.g. patient 1, patient 2, etc. or lab results 1, lab results 2, etc.

[0017] The term pre-shuffled order or pre-shuffled as used herein may refer to the data structure or the order of the patient records therein prior to shuffling according to the independent claims and does not exclude other data processing steps prior to the methods disclosed herein, e.g. other anonymization or shuffling techniques.

[0018] The data pre-processing module of the first computing device shuffles fields of a first field type by moving fields (e.g. the field's content) having the first field types from one patient record to another patient record. Fields with the first field type have a post-shuffled order after the shuffling, thereby creating a shuffled data structure. In the process of shuffling, the fields may be moved from their pre-shuffled or original position, i.e. patient record, to another position, i.e. patient record.

[0019] Shuffling information that allows bringing the data from the post-shuffled order to the preshuffled order may be stored, e.g. within the first computing instance (e.g. a pre-processing module thereof). The shuffling information may be information that describes the shuffling, i.e. how the fields were moved during the shuffling, which may allow determining a reverse shuffling method. The shuffling information may identify a shuffling method such that the shuffling steps may be reversed, i.e. a reversed shuffling method.

[0020] For example, where a shuffling is generated by a permutation a and the shuffling is realized by applying the permutation o n — times (i.e. the shuffling function is crn) the stored information may be

[0021] - the pair (a, n), indicating that a1(where a °o1identity) must be applied n times to reverse the shuffling;

[0022] - the pair (cr1, n) indicating that a1must be applied n times to reverse the shuffling; or

[0023] - , in a case the permutation a is of order k (i.e.k=identity) the pair ( , k-n) indicating that o must be applied k-n times to reverse the shuffling.

[0024] Similar shuffling may be performed for data fields of further field types. The shuffling may allow to bring the data into a form that the different entries of a patient record of the shuffled data structure no longer belong to same patient record, e.g. where a post-shuffling patient record comprises the data of a first field type of a first pre-shuffling patient record in the first field type, the data of a second field type of a second pre-shuffling patient record in the second field type and so on. In this manner, the data in a post-shuffling patient record may not be allocable to a certain patient or patient record and, thus, may be, from a privacy point of view, deemed anonymized and be transferred to be changed (e.g. cleansed) by a third party. While in this form, the shuffled data structure may not be scientifically useful, the data in this form may still be cleansed in a meaningful way. Certain field types, e.g. those which by itself could allow identification, such as names or ID numbers, may be extracted from the data structure before a transmission. The methods and devices disclosed herein may also be considered anonymization or encryption methods and devices as they allow to anonymize resp. encrypt the data.

[0025] A communications module may be used to transmit the shuffled data structure comprising the healthcare patient data, e.g. over a communications network. The transmission may be from the first computing instance to a second computing instance, e.g. wherein the second computing instance is in a first security domain that is different from the second security domain of the second computing instance. For example, the first computing instance is in a first security domain in the control of a first entity (e.g. a laboratory or a hospital) and the second computing instance is in a second security domain managed in the control of a second entity (e.g. a data cleansing provider). For example, at least one firewall may separate the first security domain and the second security domain.

[0026] In particular, the individual fields of the data structure may be transmitted over the communications network in unencrypted (i.e. readable or clear) or in decryptable form (e.g. by additionally sending a key). In other words, the data may be encrypted and decryptable by the third party. If the data structure is sent in decryptable form, the third party may have the decryption key or receive it (e.g. in a separate message). This may enable a cleansing of the data by the third party. If the data were sent in encrypted form, a data cleansing as e.g. described below may not be possible.

[0027] Further, in the transmission, the entire data structure having at least two field types (e.g. two rows) may be transmitted. The method disclosed herein may exclude sending field type (e.g. one column) at a time. A shuffling as disclosed herein may only provide anonymization if multiple columns are shuffled, which prevents a third party from identifying individuals or individual data sets while also allowing the third party to cleanse the data as described herein.

[0028] The shuffled data structure may be provided for cleansing. The healthcare patient data in the shuffled data structure may be changed, e.g. on the second computing instance, creating a changed shuffled data structure. Changing the healthcare patient data may involve cleansing, e.g. standardizing, the healthcare patient data.

[0029] Herein, the data may be changed and in particular cleansed as will be described in more detail above and below. The changed shuffled data structure may be received, for example by the first or a third computing instance, e.g. over the communications network (being the same or a different communications network), from the second computing instance. The third computing instance may be different from the first computing instance and may additionally receive the shuffling information that allows bringing the data from the post-shuffled order to the preshuffled order directly or indirectly from the first computing instance. The reception may be by a communication module, e.g. where the communication module is part of the first resp. third computing instance. The first and third computing instance may be in a same security domain, e.g. separated from the second computing instance by at least a same firewall.

[0030] The shuffling information that allows bringing the data from the post-shuffled order to the pre-shuffled order may be obtained, e.g. by the first or third computing instance (e.g. by a data post-processing module thereof), from a memory.

[0031] Post-processing may in some examples result in a de-anonymization of the healthcare patient data. The methods and devices disclosed herein may also be considered deanonymization or decryption methods and devices as they allow an owner of the log file (akin to an encryption key) to reassemble the data.

[0032] The changed shuffled data structure is un-shuffled, e.g. by the first or third computing instance un-shuffles (e.g. using the data post-processing module), by moving fields having the same field type from one patient record to another patient record according to the information that allows bringing the changed data structure from the post-shuffled order to the pre-shuffled order creating a changed shuffled data structure. The information that allows bringing the changed data structure from the post-shuffled order to the pre-shuffled order may be stored, e.g. in a log file.

[0033] A further aspect of the present invention relates to a computer implemented method for pre-processing healthcare patient data in particular prior to a data cleansing. In the method, a data pre-processing module obtains a two-dimensional data structure comprising healthcare patient data. The data structure comprises data on a plurality of patient records. Each patient record comprises one or more fields. Each field has a field type, and the patient records have a pre-shuffled order. While the present disclosure shows examples with two-dimensional data structures, the present invention is not limited to two-dimensional data structures and may also relate to three, four or more-dimensional data structures.

[0034] The fields of a first field type are shuffled, e.g. by the data pre-processing module, by moving fields having the first field type from one patient record to another patient record. The fields with the first field type have a first post-shuffled order after the shuffling, thereby creating a shuffled data structure. A shuffling step (also shuffling round) may involve changing the position of some, most or all fields of the first field type. In some examples all fields or at least 90%, 95%, or 99% of all fields of fields having the first field type are moved at least once during the shuffling.

[0035] Shuffling information that allows bring the data from the post-shuffled order to the preshuffled order are stored, e.g. by using the data pre-processing module. This may include creating a log file in which the shuffling information is stored. The shuffling information forms a key that allows bringing the data structure into its original order, i.e. the order prior to all shuffling steps.

[0036] The shuffled data structure may be transmitted, e.g. to another computing instance for data cleaning.

[0037] The proposed method allows anonymizing data, e.g. making it more difficult to extract meaningful personal data from the data set. In this sense, the method may function as a form of encryption for content of the dataset as a whole. A third party receiving the healthcare patient data may not be able to draw conclusions about individual patient data sets or patients while still being able to read the individual fields and being able to modify, e.g. correct, the fields. This may allow a third party to format the data structure without allowing the third party to identify an individual patient. Anonymizing may mean preventing, e.g. making it difficult for, third parties to obtain personal information. In particular, within the present scope, anonymizing may mean preventing third parties from identifying which field of a shuffled field type belongs to a particular patient or patient ID.

[0038] A patient as mentioned herein may refer to a person that is taken care of by a medical professional, such as a doctor. Healthcare patient data may refer to data obtained by a medical professional and / or data relating to a health status, reproductive outcomes, causes of death, or quality of life of a patient. Example patient data may include at least one of: diagnosis, physical parameter(s) of the patient such as weight and height, lab results, names, or similar. Particular examples of the healthcare patient data are medical data, biological data, in-vitro diagnostic data (e.g. data relating to a saliva, stool, urine, and / or blood sample), and / or in-vivo data. Each of these may be described in one or more fields that may then be moved, i.e. shuffled.

[0039] In one particular embodiment, data may comprise at least an identifier (e.g. a patient or sample identifier) and at least one of the other patient data field (e.g. any combination of the one or more fields of the patient data mentioned herein).

[0040] A shuffling as described herein may involve shuffling fields of field types that do not include an identifier (e.g. a patient or sample identifier).

[0041] In some embodiments, the method may also be applied to non-healthcare data.

[0042] For example, the two-dimensional data structure may be a table. The table may comprise rows and columns. Each row may include one patient record. The columns may include or indicate the field type of the fields of each patient record. A first column might e.g. comprise an age indication for the patient records, a second column might e.g. comprise a gender indication for the patient records, a third column might e.g. comprise an indication of a first metabolism parameter for the patient records, etc. Of course, the role of the rows and the columns may be exchanged such that each column may include a patient record and the rows include or indicate the field type of the fields of each patient record. In one example the data structure may be a spread sheet, e.g. an Excel Sheet or a Google Sheet. Tables may e.g. be stored as a .csv or .xlsx file. The fields may include data related to a patient. In one example, the fields include healthcare patient data.

[0043] A plurality of patient records may mean three, five, ten, twenty, fifty, one hundred or more patient records, e.g. patients or patient samples. Each patient record may correspond to a single patient, one particular patient sample, one or more samples, or a combination thereof. Each patient sample may also correspond to an individual patient sample, e.g. a blood, stool, urine, or other sample. The data structure may comprise one or more patient records from the same patient, e.g. where a first patient record corresponds to a series of tests on a first sample of that patient and a second patient record corresponds to a series of tests on a second sample of that patient. A set of fields (e.g. of a first, second, third, etc. field type) corresponding to a patient sample may mean that the values in the fields are obtained from that sample, e.g. through measurements of properties of the sample.

[0044] Two or more records or each record may have the same number of fields. In some examples not each field may include data. For example, for some records, e.g. some patients, fields having a certain field type (e.g. height, weight) may not be indicated.

[0045] A field may comprise patient data of a particular type. For example, a field may comprise any one of the above-mentioned patient data.

[0046] Cleaning or cleansing the data may involve harmonizing the data types between fields having the same type. Example field types are diagnosis, physical parameter(s) of the patient such as weight and height, lab results, names or similar as mentioned above.

[0047] An order, e.g. an original, pre-shuffled order, of a data structure may refer to a sequence of patient records or data sets. For example, patient record 1 relates to patient A, patient record 2 relates to patient B, etc. In another example, patient record 1 relates to sample A, patient record 2 relates to sample B, etc. The order of the patient records may correspond to indices of the data structure. The data structure may be an array. E.g. array A may be an at least two-dimensional array having indices i and j, A[i] [j] . The index i may refer to the patient sample of a patient and the index j may refer to a field type. For example, if i=l, patient A is selected, if i=2, patient B is selected, etc. For example, if j=l the field type height is selected, if j= 2 the field type weight is selected, etc. In a specific example, A[l] [1] may refer to the height of patient A and is in order before A[2][l], which is the height of patient B.

[0048] Shuffling may refer to mixing a set or subset of similar data, e.g. data having the same field type, so that the order of the fields is changed. Shuffling may refer to a reorganization of the fields having the same field type. Shuffling may be such that, no field is lost during shuffling. Shuffling may include moving most fields or each field from its original position to another position. Before and after shuffling, each patient record may comprise one field for a certain field type. The field may or may not comprise data. The present disclosure also encompasses that individual fields may indicate an error or simply an indicator that this field does not contain healthcare patient data. Moving a field as mentioned herein may refer to copying the data contained in a data memory and writing the data into another portion of the data memory. Moving a field may involve moving the entire field including the data therein or only parts of it, e.g. only the data (e.g. an entry) within the field, e.g. without a formatting, a data type, etc. Data memory may refer to a memory or a section of a memory storing the patient data. In some examples, the data may be moved by changing a pointer or otherwise as is known in the art.

[0049] A pre-shuffled order refers to an order of patient records or fields of a particular field type prior to a shuffling step or multiple shuffling steps. An order after the shuffling or postshuffled order may refer to an order of patient samples or fields of a particular field type after a shuffling step or multiple shuffling steps. The initial order of the fields may be determined by the order of the patient records.

[0050] A similarity measure between the post-shuffled order of the first field type and the preshuffled order of the first field type may be determined, e.g. by the data pre-processing module. The similarity measure may indicate how different the order of the fields of first field type are after the shuffling compared to before the shuffling. The similarity measure may allow a determination indication whether the shuffling was sufficient to anonymize the data, e.g. to impede the third-party from conclusions about individual patients or individual patient data samples. If the similarity measure indicates that the post-shuffled order and the pre-shuffled order are not similar, no more shuffling may be applied. If the similarity measure indicates that the post-shuffled order and the pre-shuffled order are similar, further shuffling may be performed.

[0051] In one example, the similarity measure may be a share (e.g. percentage) of fields having the first field type that are in the same position before and after shuffling. For example, if a data set includes four fields and, after shuffling, one of four fields is in the same position as prior to shuffling, the similarity measure may be 0.25. In a further example, the similarity measure may be a share (e.g. percentage) of fields having the same or similar value before and after shuffling. The similarity measure may be calculated for each field type respectively. A similarity measure, e.g. an overall similarity measure, may be calculated based on the similarity measure(s) of some or all field types (e.g. in case of a table, some or all columns). This similarity measure may be chosen to be calculated with a low computational effort. The present method may include many shuffling steps and thus a simple calculation of the similarity measure may reduce computational resource consumption.

[0052] According to an example, determining the similarity measure may include determining an entropy of positions of the first post-shuffled order of the fields having the first field type in the patient records. The entropy may be calculated based on the number of patient records. The entropy may be a Shannon entropy. The entropy determination may include determining a distance between positions of fields (e.g. a difference in numbered positions between pre- and post-shuffled positions). Using the entropy is particularly advantageous as it allows for a very accurate measurement of randomness.

[0053] These similarity measures are only examples, and the scope of the present disclosure extends to any suitable similarity measure. The similarity measure may be calculated after each shuffling step. The similarity measure may be regularly calculated (e.g. every two, three or more shuffling steps) or be triggered by an event or timer (e.g. after shuffling for 2 ms, the similarity measure is calculated).

[0054] Where the similarity measure fails to meet a predefined level, further shuffling may be performed.

[0055] According to an example, a subset of fields or patient records may be selected for shuffling, thereby further shuffling the patient records. The subset may, e.g., be selected by the data pre-processing module. A subset as mentioned herein is a number of fields or patient records that is less than the total number of fields or patient records. The data preprocessing module shuffles the fields of the selected patient records by moving fields of the subset having the first field type from their current patient record of the subset to another patient record of the subset. The fields with the first field type have a subsequent postshuffled order after shuffling the subset of the plurality of records.

[0056] The similarity measure between the subsequent post-shuffled order of the first field type and the pre-shuffled order of the first field type may then be determined, e.g. by the data pre-processing module. The similarity measure may also be determined by comparing the order the fields having the first field type prior to shuffling the first subset and after shuffling the first subset. In some examples, the similarity measure may be calculated, e.g. only, for the subset. The data pre-processing module stores shuffling information that allows unshuffling the selected patient records. The shuffling of the subset may start with the order of the fields having the first field type according to the post-shuffled order of the first shuffling step. Storing as mentioned herein may refer to recording information in an electronic storage medium. The present disclosure is not limited to any particular form of storage, and storing as described herein may refer to any recording of information in a data memory.

[0057] Shuffling only a subset in subsequent shuffling steps (i.e. after a first shuffling) may provide for a reduction in memory operations (read / write) and reduce the computation time of the shuffling.

[0058] The shuffling step and the storing step may be repeated if the similarity measure fails to meet a predefined level. The repetition may e.g. be performed by the data pre-processing module. While in some examples, only a subset may be shuffled in subsequent shuffling steps, in other examples, the entire (or substantially the entire) set of fields having the first type is shuffled again.

[0059] A subset may comprise a randomly selected number of patient records.

[0060] Selecting and shuffling a subset may increase the mixing of the entries of the chosen field type (e.g. first, second, third, etc. field type). For example, the subset may exclude a part of the data that already is deemed sufficiently shuffled (e.g. as determined by a similarity measure of that part) and thereby guarantee that part's mixture is not diluted by further shuffling steps.

[0061] For example, the subset may be selected so that the similarity measure of the patient records outside of the selected subset meets the predefined level of the similarity measure and the similarity measure of the patient records in the selected subset fails to meet the predefined level. For example, the selected subset may comprise all fields that are still in their initial position or that still have the same neighboring fields.

[0062] For example, the subset may be randomly selected and then, the random selection may be tested according to one of the similarity measures described above. In a further example, the subset may comprise all fields that are in the same position as prior to shuffling.

[0063] For example, the shuffling step (with a subset or the entire set) is repeated until a similarity between a subsequent post-shuffled order of the first field type and the pre-shuffled order of the first field type meets the predefined level or converges or until a set number of repetitions is reached.

[0064] In case the post-shuffled order meets the predefined level, then the shuffling may be deemed to have sufficiently mixed the fields relating to the first field type, which may amount to an anonymization or a step toward anonymization of the data set. In case the mixing is deemed to be sufficient, e.g. by surpassing a predefined level of a similarity measure, further shuffling step may not result in a significant further decrease in similarity and the shuffling may be stopped. This avoids spending computational resources on shuffling steps that do not result in an improvement in anonymization. In some examples the similarity measure may converge which may indicate that pre-defined level of the similarity measure is met.

[0065] During one or more of the repetitions, a second different subset of the patient records may be selected. In one example, during each repetition, a further subset of the patient records of the first subset is selected. Shuffling different subsets may result in an increase in entropy and / or a decrease in similarity measure.

[0066] Storing the shuffling step information may include at least one of: storing shuffling information configured to retrace the shuffling of the selected subset of patient data and storing shuffling information configured to retrace all shuffling steps on the data structure. For example, each individual shuffling step and corresponding shuffling parameters may be identified. The shuffling may use a permutation function. The information may indicate a certain further shuffling operation that may result in the original order without having to retrace all or any shuffling steps.

[0067] The storing may involve a volatile or non-volatile storage. For example, during shuffling (e.g. prior to transmission) the information may be partly or entirely stored in volatile or temporary memory. The method may involve, e.g. only, storing the information of all shuffling steps in a non-transitory memory after all shuffling steps or complete. In another example, the shuffling information may be, e.g. exclusively stored in non-volatile memory.

[0068] Shuffling fields of the first type and / or shuffling a subset of fields having the first field type may comprise randomly selecting or randomly generating, e.g. with the data pre-processing, module a seed value. A permutation function may be applied to the fields within each of the one or more patient records based on the seed value, e.g. by the data pre-processing module. This may mean that the seed value determines how the fields having the first type are moved in between each other. For example, where the permutation is a and the seed value is n, the shuffling function may be crn.

[0069] Thereby, the fields having the first field type can be efficiently shuffled.

[0070] The step of storing shuffling information may comprise for one or more or all shuffling steps at least one of:

[0071] Storing the number of an iteration step in a log file;

[0072] Storing a unique identifier for each field shuffled during the respective iteration step in the log file;

[0073] Storing the shuffled field type in the log file;

[0074] Storing a shuffling method, in particular a permutation function, in the log file; and

[0075] Storing a seed value for a permutation function used in shuffling the data structure in the log file.

[0076] The shuffling information may allow deducing information that allows bringing the data from the post-shuffled order to the pre-shuffled order. The log file may be stored in a log file memory, e.g. connected to (in particular comprised in) the first computing instance.

[0077] Further, the step of storing shuffling information include a step of storing a link to the shuffled data structure. For example, the log file storing the shuffling information may comprise a link to the shuffled structure.

[0078] The data structure may be a table. The table may comprise rows and columns. Each of the patient records may form a row (or, respectively, a column) of the table and each field type may form a column (or, respectively, a row) of the table. This may be a particularly intuitive form for the data structure, in particular for healthcare personnel who are not used to more complex data structures.

[0079] In an example, the data structure comprises at least two different field types, i.e. a first and a second field type. The data pre-processing module may shuffle the data structure by moving fields having a second field type from their pre-shuffled patient record to another patient record. The fields with this second field type have a post-shuffled order after the shuffling. The method may execute any of the steps mentioned above for the first field type for the second field type. Thereby, data sets with multiple different field types may be shuffled. The first field type and the second field type may be shuffled independently of each other. This avoids clustering information together that may allow an identification, e.g. of individual patients or data sets.

[0080] The method may involve shuffling identifying data (e.g. names, IDs, etc) and non-identifying data (e.g. height, weight, etc.). At least the non-identifying data may be shuffled, i.e. the first field type and / or the second field type may relate to non-identifying data. In one embodiment, the data structure comprises both, identifying data (e.g. names, IDs, etc) and non-identifying data (e.g. height, weight, etc.) and all field types of the data structure may be shuffled as described herein.

[0081] In a further example, the data structure comprises one or more further field types (e.g. third, fourth, fifth, etc.). In practice, the data structure may comprise tens (10, 20, 30) or hundreds (100, 200, 300) or thousands (1000, 2000) or more field types, depending on the data set.

[0082] Data sets may also comprise several patient records, i.e., (10, 20, 30) or hundreds (100, 200, 300) or thousands (1000, 2000) or more.

[0083] The data structure may be shuffled, e.g. by the data pre-processing module. Shuffling may include moving the fields having a first, second, third, or further field type from their preshuffle patient record to another patient record within their respective field type, wherein the fields with the first, second, third, type have a post-shuffled order after the shuffling. The first, second, third, or further field type may be any of the field types in the data structure.

[0084] By shuffling the fields only within its field type (e.g. column if the data structure is a table) the cleansing of the data is supported. For example, in the case of a table, where a column of "heart rate per minute" is shuffled in itself, all the values in that column after shuffling will still represent heart rates, so that, at the stage of cleansing, an entry of "100000" might be classified as an error and e.g. be removed from the data structure; whereas an entry of "100000" in "weight in Gramm" might not be classified as an error.

[0085] The data pre-processing module may execute any of the steps described above for the first or second field type for the first, second, third, or further field type, e.g. independently (i.e. each field type is shuffled independently of the shuffling of the other field types). In some examples, some field types are shuffled independently of the other field types and some field types are shuffled together with some other field types.

[0086] The data structure may comprise patient data including a third field type which is intended to be linked to the first field type. Shuffling fields of the first field type may comprise moving fields having the first and third field type from one patient record to a same different patient record such that these two field types are shuffled together. By shuffling two or more field types together, valuable information for the cleansing may be preserved. For example, if one field type includes numerical values (e.g. 20) and another field type includes fields with a corresponding unit (e.g. g / mol) it may be beneficial that they are shuffled together, e.g. where, as part of the cleansing, the unit is to be homogenized.

[0087] In some examples, one or more field types are removed from the data structure, e.g. before shuffling of the data structure and / or before transmission of the shuffled data structure. For example, field type(s) comprising data not intended for cleansing or data that in itself allows an identification of an individual patient (e.g. a patient name) may be removed. Removed data may be stored, e.g. in a memory connected to (in particular of) the first computing instance, and added back to the data structure after receiving the shuffled changed / cleansed data structure. For example, a table comprises a first column the name of the patient, the second column the age, and the third column a cardiovascular diagnosis of the patient; the first column may be removed, the third column shuffled, and the shuffled data structure (comprising the un-shuffled ages and the shuffled diagnoses) may be transferred to a third party for cleansing, the shuffled cleansed data structure is then received after cleansing, the third column is de-shuffled (according to stored information for de-shuffling) and the first column is added back to the data structure; thereby the data's order is restored and the data structure now comprises cleansed entries.

[0088] The data structure comprising healthcare patient data and having patient records with fields in a post-shuffled order may be transmitted, e.g. via communication module over a communications network. The post-shuffled order may be the order after any of the previously mentioned shuffling steps and the shuffled data structure may be deemed sufficiently anonymized (e.g. such that the risk of identifying personal information from the shuffled data structure is low) that it can be sent to a third party for further processing, e.g. cleansing. The third party, having received the shuffled data structure, can read the individual fields but may not be able to make any conclusions about individual patients since relevant field types have been shuffled such that the information within a patient record does not consistently refer to a same original patient record. The communications network may be any known electronic network for transmitting electronic files, e.g. a local area network, or a wide area network such as the Internet. The transmission may be wired and / or wireless.

[0089] According to an example, the method comprises the step of determining a set of field types in the data structure prior to shuffling such that only fields may be shuffled that have a field type does not belong to the set of field types. The set may, e.g. include fields for which shuffling does not increase anonymization. For example, shuffling field types that include a unit may offer little additional anonymization.

[0090] According to an example, the method comprises the step of removing fields having a field type of the determined set of field types from the data structure, e.g. prior to shuffling. Optionally, the method may additionally comprise the step of storing the removed fields separately. For example, the data structure may include the field type "patient name", which may directly allow the identification of a patient or data set. Field types including such sensitive data may be directly removed. The removed information (removed fields) may be stored, e.g. in a data storage connected to (e.g. part of) the first computing instance. Information stored, e.g. within a log file, may identify the determined set of patient data, i.e. the data that may have been removed from the data structure, and e.g. link to a storage comprising the removed information itself.

[0091] According to an example, the data structure is shuffled until the similarity measure of each shuffled field type meets the predefined level or converges or until a set number of repetitions is reached. This may only apply for field types that are shuffled and may exclude non-shuffled field types (e.g. columns). The post-shuffled data set that is transmitted (e.g. over the communications network) may only include field types (e.g. columns) that have been shuffled. The post-shuffled data that is transmitted may only include field types that meet the predefined level of the similarity measure. Thereby, the anonymization of the data structure may be ensured, and the computing instance shuffling the data structure may stop the shuffling when shuffling does not yield any significant further anonymization (e.g. in the convergence or limited number of repetitions case). A further aspect of the present invention relates to a computer implemented method for post-processing healthcare patient data, e.g. after data cleansing. The method is preferably executed after executing the method described above. A shuffled data structure comprising, e.g. cleansed, healthcare patient data may be obtained, e.g. using a data post-processing module. The data structure has a plurality of patient records. The plurality of patient records each comprise one or more fields. Each field has a field type. The fields have a post-shuffled order.

[0092] The post-processing module obtains information that allows bringing the data from the postshuffled order to a pre-shuffled order, e.g. via a log file (e.g. from a data memory). The log file may comprise information that, directly or indirectly, allows bringing the data from the post-shuffled order to a pre-shuffled order. The data structure may be un-shuffled by moving, e.g. by the data post-processing module, fields having the same field type from one patient record to another patient record based on the information that allows bringing the data from the post-shuffled order to the pre-shuffled order, which is e.g. obtained via a log file. The moving being based on information may mean a reversal of the shuffling indicated by the information, e.g. retracing the shuffling steps. Alternatively or additionally, based on may mean that the data post-processing module determines other steps for un-shuffling, e.g. if the shuffling is based on a cyclic permutation.

[0093] A shuffled order as mentioned herein may also, or alternatively, be described as an order in which the patient data does not fit, and / or an order previously shuffled by the preprocessing or post-processing module. The data not fitting may mean that a patient record includes fields that belong to another patient record. The pre-processing module and postprocessing module may be the same module or part of a same module.

[0094] The information that allows bringing the data from the post-shuffled order to the preshuffled order in the log file may include a first shuffling method. The data structure may be un-shuffled by moving fields having the first field type from one patient record to another patient record based on the first shuffling method, e.g. reversing the shuffling steps.

[0095] Thereby, patient records are moved to their original position and the data structure is brought back into its original order.

[0096] The information that allows bringing the data from the post-shuffled order to the preshuffled order may include a second shuffling method. The data post-processing module un- shuffles the data structure by moving fields having the first field type from one patient record to another based on the second shuffling method after un-shuffling the data based on the first shuffling method. The data post-processing module may also un-shuffle the data structure based on a third or a fourth or a further shuffling method.

[0097] According to an example, the data post-processing module un-shuffles the data structure according to the number of shuffling methods that are indicated in the log file.

[0098] Stored information, e.g. a log file, may indicate a subset of the plurality of patient records for any of the shuffling methods. For example, the stored information may indicate a subset of the patient records for the first, second, third and / or further shuffling methods. Thereby, it is known which records have been shuffled which allows un-shuffling the records accordingly.

[0099] A shuffling method as mentioned herein may refer to a shuffle algorithm (e.g. a sort, riffle, and / or overhand algorithm or any other suitable permutation function).

[0100] The data structure may be un-shuffled, e.g. by the data post-processing module, by moving fields having the same field type from one patient record to another patient record based on the information (e.g. retrieved from a log file) that allows bringing the data from the postshuffled order to the pre-shuffled order within a subset (e.g. as indicated in said information) of patient records for the first and / or second and / or third and / or further shuffling methods. For example, the data post-processing module may read the information and retrace the shuffling steps or at least change the order of the fields until the original order is restored based on information that was used to shuffle the data.

[0101] According to an example, the log file additionally indicates a field type for the respective shuffling method. Fields having the field type indicated by the log file for the respective shuffling method may e.g. be those that are un-shuffled, whereby it may be avoided that during un-shuffling a different set of fields is used as compared to the shuffling.

[0102] According to an example, the log file additionally indicates at least one seed value for at least one permutation function for the respective shuffling method according to which the fields having the first field type were shuffled. In some examples, the shuffling method is formed by the seed value combined with a permutation function, e.g. by applying the permutation a number of times indicated by the seed value. A seed value as mentioned herein may refer to a randomly or pseudo-randomly generated number. The seed value may be generated to be higher than a lower threshold, which may allow to exclude too simple shuffling. The seed value may be generated to be lower than an upper threshold, which may allow to limit the computational effort.

[0103] Shuffling different field types may involve using the same permutation function for all field types (e.g. columns), using different seed values for at least two field types. For example, the shuffling may be performed using different seed values for each two shuffled field types.

[0104] Shuffling different field types may involve different permutation functions for some or each field type (e.g. column) and the same or multiple seed values for the different permutation functions.

[0105] The proposed method may comprise the step of cleansing the shuffled data structure. The cleansing may comprise non-automated processing of the data.

[0106] The data structure comprising the patient data and having patient records with fields in a post-shuffled order may be received, e.g. via communications module over a communications network. Thereby, the data structure may be fed back to the healthcare provider for un-shuffling the cleansed data structure.

[0107] A further aspect of the present invention relates to a data processing system comprising means for carrying out the methods described above.

[0108] A further aspect of the present invention relates to a computer program comprising instructions. When the program is executed by a computer, the instructions cause the computer to carry out the methods described above.

[0109] A further aspect of the present invention relates to a computer readable storage medium comprising instructions. The instructions, when executed by a computer, cause the computer to carry out the methods described above. In the following, a non-limiting example of the invention will be described with reference to the following drawings:

[0110] Figure 1 shows a pictorial illustration of a data exchange between a hospital and a third party.

[0111] Figure 2 shows an example data structure according to the present disclosure.

[0112] Figure 3 shows a flow chart illustrating a method of cleansing data.

[0113] Figure 4 shows a graph of the similarity measure for a given data set.

[0114] Figure 5 shows an example log file.

[0115] Figures 1 to 3 show an example use case for the present invention. Figure 1 shows a healthcare provider (here: hospital 1). The hospital 1 collects data on patients. In particular the hospital 1 records names, birthdates (i.e. ages of the patients), weight, height, etc. Further, healthcare professionals (e.g. doctors, nurses) examine the patient before, during, and after treatment. Examination may involve the collection of further data during each examination and / or the collection of diagnosis and physical parameters. Also, the hospital 1 may examine samples (blood, stool, urine, etc.) and determine physical parameters of the patient on the samples, e.g. in an external or internal lab. Such data may also be received from other healthcare providers. The healthcare professionals may in some case not record the data consistently in the same manner. For example, some healthcare professionals may describe a diagnosis as an ICD code, while others may describe the diagnosis explicitly and still others may only describe symptoms. Some healthcare professionals may record the height in cm and others in meters. This results in the data being non-standardized and difficult to use in other applications.

[0116] Unfortunately, hospital 1 usually neither has the resources nor the time to cleanse the data. One example data structure 10 is shown in figure 2. In the example, the data structure 10 has the form of a table. The data structure 10 includes seven columns, i.e. field types, which indicate a patient name, patient ID, weight, a weight unit, height, a height unit, and a diagnosis. The rows indicate four patients (patients 1 to 4). The data set in figure 2 is simplified for ill ustrational purposes. As can be seen, the data set includes problems. The height of patient 4 is given in meters while the height of the other patients is given in cm. Further, the diagnosis (last field type) is given in several different formats. Typical data sets will include many more patients (or patient samples) and may include many more field types. While third parties 2 (see figure 1) such as, e.g. pharmaceutical companies or external service providers, may have the resources to cleanse the data, privacy concerns prevent the hospital 1 from sharing its data with the third parties 2 for cleansing.

[0117] Thus, data should be anonymized before being shared, but there are no anonymization capabilities within the hospital 1. Classic automated anonymization approaches require datasets to be in a certain format, but such dataset cleansing capabilities are also not available within the hospital 1. Third parties can provide required data cleansing capabilities but do not have access to the data structure 10.

[0118] According to the present invention, the hospital 1 that wants to provide data is provided with a shuffling application to be executed in its own computing environment 3. The computing environment may e.g. be server or, as shown in figure 1, be cloud based. In a preparatory first step, the hospital 1 may share a data scheme without sharing the full data structure 10 with the third party 2, e.g. with a computing environment 4 of the third party. The data is shared over a communications network 7 (e.g. the internet), The data scheme may include the field types (e.g. the rows) to the third party 2. The third party 2 may then provide a recommendation, such as which field types are to be shuffled, which field types need to be removed, and which field types are interlinked. For example, the field type "Patient Name" may be removed to limit a risk of identifying individuals within the dataset to be provided. Further, it may be sensible to interlink, i.e. shuffle together, certain field types, e.g. because knowing that the entries are related may be beneficial for a cleansing. For example, interlinking the field types of weight and weight_unit may allow standardizing weight information e.g. to "kilogram". Likewise, it may be beneficial to interlink height and height_unit. Alternatively, the hospital 1 may choose to shuffle all field types and choose which field types to interlink and choose which field types to remove by itself. Then, the data is imported into a shuffling application as described above and below. Alternatively, these initial operations may be executed in the shuffling application.

[0119] The shuffling application may first estimate (e.g. determine) a lower boundary threshold (i.e. a similarity measure). Alternatively, the lower boundary threshold may be set to a fixed value or may be entered by a user. A distribution within the data structure 10 (e.g. the data structure shown in figure 2) affects a level of similarity (lower distance boundary) that is achievable. Thus, the lowest distance threshold may be defined using an entropy-based estimation and used as a measure of how well-spread the values are within a column. The more diverse the values are, the higher the entropy, and the lower the estimated minimum or lowest row-wise similarity. The entropy may be transformed into a similarity score by subtracting it from 1 and normalizing by the log of the number of rows in the data structure 10. Alternatively, entropy may form the similarity score (i.e. similarity measure).

[0120] Alternatively to the entropy-based approach, the similarity measure may also be determined by using correlation methods between rows and columns or frequency-based measures of unique values and estimation the minimum similarity by considering the effect of shuffling on the frequency of unique values in each column.

[0121] If some of the rows include empty fields, such as the column diagnosis, the normalization may be based on the number of non-empty fields for a given field type (column). A mean of the individual column similarity scores is used as the estimate for the minimum row-wise similarity between the original data structure 10 and a shuffled version of it. The estimation of a lowest distance boundary is a useful measurement to set the similarity level. In some cases, the lowest distance boundary can be tuned manually even after the initial estimation.

[0122] Then, the data structure 10 is shuffled (see also figure 3 step 11). A shuffling can be applied to a single column or to multiple related columns at the same time. Though the example is described with reference to a table, i.e. in terms of columns (i.e. field types) and rows (i.e. records), the concepts described herein can be used in connection with any two-dimensional data structure 10. The shuffling may be performed by randomly selecting a seed value (for a permutation function) for each column to be shuffled. Alternatively, the hospital 1 may define a seed manually. The seed value, which is used, can be saved as a string. If multiple columns are shuffled using different seeds, the seed values can be saved as a string of seed values. The seed values also can be saved as a mapping table. The seed values (formatted as a string or mapping table or otherwise) are stored in a log file 20 (see figure 5). Any shuffling algorithm may be used (shuffle (default), sort, riffle, overhand).

[0123] In cases where two columns are interlinked, both columns are shuffled together using the same seed and shuffling method such that the entries of both columns for a given row are shuffled to the same row. The information (seed / shuffling method) related to the shuffling process is saved in the log file 20 (see figure 5). After the shuffling, the shuffled column is compared to the lower boundary threshold (see step 12 in figure 3). The lower boundary threshold may be checked f ield-by-f ield . If a field does not meet the lower boundary threshold, it may be assigned to a subset. In case a column (i.e. all fields of a field type) is below the lower boundary threshold, then shuffling of the respective column is complete and the method can move to the next column that was not shuffled yet and the next column is shuffled in the same manner.

[0124] If the shuffled column is above the lower boundary threshold, then rows (i.e. fields) found to be outside of the similarity threshold may be separated into a subset of data. This subset is then shuffled in isolation from the other fields that meet the similarity threshold. Whenever a data subset is shuffled, it should be replaced at the same position in the data structure 10 it was removed from. If not, it does perturb the seed-based reshuffle. This is, e.g. ensured by adding a primary key value for each row field to uniquely identify each field in the dataset. The primary key values of the shuffled fields are also stored in the log file 20 (see figure 5, column "records").

[0125] Alternatively (or additionally) to shuffling a subset as described above, all fields of a field type may be reshuffled in a second or further shuffling step.

[0126] This process may be repeated until either all fields are within the similarity threshold or further iterations make no to little difference to the number of rows being moved within the threshold. In case of the latter, a randomly selected variable number of rows (which were already within the threshold) are included in the subset to be reshuffled. After reshuffling, the similarity is assessed, and the process is repeated until the algorithm converges (i.e. that all rows are sufficiently shuffled and beneath the threshold). This is when any further shuffling may be stopped. If none of that happens, then the shuffling terminates after a total of e.g. 1000 iterations. This is illustrated in figure 4 which shows how a similarity measure (y- axis) changes with each iteration (x-axis). With each iteration, the similarity measure improves, i.e. decreases. The shuffling may stop after a maximum number of iterations (Imax) or once the pre-defined level (L) is met or if a convergence criterion is met. For every shuffling iteration, seed values for all columns are saved in the log file 20.

[0127] The iterative shuffle algorithm could also be completed with different optimization approaches including different convergence criteria or 'jump-out' options. More specifically, the subset may be selected by selecting random fields of the column. The fields can also be non-random and chosen based on a certain criterion e.g. fields which similarity are furthest away from the similarity threshold or fields that are more likely to reach the threshold. The number of fields selected in the subset could be a set percentage of the total number of rows or based on the number of rows which were within the threshold after the first x number (e.g. after at least 1, 2, 5, 10, etc.) of iterations and the number could also change across each iteration or optimization step.

[0128] Further, one could use other distance measurements or distance calculations (e.g. using Levenshtein distance, or Hamming distance) to check for a sufficient shuffle, i.e. as a similarity measure and as a pre-defined similarity level.

[0129] There could also be a hybrid shuffle approach. E.g. one column can be shuffled and then the other columns are then shuffled based on a specific criterion (or criteria) that is dependent on the first column. E.g. the second column is shuffled such that the shuffled value in the second column is swapped with the most distant value as was there originally (with the value in the first column), disrupting the pairwise correlation.

[0130] While the example uses tabular data, the concept is also applicable to other data forms (e.g. databases including relations between variables / tables that can also be tracked when being changed or shuffled. The present invention applies the shuffling method to healthcare patient data (e.g. clinical or medical). The presently shown approach could also be applied to any other data resources (e.g. finance, energy). The shuffle concept could be implemented and utilized in any product, executable, software, or tool.

[0131] The presently disclosed method enables the data owner (i.e., hospital 1) to manipulate or complete the shuffling and anonymization on its own without a key exchange or any disclosure of the data. The log file needed to un-shuffle the data always remains with the data owner and does not need to be shared. After the data has been shuffled, it can be shared with the third party 2 as indicated by arrow 5 in figures 1 and 3. The data can, e.g. be sent to a computing environment 4 of the third party 2. In some examples, sending or transmitting the data to the third party may simply involve giving the third party 2 access to the shuffled data structure 10 in computing environment 3. Of course, the data may additionally be transferred using any known encryption method. Then, the data structure 10 can be cleansed by the third party 2. For example, the third party may correct the erroneously used unit m (column "Height_unit") and the corresponding measurement (1.882) and the third party may harmonize the diagnosis to ICD codes (e.g. change "COVID-19, virus identified" to "U07.ll".

[0132] After the third party 2 has cleansed the data structure, the shuffled data structure is transmitted back to the hospital 1 as indicated by arrow 6 in figures 1 and 3.

[0133] Then using log file 20, the hospital 1 may retrace the shuffling steps and reverse or unshuffle the data (step 13 in figure 3). Based on the above, the log file 20, e.g. stores any, some or all of the following information (see also figure 5):

[0134] A number of each iteration (first column) that is performed

[0135] Records that are shuffled in this iteration (second column)

[0136] The name of the columns that is shuffled (third column)

[0137] - The method that is used for shuffling (fourth column)

[0138] The seed that is used for this shuffle (fifth column)

[0139] As the log file 20 presents the key for bringing the data into its original order, it should not be shared with the third party 2 (or otherwise) and the log file can stay with the hospital 1.

[0140] In the shown example, un-shuffle process (step 13 in figure 3) starts at the last row of the log file (last iteration) i.e. un-shuffling is in reverse order. During un-shuffling, first, the list of fields shuffled in a given iteration is extracted. Then a data frame including only these fields (or at least the data in these fields) may be created. Using the seed and the shuffling method, the selected fields are un-shuffled, e.g. by retracing the shuffling steps. Then, the extracted fields are re-entered into the full data structure. This process is iteratively repeated for each shuffling step from last shuffling step to the first shuffling step until the top of the log file is reached.

[0141] Additionally, the presently disclosed method may include an evaluation step 14 (see figure 3). The evaluation may include a determination, if the data structure still has the same shape, i.e. a number of columns and rows number, as compared to the original data structure. The evaluation may include if the field types are identical between the original and the cleansed data structure. The evaluation may include a comparison between the columns and rows separately between the original and the cleansed data structure (also data frame herein). For example, the comparison may check if at least some the fields are identical. The evaluation may indicate columns that have not been correctly cleansed and distinguish between incorrect or correctly reshuffled columns.

[0142] Thereafter, the hospital 1 may review the changes made by the third party 2 in the data structure 10 manually.

[0143] The presently disclosed methods may have one or more of the following advantages: the shuffling algorithm is applied at the healthcare providers (e.g., hospital 1) site. The healthcare provider is the only party who has full control of the data structure.

[0144] - The keys to reverse the shuffle stay at the healthcare provider's site. The third party has no way to reverse the shuffle.

[0145] - The algorithm is easy to use and can be applied to tabular data structures with no pre-processing of the data being necessary.

[0146] - The shuffled data set can be cleansed by a third party while having no other data utility for the third party.

[0147] Further aspects of the present invention are disclosed in the following:

[0148] 1. A computer-implemented method for data cleansing comprising the steps of:

[0149] - Obtaining, e.g. with a data pre-processing module of a first computing instance, an at least two-dimensional un-shuffled data structure comprising healthcare patient data, the data structure comprising data on a plurality of patient records, wherein each patient record comprises one or more fields, wherein each field has a field type, and wherein the patient records have a pre-shuffled order;

[0150] - Shuffling, e.g. with the data pre-processing module of the first computing instance, fields of a first field type by moving fields having the first field type from one patient record to another patient record, wherein the fields with the first field type have a post-shuffled order after the shuffling, thereby creating a shuffled data structure;

[0151] - Storing, e.g. with the data pre-processing module of the first computing instance, shuffling information that allows bringing the data from the postshuffled order to the pre-shuffled order;

[0152] - Transmitting, e.g. with a communications module, the shuffled data structure comprising the healthcare patient data, e.g. over a communications network from the first computing instance to a second computing instance, for data cleansing;

[0153] Receiving, e.g. with a communications module, the cleansed, shuffled data structure (e.g. over the communications network), e.g. from the second computing instance on the first computing instance or a third computing instance;

[0154] - Obtaining the shuffling information that allows bringing the data from the post-shuffled order to the pre-shuffled order, e.g. with a data postprocessing module of the first or third computing instance, from a memory, and

[0155] Un-shuffling, e.g. with the data post-processing module of the first or third computing instance, the cleansed shuffled data structure by moving fields having the same field type from one patient record to another patient record according to the information that allows bringing the cleansed shuffled data structure from the post-shuffled order to the pre-shuffled order creating a cleansed un-shuffled data structure.

[0156] 2. Computer-implemented method for pre-processing healthcare patient data prior to a data cleansing, comprising the steps: - Obtaining, e.g. with a data pre-processing module, an at least two- dimensional un-shuffled data structure comprising healthcare patient data, the data structure comprising data on a plurality of patient records, wherein each patient record comprises one or more fields, wherein each field has a field type, and wherein the patient records have a pre-shuffled order;

[0157] - Shuffling, e.g. with the data pre-processing module, fields of a first field type by moving fields having the first field type from one patient record to another patient record, wherein the fields with the first field type have a postshuffled order after the shuffling, thereby creating a shuffled data structure;

[0158] - Storing, e.g. with the data pre-processing module, shuffling information that allows bringing the data from the post-shuffled order to the pre-shuffled order; and

[0159] - Transmitting the shuffled data structure. Computer-implemented method of aspect 1 or 2, additionally comprising the step of determining a similarity measure, e.g. with the data pre-processing module, between the post-shuffled order of the first field type and the pre-shuffled order of the first field type. Computer-implemented method of aspect 3, wherein determining the similarity measure includes determining an entropy of positions of the fields having the first field type in the post-shuffled order in the patient records. Computer-implemented method of aspect 3 or 4, additionally comprising the following steps, if the similarity measure fails to meet a pre-defined level:

[0160] - Selecting, e.g. with the data pre-processing module, a first subset of the plurality of patient records;

[0161] - Shuffling, e.g. with the data pre-processing module, the selected patient records by moving fields of the subset having the first field type from their patient record of the subset to another patient record of the subset, wherein the fields with the first field type have a subsequent post-shuffled order after shuffling the subset of the plurality of records; and

[0162] Determining, e.g. with the data pre-processing module, the similarity measure between the subsequent post-shuffled order of the first field type and the pre-shuffled order of the first field type; and

[0163] - Storing, e.g. with the data pre-processing module, shuffling information that allows un-shuffling the selected patient records. Computer-implemented method of aspect 3 or 4, comprising the following steps: repeating the shuffling step and the storing step, if the similarity measure fails to meet a pre-defined level. Computer-implemented method of aspect 5, wherein the subset comprises a randomly selected number of patient records. Computer-implemented method of aspect 5 or 7, wherein the similarity measure of the patient records outside of the selected subset meets the pre-defined level and the similarity measure of the patient records in the selected subset fails to meet the pre-defined level. Computer-implemented method of any one of aspects 5 to 8, wherein the shuffling step is repeated until a similarity between a subsequent post-shuffled order of the first field type and the pre-shuffled order of the first field type meets the pre-defined level or converges or until a set number of repetitions is reached. Computer-implemented method of any one of aspects 7 to 9, wherein during one or more of the repetitions a second, different subset of patient records is selected and / or wherein during each repetition a further subset of patient records of the first subset is selected. Computer-implemented method of any one of aspects 5 to 10, wherein the storing shuffling information step includes at least one of: storing shuffling information configured to retrace the shuffling of the selected subset of patient data, and storing shuffling information configured to retrace all shuffling steps on the data structure. Computer-implemented method of any one of the preceding aspects, wherein shuffling fields of the first field type and / or shuffling the subset according to any one of aspects 5 to 10 comprises:

[0164] Randomly selecting or randomly generating, e.g. with the data preprocessing module, a seed value; and

[0165] - Applying, e.g. with the data pre-processing module, a permutation function to the fields within each of the one or more patient records based on the seed value. Computer-implemented method of any one of the preceding aspects, wherein the step of storing shuffling information comprises, for one or more or all shuffling steps, at least one of:

[0166] Storing the number of an iteration step in a log file (20); - Storing a unique identifier for each field shuffled during the respective iteration step in the log file;

[0167] - Storing the shuffled field type in the log file;

[0168] - Storing a shuffling method, in particular a permutation function, in the log file; and

[0169] - Storing a seed value for a permutation function used in shuffling the data structure in the log file. Computer-implemented method of any one of the preceding aspects, wherein the data structure is a table comprising rows and columns and wherein each of the patient record forms a row of the table, and each field type forms a column of the table, or vice versa. Computer-implemented method of any one of the preceding aspects, wherein the healthcare patient data comprises at least one of the following field types: a diagnosis, a physical parameter of the patient, an in-vitro-test result, an in-vivo test result, a patient identifier, a sex indication, and an age indication. Computer-implemented method of any one of the preceding aspects, wherein the data structure comprises at least two different field types, comprising the steps of:

[0170] - Shuffling, e.g. with the data pre-processing module, the data structure by moving fields having a second field type from their pre-shuffle patient record to another patient record, wherein the fields with the second field type have a further postshuffled order after the shuffling. Computer-implemented method of aspect 16, wherein the method executes the method steps of any one of aspects 2 to 15 for the second field type. Computer-implemented method of any one of the preceding aspects, wherein the data structure comprises one or more further field types, comprising the steps of:

[0171] - Shuffling, e.g. with the data pre-processing module, the data structure by moving fields having a further field type from their pre-shuffle patient record to another patient record, wherein the fields with the further field type have a post-shuffled order after the shuffling. Computer-implemented method of aspect 18, wherein the method executes the method steps of any one of aspects 2 to 15 for the further field type. Computer-implemented method of any of the preceding aspects, wherein the data structure comprises patient data including a third field type, which is linked to the first field type and wherein shuffling fields of the first field type comprises moving fields having the first and third field type together from one patient record to a different patient record. Computer-implemented method of any of the preceding aspects, comprising the step of:

[0172] - Transmitting, e.g. with a communications module, the data structure comprising healthcare patient data and having patient records with fields in a post-shuffled order over a communications network. Computer-implemented method of any of the preceding aspects, comprising the step of determining a set of field types in the data structure prior to shuffling, and wherein only fields are shuffled that do not have a field type that belongs to the set of field types. Computer-implemented method of aspect 22 comprising the step of removing fields having a field type of the determined set of field types from the data structure and optionally comprising the step of storing the removed fields separately. Computer-implemented method of aspects 22 or 23 comprising the step of storing information identifying the determined set of patient data in a log file. Computer-implemented method of any one of the preceding aspects, wherein the data structure is shuffled until the similarity measure of each field type meets the pre-defined level or converges or until a set number of repetitions is reached. A computer-implemented method for post-processing healthcare patient data after data cleansing, preferably after executing the method of one of aspects 2 to 25, comprising the steps:

[0173] - Obtaining, e.g. with a data post-processing module, a cleansed data structure comprising healthcare patient data, the data structure having a plurality of patient records, wherein the plurality of patient records each comprise one or more fields, and wherein each field has a field type, and wherein the fields have a post-shuffled order;

[0174] - Obtaining the shuffling information that allows bringing the data from the post-shuffled order to the pre-shuffled order, e.g. with a data postprocessing module of the first or third computing instance, from a memory, and

[0175] Un-shuffling, e.g. with the data post-processing module of the first or third computing instance, the cleansed shuffled data structure by moving fields having the same field type from one patient record to another patient record according to the information that allows bringing the cleansed shuffled data structure from the post-shuffled order to the pre-shuffled order creating a cleansed un-shuffled data structure. Computer-implemented method of aspect 26, wherein the information that allows bringing the data from the post-shuffled order to the pre-shuffled order includes a first shuffling method and comprising the step of

[0176] Un-shuffling, e.g. with the data post-processing module, the data structure by moving fields having the first field type from one patient record to another patient record based on the first shuffling method. Computer-implemented method of aspect 27, wherein the information that allows bringing the data from the post-shuffled order to the pre-shuffled order includes a second shuffling method and comprising the step of

[0177] Un-shuffling, e.g. with the data post-processing module, the data structure by moving fields having the first field type from one patient record to another patient record based on the second shuffling method after un-shuffling the data based on the first shuffling method. Computer-implemented method of aspect 27 or 28, wherein information indicates a subset of the plurality of patient records for the first and / or second shuffling method. Computer-implemented method of aspect 29, comprising the step of un-shuffling, e.g. with the data post-processing module, the data structure by moving fields having the same field type from one patient record to another patient record based on the information that allows bringing the data from the post-shuffled order to the pre-shuffled order within the subset of patient records for the first and / or second shuffling method. Computer-implemented method of any one of aspects 27 to 30, wherein the information additionally indicates a field type for the respective shuffling method and wherein the fields having the field type indicated in the log file for the shuffling method are un-shuffled. Computer-implemented method of any one of aspects 27 to 31, wherein the information additionally indicates at least one seed value for at least one permutation function for the respective shuffling method according to which the fields having the first field type were shuffled, wherein the field type indicated in the log file for the shuffling method is un-shuffled based on the seed value and the permutation function. Computer-implemented method of any one of aspects 27 to 32, comprising the step of: Receiving, e.g. with a communications module, the data structure comprising patient data and having patient records with fields in a post-shuffled order over a communications network. Data processing system comprising means for carrying out the method of one of aspects 1 to 33. Computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of aspects 1 to 33. Computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of aspects 1

Claims

1. Claims1. A computer-implemented method for data cleansing comprising the steps of:- Obtaining an at least two-dimensional un-shuffled data structure (10) comprising healthcare patient data, the data structure (10) comprising data on a plurality of patient records, wherein each patient record comprises one or more fields, wherein each field has a field type, and wherein the patient records have a pre-shuffled order;Shuffling fields of a first field type by moving fields having the first field type from one patient record to another patient record, wherein the fields with the first field type have a post-shuffled order after the shuffling, thereby creating a shuffled data structure (10);Storing shuffling information that allows bringing the data from the postshuffled order to the pre-shuffled order;- Transmitting (5) the shuffled data structure (10) comprising the healthcare patient data over a communications network (7) for data cleansing;Receiving the cleansed shuffled data structure (10) over the communications network (7);Obtaining the shuffling information that allows bringing the data from the post-shuffled order to the pre-shuffled order, andUn-shuffling the cleansed shuffled data structure (10) by moving fields having the same field type from one patient record to another patient record according to the information that allows bringing the cleansed shuffled data structure (10) from the post-shuffled order to the pre-shuffled order creating a cleansed un-shuffled data structure (10).

2. Computer-implemented method of claim 1, additionally comprising the step of determining a similarity measure between the post-shuffled order of the first field type and the pre-shuffled order of the first field type.

3. Computer-implemented method of claim 2, additionally comprising the following steps, if the similarity measure fails to meet a pre-defined level:- Selecting a first subset of the plurality of patient records;- Shuffling the selected patient records by moving fields of the subset having the first field type from their patient record of the subset to another patientrecord of the subset, wherein the fields with the first field type have a subsequent post-shuffled order after shuffling the subset of the plurality of records; andDetermining the similarity measure between the subsequent post-shuffled order of the first field type and the pre-shuffled order of the first field type; and- Storing shuffling information that allows un-shuffling the selected patient records.

4. Computer-implemented method of claim 3, comprising the following steps: repeating the shuffling step and the storing step, if the similarity measure fails to meet a pre-defined level.

5. Computer-implemented method of any one of the preceding claims, wherein the step of storing shuffling information comprises, for one or more or all shuffling steps, at least one of:- Storing the number of an iteration step in a log file (20);- Storing a unique identifier for each field shuffled during the respective iteration step in the log file;- Storing the shuffled field type in the log file;- Storing a shuffling method, in particular a permutation function, in the log file; and- Storing a seed value for a permutation function used in shuffling the data structure (10) in the log file.

6. Computer-implemented method of any one of the preceding claims, wherein the data structure (10) is a table comprising rows and columns and wherein each of the patient record forms a row of the table, and each field type forms a column of the table, or vice versa.

7. Computer-implemented method of any one of the preceding claims, wherein the healthcare patient data comprises at least one of the following field types: a diagnosis, a physical parameter of the patient, an in-vitro-test result, an in-vivo test result, a patient identifier, a sex indication, and an age indication.

8. Computer-implemented method of any one of the preceding claims, wherein the data structure (10) comprises at least two different field types, comprising the steps of:- Shuffling the data structure (10) by moving fields having a second field type from their pre-shuffle patient record to another patient record, wherein the fields with the second field type have a further post-shuffled order after the shuffling.

9. Computer-implemented method of any of the preceding claims, wherein the data structure (10) comprises patient data including a third field type, which is linked to the first field type and wherein shuffling fields of the first field type comprises moving fields having the first and third field type together from one patient record to a different patient record.

10. Computer-implemented method of any of the preceding claims, comprising the step of determining a set of field types in the data structure (10) prior to shuffling, and wherein only fields are shuffled that do not have a field type that belongs to the set of field types.

11. Computer-implemented method for pre-processing healthcare patient data prior to a data cleansing, comprising the steps:- Obtaining an at least two-dimensional data structure (10) comprising healthcare patient data, the data structure (10) comprising data on a plurality of patient records, wherein each patient record comprises one or more fields, wherein each field has a field type, and wherein the patient records have a pre-shuffled order;- Shuffling fields of a first field type by moving fields having the first field type from one patient record to another patient record, wherein the fields with the first field type have a post-shuffled order after the shuffling, thereby creating a shuffled data structure (10);- Storing shuffling information that allows bringing the data from the postshuffled order to the pre-shuffled order; and- Transmitting the shuffled data structure (10).

12. A computer-implemented method for post-processing healthcare patient data after data cleansing, preferably after executing the method of one of claims 1 to 11, comprising the steps:- Obtaining a shuffled data structure (10) comprising healthcare patient data, the data structure (10) having a plurality of patient records, wherein the plurality of patient records each comprise one or more fields, and wherein each field has a field type, and wherein the fields have a post-shuffled order;Obtaining the shuffling information that allows bringing the data from the post-shuffled order to the pre-shuffled order from a memory, andUn-shuffling the cleansed shuffled data structure (10) by moving fields having the same field type from one patient record to another patient record according to the information that allows bringing the cleansed shuffled data structure (10) from the post-shuffled order to the pre-shuffled order creating a cleansed, un-shuffled data structure (10).

13. Data processing system comprising means for carrying out the method of one of claims 1 to 12.

14. Computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 1 to 12.

15. Computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 12.

Images