Data preservation with augmentation

Abstract

A computer-implemented method for automatically preserving data in a data preservation system. The method comprises receiving, by the data preservation system, first data for preservation. The method further comprises transmitting, by the data preservation system, the first data to a data augmentation system configured to augment the first data. The method further comprises receiving, by the data preservation system from the data augmentation system, second data for preservation, the second data comprising one or more data augmentations to the first data. The data preservation system maintains a first set of data permissions for the first data and maintains a second set of data permissions for the second data. The first set of data permissions permits a different level of access than the second set of data permissions. The method further comprises storing, by the data preservation system, the first and second data in the data preservation system for later retrieval.

Description

DATA PRESERVATION WITH AUGMENTATION Technical Field

[0001] The following relates to computer-implemented methods for preservation of digital data and in particular to augmenting data for preservation.

[0002] As of 2023, it had been estimated that 90% of the world’s data wasgenerated in the preceding two years and that 330 million Terabytes of data is createdeach day. The overwhelming majority of that data is stored digitally. In addition to therate of data generation, the environment in which that data is stored is highly dynamic.New file formats are regularly created to encode data (such as the HEIC image format)and existing formats may be updated to improve performance or address faults. Someinformation requires several digital files to completely and appropriately represent the information contained within. Within this dynamic environment, digital data can becomelost or unreadable. Old formats may become unreadable as tools used to create, viewor edit those formats become obsolete and fall out of use. Some information may berequired for access long after creation, but it is likely that file formats in widespread useat the time of content creation will not be in use at a desired time of access, maybetens or hundreds of years later.

[0003] While solutions such as storage and back-up can prevent some hardware related issues such as bit-decay (or bit-rot), the dynamic environment in which digital data is produced, stored and edited means that storage and back-up cannot mitigate all causes of loss of access to digital data. Digital preservation refers to activities performed to ensure continued availability of access to digital data and refers to all of the actions required to maintain access to digital materials beyond the limits of media failure or technological and organisational change.

[0004] Digital preservation systems often store data from multiple sources ofvarying sensitivity. That is, digital preservation systems may store data which is readilyavailable to the public, in addition to strictly confidential documents. It is thus desirableto provide a data preservation system that respects the sensitivity of information preserved within. 69679310-1Summary

[0005] There is described herein a computer-implemented method forautomatically preserving data in a data preservation system. The method comprisesreceiving, by the data preservation system, first data for preservation. The method further comprises transmitting, by the data preservation system, the first data to a data augmentation system configured to augment the first data. The method further comprises receiving, by the data preservation system from the data augmentation system, second data for preservation, the second data comprising one or more data augmentations to the first data. The data preservation system maintains a first set of data permissions for the first data and maintains a second set of data permissions for the second data. The first set of data permissions permits a different level of access than the second set of data permissions. The method further comprises storing, by the data preservation system, the first and second data in the data preservation system for later retrieval.

[0006] The first and second data may comprise text data, image data, videodata, and / or audio data.

[0007] The data augmentations may comprise the removal of sensitive information from the first data or the addition of sensitive information to the first data.

[0008] The sensitive information may comprise personally identifiableinformation, financial information, health information, biometric information, confidential information, legal information, location information, personally generated information, government identification information, and / or employment information.

[0009] The first data and / or the second data may be a document file, an imagefile, a video file, an audio file, a Microsoft® SharePoint List Record, a Microsoft®Teams Message, a Microsoft® SharePoint Library Record, a Microsoft® ExchangeEmail, or a Lotus Notes Database Record.

[0010] The data augmentation system may be a system external to the datapreservation system.

[0011] The transmitting and the receiving may comprise interfacing with anapplication programming interface of the data augmentation system.

[0012] The data augmentation system may comprise a Microsoft Azure®service and / or an Amazon Web Services® service.

[0013] The data augmentation system may comprise one or more machine-learning models configured to process the first data for augmentation. 69679310-1

[0014] The second data may be generated by the data augmentation systemaugmenting the first data. The augmentation may comprise redacting, obscuring, translating, transcribing, enhancing, classifying, and / or captioning the first data to generate the second data.

[0015] The first data may be stored in a first location in the data preservationsystem different to a second location in the data preservation system.

[0016] The respective locations may be file directories in the data preservationsystem.

[0017] The sets of data permissions may comprise one or more of read access permissions, write access permissions, delete access permissions, and / or time-limited access permissions.

[0018] The method may further comprise generating link data indicating a link between the first data and the second data. The method may further comprise retrieving, based upon the link data, the first or second data.

[0019] The method may further comprise receiving, by the data preservation system, a request to retrieve the first or second data from the data preservation system. The method may further comprise determining a level of access for the respective first or second data based on the respective first or second set of data permissions. The method may further comprise identifying the respective second or first data preserved in the data preservation system using the link data. The method may further comprise determining a level of access for the respective second or first data based on the respective second or first set of data permissions. Retrieving the firstor second data may be based upon the level of access for the respective first orsecond data.

[0020] The link data may indicate the first and second location of the respectivefirst and second data in the data preservation system. Retrieving the first or seconddata may be further based upon the respective first or second location indicated by thelink data.

[0021] The method may further comprise receiving, by the data preservation system, a request to update the second data. The method may further comprise transmitting, by the data preservation system, the first data to the data augmentation system. The method may further comprise receiving, by the data preservation system from the data augmentation system, an updated version of the second data for preservation. The method may further comprise storing, by the data preservation system, the updated version of the second data. 69679310-1

[0022] The request to update the second data may be based upon identifyingone or more errors in the second data.

[0023] The request to update the second data may be based upon identifyingan update to the data augmentation system.

[0024] The request to update the second data may be based upon determiningthat the quality of the second data is below a threshold quality level.

[0025] Storing the updated version of the second data may comprise replacingthe second data with the updated version of the second data in the data preservation system.

[0026] The second data may be an augmented version of the first data.

[0027] The second data may comprise metadata corresponding to the firstdata.

[0028] There is described a computing system comprising one or more processors and one or more non-transitory computer-readable media storing computer-readable instructions. The computer-readable instructions are configured to cause theone or more processors to perform the method as described above.

[0029] There is described a computer-readable storage medium comprising computer-readable instructions. The computer-readable instructions, when executed bya processor, cause the processor to carry out the method as described above.Brief Description of the Drawings

[0030] Detailed discussion directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended figures, in which:

[0031] Figure 1 schematically depicts an example computing system that performs data preservation with augmentation.

[0032] Figure 2 schematically depicts a data augmentation system thatperforms data augmentation.

[0033] Figure 3 schematically depicts a data preservation system that performs data preservation.

[0034] Figure 4 schematically depicts a flow diagram of a data preservation process performed by a data preservation system.

[0035] Figure 5 schematically depicts a flow diagram of a data preservation process performed by a data preservation system.

[0036] Figure 6 is a schematic illustration of an exemplary computer system onwhich aspects described herein may be implemented. 69679310-1

[0037] Reference numerals that are repeated across plural figures are intended to identify the same features in various implementations. Detailed Description

[0038] Figure 1 schematically depicts an example system in which techniques described herein may be implemented.

[0039] A data preservation system 100 is a computing system configured topreserve (e.g. store) data. The data preservation system 100 may comprisecomponents that enable one or more users to preserve data. That is, the datapreservation system 100 enables data to be stored in such a way that ensures continued access to the data over time. For example, the data preservation system 100may monitor for ongoing changes in operating system environments and performpreservation actions on the data in order to ensure that the data remains compatible.More generally, the data preservation system 100 may preserve data in accordancewith US patent publication number US 20230409521 A1 titled “AutomaticPreservation”. Further detail regarding how the data preservation system 100 maypreserve data is described in the description of UK patent application number2305689.8 titled “Data Preservation”. The data preservation system 100 is configuredto receive data for preservation, for example, first data 102. The first data 102 maycomprise image data, video data, audio data, textual data, or any other suitable form ofdata for preservation. The first data 102 may be any data suitable for preservation. Thefirst data 102 may be a document file, an image file, a video file, an audio file, or anyother suitable file. For example, the first data 102 may be a .docx file, a.png file, a .mp4file, a .wav file, etc. That is, the first data 102 may be a single-part asset. A single-partasset is a single data item (e.g. a .docx file). Equally, the first data 102 may be a multi-part asset. A multi-part asset is a data item comprising a plurality of other files (e.g. a.png file, a .wav file, and a .txt file). For example, the first data 102 may be a Microsoft®SharePoint List Record, a Microsoft® Teams Message, a Microsoft® SharePointLibrary Record, a Microsoft® Exchange Email, or a Lotus Notes Database Record.Other suitable data for preservation are envisaged.

[0040] The data preservation system 100 may be configured to perform variousoperations on data. These operations may serve the purpose of ensuring that usershave ongoing access to the data as mentioned. In some cases, the data preservationsystem 100 may perform operations for some other or related purpose, for example, tomanage and / or facilitate access to information in data comprising sensitive information.69679310-1In another example, the data preservation system 100 may perform actions aimed atenhancing or improving content represented by preserved data. In the example ofFigure 1, the data preservation system 100 is configured to transmit data, for examplethe first data 102, to a data augmentation system 110 for augmentation. The datapreservation system 100 may comprise a network interface to facilitate transmitting and receiving data over a network, such as the internet. The data augmentation system 110is configured to receive, augment, and transmit new data based on the received data).For example, the data augmentation system 110 may receive and transmit data over a network using an application programming interface (API) of the data augmentationsystem 110. The data augmentation system 110 may comprise one or more Softwareas a Service (SaaS), Infrastructure as a Service (IaaS), or Platform as a Service(PaaS) services, such as a Microsoft® Azure service or an Amazon Web Services®service. The augmentation operations performed by the data augmentation system 110 may be highly specific, complex and / or generally only able to be performed by certain entities. As such, the data augmentation system 110 may be implemented by one or more servers adapted to perform the data augmentation. The transmission of data to and / or from the data augmentation system 110 may subject to one or more security operations. For example, the data may be encrypted. Alternatively, the data preservation system 100 itself may comprise the data augmentation system 110. That is, the data preservation system 100 may not be required to transmit and receive data to the data augmentation system 110 over the internet.

[0041] In response to receiving the first data 102, the data augmentationsystem 110 augments the first data 102 to generate second data 104 using one ormore machine learning models 112. While the machine learning models 112 may beused to augment the first data 102, the data augmentation system 110 may, also oralternatively, use any other suitable means for augmenting the first data 102. Thesecond data 104 comprises one or more data augmentations. That is, the second data104 comprises at least one augmentation made to the first data 102. A dataaugmentation may also include new data generated based upon other data. Thesecond data 104 may be an augmented version of the first data 102. That is, thesecond data 104 may be generated based upon the first data 102 and generallyrepresents the first data 102 but for the one or more augmentations. For example, thedata augmentation system 110 may augment the first data 102 in order to removesensitive information in the first data 102. In this example, the second data 104 is anaugmented version of the first data 102 in which the mentioned sensitive information is 69679310-1redacted. While this example describes removing sensitive information from the firstdata 102, other types of augmentation are envisaged. More particularly, the seconddata 104 may comprise augmentations in the form of redactions, obscurations,translations, transcriptions, enhancements, classifications, captions, or any othersuitable augmentation of the first data 102. The second data 104 may comprise imagedata, video data, audio data, textual data, or any other suitable form of data for preservation.

[0042] The second data 104 may be a document file, an image file, a video file, an audio file, or any other suitable file. For example, the second data 104 may be a .docx file, a.png file, a .mp4 file, a .wav file, etc. In another example, the second data104 may be a Microsoft® SharePoint List Record, a Microsoft® Teams Message, aMicrosoft® SharePoint Library Record, a Microsoft® Exchange Email, or a Lotus NotesDatabase Record. That is, the second data 104 may be a single-part or multi-part assetas described above. Other suitable data for preservation are envisaged.

[0043] It will be appreciated that the nature of the augmentation may bedepend upon the nature of the data being augmented. To this end, the machinelearning models 112 used to generate the second data 104 may be determined basedupon the first data 102. In particular, the data preservation system 100 or the dataaugmentation system 110 may identify one or more types (e.g. image, text, etc.) ofdata for the first data 102, the types of data being used to determine the machinelearning models 112 used to generated the second data 104. Each one of the machinelearning models 112 may be configured to perform a different type of dataaugmentation to the first data 102. For example, a first machine learning model may beconfigured to perform redaction, while a second machine learning model may beconfigured to perform obscuration. In some examples, the data preservation system100 or the data augmentation system 110 may use a separate machine learning model(not depicted) to identify the one or more types of data for the first data 102. Multipletypes of data may be identified for the first data 102 which may indicate that multipleones of the machine learning models 112 may be used to generate the second data104. The data preservation system 100 or the data augmentation system 110 maydetermine a machine learning model of the machine learning models 112 for each typeof data identified for the first data 102. For example, the first data 102 may compriseimage and text data. In this example, a first and second machine learning model thatare configured to receive image and text data respectively may be determined. Amachine learning model may be a support vector machine (SVM), an artificial neural 69679310-1network (ANN), a recurrent neural network (RNN), a convolutional neural network(CNN), an attention-based neural network (e.g. a generative pre-trained transformer(GPT)), or any other suitable type of machine learning model. Generally, any type ofmachine learning model is suitable if the machine learning model permits some data(e.g. image, text, etc.) in the first data 102 to be received as input. However, certaintypes of machine learning model may be identified for their improved performance oncertain types of data in the first data 102 (e.g. a CNN may be identified for first data102 comprising image data). Alternatively, the data preservation system 100 or the data augmentation system 110 may determine a machine learning model of themachine learning models 112 for multiple different types of data identified in the firstdata 102. For example, a multi-modal machine learning model may be identified whichis configured to receive both image and text data as input. Furthermore, some (e.g. themulti-modal machine learning model) of the machine learning models 112 may beconfigured to perform multiple different types of data augmentation (e.g. both redactionand obscuration). In this way, different machine learning models may be used toprocess different types of data, thereby providing different types of data augmentationfor the first data 102.

[0044] The second data 104 may comprise metadata based upon the first data 102. That is, the second data 104 may include information relating to, or indicative of, the first data 102. For example, the second data 104 may indicate that the first data102 represents an image depicting “A person”. In this example, the caption “A person”is a data augmentation (i.e. new data generated based upon the first data 102) andmay be considered metadata corresponding to the first data 102. In another example,the second data 104 may indicate a location of sensitive information in text dataincluded in the first data 102. The metadata may be any data suitable for indicatinginformation relating to the first data 102.

[0045] Once the data augmentation system 110 has generated the second data104, the data augmentation system 110 transmits the second data 104 to the datapreservation system 100. The data preservation system 100, after receiving the seconddata 104 from the data augmentation system 110, preserves the second data 104 inthe data preservation system 100. The data preservation system 100 also preservesthe first data 102 in the data preservation system 100. The first data 102 may bepreserved in the data preservation system 100 at any point once it has been receivedby the data preservation system 100. For example, the first data 102 may be preservedbefore or after a copy of the first data 102 is transmitted to the data augmentation69679310-1system 110 for augmentation. The first 102 and second 104 data may be preserved inthe data preservation system 100 with data indicating a link between the first 102 andsecond 104 data. For example, third data (i.e. {0: {original: first_data, augmented:second_data}}) may be preserved in the data preservation system 100 indicating a link between the first 102 and second 104 data. The data indicating a link may be used to locate and retrieve one version of the data (e.g. augmented) based upon the other (e.g. original). Specific details on the data indicating a link are given with reference to Figure3 below. Once the respective first 102 and second 104 data are preserved, the user120 may access the first data 102 and second data 104 preserved in the datapreservation system 100 according to respective first 122 and second 124 set of datapermissions.

[0046] The data preservation system 100 may restrict or permit access by theuser 120 to the first 102 and second 104 data using the first 122 and second 124 set ofdata permissions respectively. For example, it may be desirable to prevent the user 120 from having read access to the first data 102 if the first data 102 comprises sensitive information. In this example, read access to the first data 102 may be restricted by the first set of data permissions 122. In another example, it may be desirable to prevent the user 120 from writing the first data 102. In this example, write access to the first data 102 may be restricted by the first set of data permissions 122. Other suitable restrictions / permissions may be imposed upon the user 120 through the use of the first 122 and second 124 set of data permissions. While a single user 120 is depicted for clarity, there may be a plurality of users 120, each permitted / restricted bythe sets of data permissions.

[0047] The first 122 and / or second 124 sets of data permissions may be determined by the data preservation system 100. For example, the data preservation system 100 may determine to restrict access by the user 120 to the first data 102, whilst permitting access by the user 120 to the second data 104. In some examples, a user (not depicted) providing the first data 102 to the data preservation system 100 for preservation may determine the first set of data permissions 122 for the first data 102and / or the second set of data permissions 124 for the second data 104. Alternatively,the first 122 and / or second 124 sets of data permissions may be determined by the data augmentation system 110. For example, the data augmentation system 110 may determine that the first data 102 comprises sensitive information, and therefore determine to restrict access by the user 120 to the first data 102. In this example, the data augmentation system 110 may determine that the generated second data 104 69679310-1may not comprise the sensitive information, and therefore permit access by the user 120 to the second data 104. In other examples, the data augmentation system 110 provides the data preservation system 100 an indication whether the first 102 and / or second data 104 comprises sensitive information (e.g. {0: {name: “first_data”}, {sensitive_info: true}}, {1: {name: “second_data”}, {sensitive_info: false}}). The data preservation system 100 may therefore determine the first 122 and / or second 124 sets of data permissions accordingly.

[0048] The second data 104 comprising one or more data augmentations maycomprise information in the first data 102. For example, both the first 102 and second104 data may comprise “Paris is the capital city of France”. In this example, the information is innocuous and may not be considered sensitive information. The second data 104 may be used to provide access to certain information in the first data 102 without providing full access to all information in the first data 102. For example, the first data 102 may comprise both sensitive and non-sensitive information. In this example, the user 120 may be provided with access to the non-sensitive information in the first data 102, without being provided access to the sensitive information, by being provided with access to the second data 104. In detail, the data preservation system 100 may limit access by the user 120 to the second data 104 according to a second set of data permissions 124. The second set of data permissions 124 may provide the user 120 with more permissive access to the second data 104 than the user 120 has to the first data 102. As explained in more detail with reference to Figure 3, in some cases the reverse may be true. For example, the data augmentation system 110 may augment the first data 102 to generate second data 104 such that the second data 104 comprises sensitive information which is not present in the first data 102. By providing respective sets of data permissions for original and augmented data, the data preservation system 100 may limit or permit access to the first data 102 and second data 104 as required. The particular permissions applied in each case may depend upon the nature of the information within the first data 102 and second data 104 respectively.

[0049] The data preservation system 100 may perform its ongoing datapreservation functions on the first 102 and / or second 104 data in order to ensure thatthe data is readily accessible to the user 120 in light of the ongoing changes to theoperating system environment as discussed above. Specific details as to how the firstdata 102 and second data 104 are preserved and accessed in the data preservation system 100 will be described with reference to Figure 3 below. 69679310-1

[0050] The second data 104 may be required to be updated. For instance, theuser 120 or the data preservation system 100 itself may require the second data 104 tobe updated. The data preservation system 100 may receive a request to update thesecond data 104. For example, the user 120 may provide the request, or the datapreservation system 100 may provide the request to itself as part of its regular andongoing data preservation actions as described above. The request to update thesecond data 104 may be based upon an identification of one or more errors in thesecond data 104. For example, the user 120 or the data preservation system 100 may determine that the second data 104 comprises sensitive information which should be augmented. Additionally, or alternatively, the request to update the second data 104may be based upon identifying an update to the data augmentation system 110. Forexample, the user 120 or the data preservation system 100 may identify that the data augmentation system 110 comprises a new machine learning model 112 that was notavailable when the second data 104 was initially generated. It may be determined thatthe new machine learning model 112 is capable of one or more augmentations to the first data 102 and / or second data 104 that were not possible using previously available machine learning models 112. Additionally, or alternatively, the request to update thesecond data 104 may be based upon a determination that the quality of the seconddata 104 is below a threshold quality level. For example, the second data 104 maycomprise image data enhanced by the data augmentation system 110. In this example,the data preservation system 100 or the user 120 may determine that the quality of theenhancement is below a threshold quality level or below a level that is available by anewly available machine learning model 112. The determination of quality andcomparison to a threshold quality level may be made by a user 120, or using standardmeasurements, such as a Peak Signal-to-Noise Ratio (PSNR).

[0051] In response to the request to update the second data 104, the datapreservation system 100 may transmit (or retransmit) the preserved copy of the firstdata 102 to the data augmentation system 110. The data augmentation system 110may generate an updated version of the second data 104 based on the first data 102 and / or the second data 104. By transmitting the preserved copy of the first data 102, it can be ensured that updated versions of the second data 104 do not retain errorspresent within the second data 104. However, the data preservation system 100 mayalso, or alternatively, transmit the second data 104. This may be advantageous, forexample, where multiple machine learning models 112 have been used to perform respective augmentations on the first data 102. Where only a single machine learning 69679310-1model 112 has been updated, it may be beneficial for the data augmentation system 110 to have access to the originally generated second data 104. For example, it may be determined that it is unnecessary to re-perform all of the augmentations on the first data 102 and that the updated version of the second data 104 may be obtained by processing at least part of the originally generated second data 104.

[0052] The updated version of the second data 104 may be generated by thedata augmentation system 110 in the same manner as for the second data 104described above. Generating the updated version of the second data 104 may beconsistent with the description of Figure 2 provided below. The data preservationsystem 100 may receive from the data augmentation system 110 the updated versionof the second data 104 for preservation. The updated version of the second data 104may then be preserved in the data preservation system 100. In some examples, theupdated version of the second data 104 replaces, or merges with, the second data 104.In other examples, the updated version of the second data 104 is stored alongside (i.e.in the same file directory) or separately (i.e. in separate file directories) from theoriginally generated second data 104 preserved in the data preservation system 100,thereby maintaining each updated version of the second data 104. By updating the second data 104 as described, an optimal version of the second data 104 is preservedin the data preservation system 100. For example, versions of the second data 104which do not include errors and are of the highest possible quality in light of theaugmentation capabilities of the data augmentation system 110 are maintained.

[0053] Figure 2 schematically depicts the data augmentation system 110 inwhich techniques described herein may be implemented.

[0054] The data augmentation system 110 receives the first data 102 andprocesses it with one or more machine learning models 112 to generate the seconddata 104. As described above, the data augmentation system 110 may process the firstdata 102 to generate the second data 104 using any suitable means, and notnecessarily using the one or more machine learning models 112. For illustrationpurposes, the first data 102 represents a communication.

[0055] The first data 102 comprises image data 210, audio data 220, and textdata 200. The image data 210 comprises sensitive information 230. In this case, thesensitive information is personally identifiable information (PII). More particularly, thesensitive information 230 is a face of a person. The text data 200 also comprisessensitive information 240, 250, 260. In this case, the sensitive information 240, 250,260 are a name, address, and account number respectively. The audio data 220 may 69679310-1also comprise sensitive information (not depicted). For instance, the audio data may be data representing an audio recording of a person speaking “My name is John Smith. Icalled your agents last Tuesday. I had very poor service”. In this case, the sensitiveinformation may be “John Smith”.

[0056] The machine learning models 112 may process the first data 102 togenerate the second data 104. The machine learning models 112 may comprise adifferent machine learning model for each function required by the data augmentationsystem 110. For example, the machine learning models 112 may comprise a machinelearning model specifically adapted for obscuring sensitive information in image data, and a machine learning model specifically adapted for redacting sensitive informationin text data. Generally, the machine learning models 112 may comprise a machinelearning model specifically adapted for the function of redaction, obscuration,translation, transcription, enhancement, classification, captioning, or any other dataaugmentation function required by the data augmentation system 110. It is envisagedthat an individual machine learning model may be capable of performing a plurality of the foregoing functions.

[0057] The second data 104 being generated by the data augmentation system110 comprises one or more data augmentations. As depicted, the second data 104comprises augmented image data 212, augmented audio data 222, and augmentedtext data 202. The augmented image data 212 is an augmented version of the imagedata 210 and represents an image depicting the same person as represented by theimage data 210. However, the augmented image data 212 no longer comprises thesensitive information 230 (i.e. the face of the person). In this case, the face of theperson has been obscured (e.g. blurred, pixelated, replaced, removed, etc.) using themachine learning models 112 to replace the face of the person with a blurred face 232(i.e. a data augmentation). The augmented image data 212 further comprises abounding box 234 indicating a location of the person’s obscured face in the imagerepresented by the augmented image data 212. The second data 104 furthercomprises caption data 236 indicating a caption describing at least a portion of theimage represented by the image data 212, the caption data 236 also generated by themachine learning models 112. Likewise, the augmented text data 202 is an augmented version of the text data 200. The augmented text data 202 no longer comprises thesensitive information 240, 250, 260. In particular, the augmented text data 202comprises three redactions (i.e. redactions 242, 252, 262) corresponding to thesensitive information 240, 250, 260 in the text data 200. Likewise, the augmented audio 69679310-1data 222 is an augmented version of the audio data 220. The augmented audio data222 no longer represents audio including the name “John Smith”. As depicted, thesecond data 104 further comprises transcription data 224 (e.g. text data) indicating atranscription of the audio represented by the augmented audio data 222.

[0058] As mentioned above, the first data 102 may comprise sensitiveinformation 230, 240, 250, 260. Sensitive information is any information that couldpotentially be used to cause, for example, harm, damage, embarrassment, ordiscrimination to an individual or entity if disclosed, accessed, or used without authorization. The sensitive information may include personally identifiable information, financial information, health information, biometric information, confidential information, legal information, location information, personally generated information, governmentidentification information, and / or employment information. Personally identifiableinformation (PII) is any data that can be used to identify, contact, or locate an individualor entity, either on its own or when combined with other information. This includes butis not limited to names, addresses, phone numbers, or email addresses. Financialinformation is any information regarding the finances of a person or entity includingbank account numbers, credit card numbers, income, revenue, profits, and financialstatements. Health information may be any information regarding health, such asmedical history, conditions, treatments, medical records, prescriptions, and test results. Biometric information is any information regarding unique physical or behaviouralcharacteristics of an individual such as fingerprints, facial recognition data, iris scans,or voiceprints. Confidential information is any information that is not intended to bedisclosed to unauthorized parties such as trade secrets or proprietary information.Legal information is any information regarding legal matters such as court records,case details, attorney-client communications, or any data protected by attorney-clientprivilege. Location information is any information regarding a geographic location of anindividual or entity such as GPS coordinates, IP addresses, or location history. Personally generated information is any information that an individual or entity createsor generates themselves such as social media posts, online activity, personal notes,user-generated content (e.g. image / video content items). Government identificationinformation is any information regarding official government-issued identification documents such as driver's license numbers, passport numbers, or social securitynumbers. Employment information is any information regarding an individual’semployment, such as job titles, salary details, performance evaluations, and employee 69679310-1records. Non-sensitive information may include any information not falling within thedefinition of sensitive information provided above.

[0059] The data augmentation system 110 may in some examples, as outlinedabove, generate additional information for inclusion in the second data 104, such asinformation expressed by the bounding box 234, caption data 236, and transcriptiondata 224. In another example, the first data 102 may comprise image data representingan image depicting a plurality of individuals. The image may not be clear enough todiscern the faces of the individuals depicted. It is envisaged that the data augmentationsystem 110 may generate enhanced image data for inclusion in the second data 104.That is, the quality of the image represented by the enhanced image data may beincreased in comparison with that included in the first data 102. The information withinthe image data may then become sensitive information. This is described in furtherdetail below with reference to Figure 3 with regard to the use of two different sets ofdata permissions.

[0060] Advantageously, the data augmentation system 110 may augment the first data 102 to generate the second data 104 such that the second data 104 does not include certain information, such as sensitive information. Equally, the data augmentation system 110 may augment the first data 102 to generate the second data104 such that the second data 104 includes additional information not present in thefirst data 102. Thus, the data augmentation system 110 provides control over theinformation included in the second data 104.

[0061] Figure 3 schematically depicts an example implementation of the datapreservation system 100.

[0062] As described above, the data preservation system 100 may preservethe first data 102 and the second data 104. The first 102 and second 104 data may bepreserved in different locations of the data preservation system 100. For example, the data preservation system 100 may comprise a file system 300. The file system 300 may comprise one or more file directories, such as a first file directory 300a and a second file directory 300b. As described above, the first data 102 has a first set of data permissions 122 and the second data 104 has a second set of data permissions 124.The first and second set of data permissions 122, 124 may determine (i.e. permit orrestrict) access permissions of the user 120 to the first and second file directories 300a,300b, respectively. For example, the second set of data permissions 124 may permitthe user 120 read access to the second file directory 300b, while the first set of datapermissions 124 may restrict read access by the user 120 to the first file directory69679310-1300a. Thus, the user 120 may be able to read the second data 104 preserved in the second file directory 300b, but not be able to read the first data 102 preserved in thefirst file directory 300a. Alternatively, the sets of data permissions may restrict or permitaccess to data on a file-by-file basis, rather than a directory-by-directory basis.

[0063] By way of further example, the data augmentation system 110 maygenerate fourth data 108 by augmenting third data 106. The fourth data 108 may bereceived and preserved by the data preservation system 100. In this example, the thirddata 106 does not comprise sensitive information. However, by virtue of the fact thatthe fourth data 108 has been augmented by the data augmentation system 110, thefourth data 108 now does comprise sensitive information. For instance, the dataaugmentation system 110 may augment image data in the third data 106 to enhance itsquality and reveal facial information. Such facial information may be sensitiveinformation, such as personally identifiable information. As depicted, the third data 106is preserved in the first file directory 300a and the fourth data 108 is preserved in thesecond file directory 300b. Thus, the data preservation system 100 may permit the user120 read access to the third data 106 based upon the first set of data permissions 122and restrict read access by the user 120 to the fourth data 108 based upon the secondset of data permissions 124. In this way, a different level of access to the original (e.g.third data 106) and augmented data (fourth data 108) is provided to the user 120. Alevel of access may be a subset of the total possible permissions that may be providedto the user 120. It will be appreciated that while read and write access are describedabove, other access permissions, such as delete access or time-limited permissions may also be provided. For example, a level of access may be {read access, writeaccess} – i.e. not including other types of access permissions such as delete access.Different levels of access provided to the user 120 may be used to account for the nature of the information in the data, for example, if sensitive information is present in either original or augmented copies.

[0064] Depending upon the nature of the original or augmented data, different levels of access may be required in order to restrict or permit access to such data.Therefore, the first 122 and second 124 sets of data permissions may determine read,write, delete, and / or time-limited access permissions by the user 120 to the data inquestion for any particular user. Read access permissions may be permissions grantedto the user 120 (or entity) permitting the user 120 to view or read the content of thedata in question. Write access permissions may be permissions granted to the user120 (or entity) permitting the user 120 to add to or modify the content of the data in69679310-1question. Delete access permissions may be permissions granted to the user 120 (orentity) permitting the user 120 to delete the data in question. Time-limited accesspermissions are permissions granted to the user 120 (or entity) permitting the user 120any one of the foregoing permissions restricted to one or more predetermined period(s) of time.

[0065] The data preservation system 100 may generate link data indicating alink between original and augmented data. A link may indicate that one piece of data(e.g. second data 104) is associated with (e.g. is based upon) the other (e.g. first data102). For example, first link data 304 may be generated to indicate a link between thefirst data 102 and the second data 104. Likewise, second link data 306 may begenerated to indicate a link between the third data 106 and the fourth data 108. Thefirst link data 304 may be used to retrieve the second data 104 located in the filesystem 300. For example, the user 120 may wish to gain read access the first data102. In this example, the first set of data permissions 122 may restrict read access by the user 120 to the first data 102. In this case, the first link data 304 may indicate thatthe first data 102 is linked to (i.e. corresponds to) the second data 104. The link datamay indicate a location of a respective data item preserved in the data preservationsystem 100. For example, the first link data 304 may indicate that the first data 102 ispreserved in the first file directory 300a of the file system 300, and that the second data 104 is preserved in the second file directory 300b of the file system 300. Alternatively, the link data may provide a means for conducting a search of the data preservation system 100 for a respective data item. For example, the link data may indicate that first data 102 is linked to the second data 104, and further provide their respectivefilenames for use in a search function of the data preservation system 100. Thus, thesecond data 104 may be located in the data preservation system 100and retrieved.This may be useful if, for example, the second set of data permissions 124 provides thelevel of access required by the user 120. The first link data 304 may be providedalongside (e.g. in the same directory) or as part of the first 102 or second 104 data.Link data may additionally be used upon determining that augmented data needs updating, as described above. In this way, the original data on which the augmented data may be retrieved to allow for new augmentations to be generated based upon the original data.

[0066] The link data may have none or minimal user access restrictions inorder to allow the user 120 to readily locate data in the file system 300 of the datapreservation system 100. Thus, if the user 120 is not permitted read access to the first69679310-1data 102 according to the first set of data permissions 122, the first link data 304 provides a means for readily locating and retrieving (e.g. accessing) an augmentedversion of the first data 102 (i.e. the second data 104) which the user 120 may bepermitted to access according to the second set of data permissions 124. This improves data accessibility within the data preservation system 100, whilst respectingdata permissions which may, for example, have been put in place to protect sensitiveinformation in the first 102 or second 104 data. The link data may comprise both thefirst 304 and second 306 link data. For example, the link data may be data indicating a plurality of one-to-one relationships, links, correspondence, etc. between an originaland an augmented version of the original data. In other examples, the link data may bedata indicating a plurality of one-to-many relationships, links, correspondence, etc. between original and multiple different augmented versions of the original data. Thelink data may be a database record (e.g. a record in an SQL database). For example,the link data may be {“123456, first_data.png, / path / to / first_data.png, second_data.png, / path / to / second_data.png”} for corresponding fields {“ID, first_filename, first_location,second_filename, second_location“} in the SQL database. Equally, the link data maybe a JSON file, or any other suitable data. For example, link data may represent aJSON file: {original: “first_data.png”, augmented: [“second_data_v1.png”,“second_data_v2.png”, …]}. The link data may be stored centrally within the datapreservation system 100, thereby acting as a manifest for the data preserved in the data preservation system 100. Thus, a means for identifying augmented or originalversion(s) of data based upon a respective original or augmented version(s) of saiddata is provided.

[0067] Figure 4 depicts a flow diagram of a data preservation process that maybe carried out by a data preservation system such as the data preservation system 100described herein.

[0068] At step 400, first data is received by the data preservation system for preservation.

[0069] The first data may comprise text data, image data, video data, audiodata, and / or any other suitable form of data.

[0070] The first data may comprise sensitive information or non-sensitiveinformation. The sensitive information may comprise personally identifiable information, financial information, health information, biometric information, confidential information, legal information, location information, personally generated information, government 69679310-1identification information, employment information, and / or any other suitable form of sensitive information.

[0071] The first data may be a document file, an image file, a video file, anaudio file, or any other suitable file. The first data may be a Microsoft® SharePoint ListRecord, a Microsoft® Teams Message, a Microsoft® SharePoint Library Record, aMicrosoft® Exchange Email, a Lotus Notes Database Record, or any other suitableform of data for preservation. More generally, the first data may be as describedanywhere in this specification.

[0072] At step 402, the first data is transmitted by the data preservation systemto a data augmentation system, for example, the data augmentation system 110 described herein.

[0073] The data augmentation system may be a system external to the datapreservation system. For example, the data augmentation system may not be aninternal system / process of the data preservation system. As such, the transmission ofthe first data to the data augmentation system may comprise transmission over the internet.

[0074] Transmitting and receiving data to and from the data augmentationsystem may comprise using an application programming interface (API) of the dataaugmentation system.

[0075] The data augmentation system may comprise a Microsoft Azure®service, an Amazon Web Services® service, and / or any other suitable Software as aService (SaaS) that is configured to augment the first data.

[0076] The data augmentation system may comprise one or more machine learning models configured to process the first data for augmentation.

[0077] At step 404, second data is received by the data preservation systemfrom the data augmentation system for preservation. The second data comprises oneor more data augmentations. The first data corresponds to a first set of data permissions that is different than a second set of data permissions corresponding tothe second data (i.e. provides a different level of access). The second data maycomprise information that enables the data preservation system to internally identify the first data stored within the data preservation system, such as an identification of thefirst data. The second data is an augmented version of the first data. The second datamay generally take the form describe above. The second data is generated by the data augmentation system augmenting the first data. As described above, the augmentingmay comprise redacting, obscuring, translating, transcribing, enhancing, classifying,69679310-1and / or captioning the first data to generate the second data. The second data may be adocument file, an image file, a video file, an audio file, or any other suitable file. Thesecond data may be a Microsoft® SharePoint List Record, a Microsoft® TeamsMessage, a Microsoft® SharePoint Library Record, a Microsoft® Exchange Email, aLotus Notes Database Record, or any other suitable form of data for preservation. More generally, the second data may be as described anywhere in this specification.

[0078]

[0079] At step 406, the first and second data are stored by the datapreservation system for later retrieval. The storage may be as described above.

[0080] For example, the first data is stored in a first location of the dataaugmentation system and the second data is stored in a second location of the data augmentation system different to the first location.

[0081] The data preservation process may (but need not) further include step408. At step 408, link data indicating a link between the first data and the second datais generated. The first or second data may then be retrieved from the data preservation system based upon the link data indicating the link.

[0082] The data preservation process may also further include (but need not)step 410. At step 410, a request is received by the data preservation system to update the second data. The first data may then be transmitted to the data augmentationsystem by the data preservation system for updating. In response, the datapreservation system may receive an updated version of the second data for preservation system from the data augmentation system. In response, the data preservation system may then store the updated version of the second data.

[0083] Figure 5 depicts a flow diagram of a data preservation process that maybe carried out by a data preservation system such as the data preservation system 100 described herein.

[0084] At step 500, a request is received by the data preservation system toretrieve first data preserved in the data preservation system. The request may bereceived from a user or entity (e.g. computer system) of the data preservation system.

[0085] At step 502, a level of access for the first data is determined. The levelof access may be for the user or entity. For example, a first set of data permissions maintained by the data preservation system for the user may indicate that the user hasno access permissions to the first data.

[0086] At step 504, link data indicating a link between the first and second data is used to identify second data preserved in the data preservation system. For 69679310-1example, the link data may be included in the first data, or stored alongside the firstdata, and may indicate a location of the second data preserved in the data preservationsystem. More generally, the link data may be as described anywhere in this specification.

[0087] At step 506, a level of access for the second data is determined. The level of access may be for the user or entity. For example, a second set of data permissions maintained by the data preservation system for the user may indicate that the user has read access permissions to the second data.

[0088] At step 508, the preserved second data is retrieved from the datapreservation system. The preserved second data may be retrieved in response todetermining that the user has certain access permissions to the second data. Thepreserved second data may be retrieved in response to determining that the user has different (e.g. more permissive) access permissions to the second data than to the firstdata. The link data may indicate a location of the second data (e.g. a file directory inwhich the second data is preserved). The retrieval of the second data may be basedupon the location indicated by the link data. For example, the link data may indicate that the second data is preserved at “path / to / second_data”.

[0089] Figure 6 schematically illustrates an exemplary arrangement ofcomponents which may provide a computing system 4 used to implement all or part ofthe preservation system 100 or the data augmentation system 110.

[0090] A processor, in this case in the form of a CPU 4a, configured to read and execute instructions stored in a volatile memory 4b which takes the form of a random access memory. It will be appreciated that the processor may take other forms, such as, for example, a GPU. The volatile memory 4b stores instructions for execution by the CPU 4a and data used by those instructions. For example, the instructions may include instructions for causing the preservation system 100 to carry out the processingdescribed above with reference to the respective Figures.

[0091] The computing system 4 comprises a storage device 5. It will be appreciated that the storage device 5 may be implemented in any way, such as for example, a hard disk drive, a solid state drive, etc. The storage device 5 may provide the means for preserving data in the data preservation system 100. The computing system 4 further comprises an I / O interface 4d to which are connected peripheral devices used in connection with the computing system. More particularly, a display 4e is configured so as to display output. Input devices are also connected to the I / O interface 4d. Such input devices include a keyboard 4f and a mouse 4g which allow 69679310-1user interaction with the computing system 4. A network interface 4h allows the computing system 4 to be connected to appropriate computer networks, such as theInternet 6, and so as to be able to send and receive from and to other computingdevices. For example, to send and receive between the data preservation system 100 and the data augmentation system 110. The network interface 4h may provide the network interface 112. The CPU 4a, volatile memory 4b, the storage device 5, I / O interface 4d, and network interface 4h, are connected together by a bus 4i.

[0092] The techniques described above may be implemented in hardware, firmware, software, or any combination thereof. The techniques may also be implemented as instructions stored on a machine-readable medium, which may be read and executed by one or more processors. A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include read only memory (ROM); random access memory (RAM); magnetic storage media; optical storage media; flash memory devices; electrical, optical, acoustical or other forms of propagated signals (e.g. carrier waves, infrared signals, digital signals, etc.), and others. Further, firmware, software, routines, instructions may be described herein as performing certain actions. However, it should be appreciated that such descriptions are merely for convenience and that such actions in fact result from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc. and in doing that may cause actuators or other devices to interact with the physical world.

[0093] While specific embodiments of the invention have been described above, it will be appreciated that the invention may be practiced otherwise than as described. The descriptions above are intended to be illustrative, not limiting. Thus it will be apparent to one skilled in the art that modifications may be made to the invention as described without departing from the spirit of the invention. 69679310-1

Claims

CLAIMS:

1. A computer-implemented method for automatically preserving data in adata preservation system, comprising at one or more computing devices: receiving, by the data preservation system, first data for preservation; transmitting, by the data preservation system, the first data to a data augmentation system configured to augment the first data; receiving, by the data preservation system from the data augmentation system, second data for preservation, the second data comprising one or moredata augmentations to the first data;wherein the data preservation system maintains a first set of datapermissions for the first data and maintains a second set of data permissions forthe second data, and wherein the first set of data permissions permits a different level of access than the second set of data permissions; and storing, by the data preservation system, the first and second data in the data preservation system for later retrieval.

2. The method of claim 1, wherein the first and second data comprises textdata, image data, video data, and / or audio data.

3. The method of any preceding claim, wherein the dataaugmentations comprise the removal of sensitive information from the first data or the addition of sensitive information to the first data.

4. The method of claim 3, wherein the sensitive information comprisespersonally identifiable information, financial information, health information, biometric information, confidential information, legal information, location information, personally generated information, government identification information, and / or employment information.

5. The method of any preceding claim, wherein the first data and / or thesecond data is a document file, an image file, a video file, an audio file, a69679310-1Microsoft® SharePoint List Record, a Microsoft® Teams Message, a Microsoft®SharePoint Library Record, a Microsoft® Exchange Email, or a Lotus NotesDatabase Record.

6. The method of any preceding claim, wherein the data augmentationsystem is a system external to the data preservation system.

7. The method of any preceding claim, wherein the transmitting and thereceiving comprises interfacing with an application programming interface of the data augmentation system.

8. The method of any preceding claim, wherein the data augmentationsystem comprises a Microsoft Azure® service and / or an Amazon WebServices® service.

9. The method of any preceding claim, wherein the data augmentationsystem comprises one or more machine-learning models configured to process the first data for augmentation.

10. The method of any preceding claim, wherein the second data isgenerated by the data augmentation system augmenting the first data, the augmenting comprising: redacting, obscuring, translating, transcribing, enhancing, classifying, and / or captioning the first data to generate the second data.

11. The method of any preceding claim, wherein the first data is stored in afirst location in the data preservation system different to a second location in the data preservation system.

12. The method of claim 11, wherein the respective locations are filedirectories in the data preservation system. 69679310-113. The method of any preceding claim, wherein the sets of data permissionscomprise one or more of read access permissions, write access permissions, delete access permissions, and / or time-limited access permissions.

14. The method of any preceding claim, further comprising:generating link data indicating a link between the first data and thesecond data; and retrieving, based upon the link data, the first or second data.

15. The method of claim 14, further comprising:receiving, by the data preservation system, a request to retrieve the firstor second data from the data preservation system; determining a level of access for the respective first or second databased on the respective first or second set of data permissions; identifyingthe respective second or first data preserved in the data preservation system using the link data; determining a level of access for the respective second or first data based on the respective second or first set of data permissions; and wherein retrieving the first or second data is based upon the level ofaccess for the respective first or second data.

16. The method of claims 14 or 15 when dependent upon claims 11 to 13,wherein the link data indicates the first and second location of therespective first and second data in the data preservation system; and wherein retrieving the first or second data is further based upon the respective first or second location indicated by the link data.

17. The method of any preceding claim, further comprising:receiving, by the data preservation system, a request to update the second data; transmitting, by the data preservation system, the first data to the data augmentation system; 69679310-1receiving, by the data preservation system from the data augmentation system, an updated version of the second data for preservation; and storing, by the data preservation system, the updated version of the second data.

18. The method of claim 17, wherein the request to update the second datais based upon identifying one or more errors in the second data.

19. The method of any of claims 17 to 18, wherein the request to update thesecond data is based upon identifying an update to the data augmentationsystem.

20. The method of any of claims 17 to 19, wherein the request to update thesecond data is based upon determining that the quality of the second data is below a threshold quality level.

21. The method of any of claims 17 to 20, wherein storing the updatedversion of the second data comprises replacing the second data with the updated version of the second data in the data preservation system.

22. The method of any preceding claim, wherein the second data is anaugmented version of the first data.

23. The method of any preceding claim, wherein the second data comprisesmetadata corresponding to the first data.

24. A computing system comprising:one or more processors; and one or more non-transitory computer-readable media storing computer- readable instructions configured to cause the one or more processors to perform a method according to any one of the preceding claims. 69679310-125. A computer-readable storage medium comprising computer-readableinstructions, the computer-readable instructions, when executed by a processor,cause the processor to carry out the method of any of claims 1 to 23.69679310-1

Images