Patient and consumer data de-identification

Abstract

The present technology provides a solution for anonymizing patient data, in particular, a solution for anonymizing potentially unique patient identification information while retaining clinically relevant information, for example, which may be used to show the outcome or outcome of a clinical intervention. In some aspects, a process of the disclosed technology may include the steps of: receiving patient data, where the patient data includes a patient image corresponding to a patient; receiving treatment data, where the treatment data is based on a treatment target for the patient; automatically determining one or more anonymization parameters for the patient image using the one or more semantic controls; and generating an anonymized patient image based on the one or more anonymization parameters. Systems and machine-readable media are also provided.

Description

Technical Field

[0001] The present technology provides a solution for anonymizing patient data, and in particular, for anonymizing or obfuscating identifying features of a potentially unique patient while retaining clinically relevant information that can be used, for example, to explain the results or outcomes of a clinical intervention to other clinical professionals. Background Art

[0002] Conventional methods of removing personally identifiable patient information from confidential patient data may be less than ideal in at least some aspects. Many countries require that medical records comply with standards for protecting patient data, such as the Health Insurance Portability and Accountability Act ("HIPAA") in the United States and the General Data Protection Regulation ("GDPR") in the European Union. These standards may require that personally identifiable information be removed before patient data such as patient images are made publicly available. Treatment professionals (such as doctors) may wish to share treatment information with other professionals in order to share knowledge about patient care and best practices. In order to present such patient data to a third party, personally identifiable information is usually removed. However, conventional methods of removing identifying features / artifacts may cause patient images to be incomplete and the images to look unnatural or aesthetically unattractive. Summary of the invention

[0003] Various example embodiments are outlined. These example embodiments are for illustration only and are not to be construed as limiting.

[0004] In a first embodiment, a computer-implemented method includes receiving patient data, wherein the patient data includes a patient image corresponding to a patient; receiving treatment data, wherein the treatment data is based on a treatment goal for the patient; automatically determining one or more anonymization parameters for the patient image using one or more semantic controls; and generating an anonymized patient image based on the one or more anonymization parameters.

[0005] The second embodiment may further extend the first embodiment. In the second embodiment, the anonymized patient image retains the clinically relevant portion of the patient image based on the treatment data.

[0006] The third embodiment may further expand the first or second embodiment. In the third embodiment, one or more anonymization parameters are based on treatment data.

[0007] The fourth embodiment may further extend any one of the first to third embodiments. In the fourth embodiment, one or more anonymization parameters are based on patient data.

[0008] The fifth embodiment may further extend any one of the first to fourth embodiments. In the fifth embodiment, one or more anonymization parameters for a patient image specify one or more anonymization regions in the patient image to be modified to obscure the identity of the patient.

[0009] The sixth embodiment may further extend any one of the first to fifth embodiments. In the sixth embodiment, one or more anonymization parameters for the patient image specify one or more clinically relevant regions in the patient image that are not to be modified.

[0010] The seventh embodiment may further extend any one of the first to sixth embodiments. In the seventh embodiment, one or more anonymization parameters specify one or more image characteristics to be applied to generate an anonymized patient image.

[0011] The eighth embodiment may further extend the seventh embodiment. In the eighth embodiment, the one or more image characteristics include color parameters, blur filter parameters, geometric parameters or a combination thereof.

[0012] The ninth embodiment can further extend any one of the first to eighth embodiments. In the ninth embodiment, one or more semantic controls are based on user input.

[0013] In a tenth embodiment, a system includes: at least one memory; and at least one processor connected to the at least one memory, the at least one processor being configured to: receive patient data, wherein the patient data includes a patient image corresponding to a patient; receive treatment data, wherein the treatment data is based on a treatment goal for the patient; automatically determine one or more anonymization parameters for the patient image using one or more semantic controls; and generate an anonymized patient image based on the one or more anonymization parameters.

[0014] The eleventh embodiment may further extend the tenth embodiment. In the eleventh embodiment, the anonymized patient image retains the clinically relevant portion of the patient image based on the treatment data.

[0015] The twelfth embodiment may further extend the tenth or eleventh embodiment. In the twelfth embodiment, one or more anonymization parameters are based on treatment data.

[0016] The thirteenth embodiment may further expand any one of the tenth to twelfth embodiments. In the thirteenth embodiment, one or more anonymization parameters are based on patient data.

[0017] The fourteenth embodiment may further expand any one of the tenth to thirteenth embodiments. In the fourteenth embodiment, the one or more anonymization parameters for the patient image specify one or more anonymization regions in the patient image to be modified to obscure the identity of the patient.

[0018] The fifteenth embodiment may further expand any one of the tenth to fourteenth embodiments. In the fifteenth embodiment, the one or more anonymization parameters for the patient image specify one or more clinically relevant regions in the patient image that are not to be modified.

[0019] The sixteenth embodiment may further expand any one of the tenth to fifteenth embodiments. In the sixteenth embodiment, one or more anonymization parameters specify one or more image characteristics to be applied to generate an anonymized patient image.

[0020] The seventeenth embodiment may further extend the sixteenth embodiment. In the seventeenth embodiment, the one or more image characteristics include color parameters, blur filter parameters, geometric parameters or a combination thereof.

[0021] The eighteenth embodiment may further extend any one of the tenth to seventeenth embodiments. In the eighteenth embodiment, one or more semantic controls are based on user input.

[0022] In a nineteenth embodiment, a non-transitory computer-readable storage medium comprises at least one instruction for causing a computer or processor to perform the following operations: receiving patient data, wherein the patient data comprises a patient image corresponding to a patient; receiving treatment data, wherein the treatment data is based on a treatment goal for the patient; automatically determining one or more anonymization parameters for the patient image using one or more semantic controls; and generating an anonymized patient image based on the one or more anonymization parameters.

[0023] The twentieth embodiment may further extend the nineteenth embodiment. In the twentieth embodiment, the anonymized patient image retains the clinically relevant portion of the patient image based on the treatment data.

[0024] The twenty-first embodiment may further extend the nineteenth or twentieth embodiment. In the twenty-first embodiment, one or more anonymization parameters are based on treatment data.

[0025] The twenty-second embodiment may further extend any one of the nineteenth to twenty-first embodiments. In the twenty-second embodiment, one or more anonymization parameters are based on patient data.

[0026] The twenty-third embodiment may further extend any one of the nineteenth to twenty-second embodiments. In the twenty-third embodiment, the one or more anonymization parameters for the patient image specify one or more anonymization regions in the patient image to be modified to obscure the identity of the patient.

[0027] The twenty-fourth embodiment may further extend any one of the nineteenth to twenty-third embodiments. In the twenty-fourth embodiment, the one or more anonymization parameters for the patient image specify one or more clinically relevant regions in the patient image that are not to be modified.

[0028] The twenty-fifth embodiment may further extend any one of the nineteenth to twenty-fourth embodiments. In the twenty-fifth embodiment, the one or more anonymization parameters specify one or more image characteristics to be applied to generate the anonymized patient image.

[0029] The twenty-sixth embodiment can further expand any one of the nineteenth to twenty-fifth embodiments. In the twenty-sixth embodiment, the one or more image characteristics include color parameters, blur filter parameters, geometric parameters or a combination thereof.

[0030] The twenty-seventh embodiment can further extend any one of the nineteenth to twenty-sixth embodiments. In the twenty-seventh embodiment, one or more semantic controls are based on user input. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to describe the manner in which the above and other advantages and features of the present disclosure can be obtained, a more particular description of the principles briefly described above will be presented by reference to the embodiments shown in the accompanying drawings. It should be understood that these drawings depict only exemplary embodiments of the present disclosure and are therefore not to be considered limiting of its scope. The principles of the present invention are described and explained in more detail and detail through the use of the accompanying drawings, in which:

[0032] Figure 1 A conceptual block diagram of an example system for anonymizing patient data using one or more semantic controls, according to some examples is shown.

[0033] Figure 2 and Figure 3 A conceptual block diagram of a process for anonymizing patient data using semantic controls, according to some examples, is shown.

[0034] Figure 4 is a flow chart of an example process for anonymizing patient data in accordance with some embodiments.

[0035] Figure 5 An example of a data flow diagram illustrating an example computer-implemented data flow for anonymizing patient data according to some examples.

[0036] Figure 6 An example of a deep learning neural network that can be implemented to facilitate a data de-identification process is shown according to some examples.

[0037] Figure 7 An example computing system according to some examples is shown. DETAILED DESCRIPTION

[0038] Various embodiments of the present disclosure are discussed in detail below. Although specific embodiments are discussed, it should be understood that this is done for illustrative purposes only. Those skilled in the relevant art will recognize that other components and configurations may be used without departing from the spirit and scope of the present disclosure. Therefore, the following description and drawings are illustrative and should not be construed as limiting. Many specific details are described to provide a thorough understanding of the present disclosure. However, in some cases, well-known or conventional details are not described to avoid obscuring the description. Reference to an embodiment or reference to an embodiment in the present disclosure may be reference to the same embodiment or any embodiment; and, such reference refers to at least one of the embodiments.

[0039] Reference to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. In addition, various features are described that may be exhibited by some embodiments but not by others.

[0040] The terms used in this specification sheet generally have their common meaning in the art in the context of the present disclosure and in the specific context in which each term is used. Alternative language and synonyms may be used for any one or more terms discussed herein, and whether or not a certain term is described in detail or discussed herein should not be considered as having special meaning. In some cases, synonyms for certain terms are provided. The recording of one or more synonyms does not exclude the use of other synonyms. The use of examples (including examples of any terms discussed herein) anywhere in this specification sheet is only illustrative, and is not intended to further limit the scope and meaning of the present disclosure or any example terms. Similarly, the present disclosure is not limited to the various embodiments provided in this specification sheet.

[0041] It is not intended to limit the scope of the present disclosure, and examples of instruments, devices, methods, and related results thereof according to embodiments of the present disclosure are given below. Note that for the convenience of the reader, titles or subtitles may be used in the examples, which should never limit the scope of the present disclosure. Unless otherwise defined, the technical and scientific terms used herein have the meanings commonly understood by those of ordinary skill in the art to which the present disclosure belongs. In the event of a conflict, the present document including the definitions shall prevail.

[0042] Additional features and advantages of the present disclosure will be set forth in the following description, and in part will become apparent from the description, or may be learned by practicing the principles disclosed herein. The features and advantages of the present disclosure may be realized and obtained by the instruments and combinations particularly pointed out in the appended claims. These and other features of the present disclosure will become more fully apparent from the following description and the appended claims, or may be learned by practicing the principles set forth herein.

[0043] Overview

[0044] In some aspects, a computer-implemented method of the disclosed technology may include receiving patient (or client / consumer) data, wherein the patient data includes a patient image corresponding to the patient; receiving treatment data, wherein the treatment data is based on treatment goals for the patient; automatically determining one or more anonymization parameters for the patient image using one or more semantic controls; and generating an anonymized patient image based on the one or more anonymization parameters.

[0045] A system of the disclosed technology may include at least one memory and at least one processor connected to the at least one memory, the at least one processor being configured to: receive patient data, wherein the patient data includes a patient image corresponding to a patient; receive treatment data, wherein the treatment data is based on a treatment goal for the patient; automatically determine one or more anonymization parameters for the patient image using one or more semantic controls; and generate an anonymized patient image based on the one or more anonymization parameters.

[0046] A non-transitory computer-readable storage medium may include at least one instruction for causing a computer or processor to receive patient data, wherein the patient data includes a patient image corresponding to a patient; receive treatment data, wherein the treatment data is based on a treatment goal for the patient; automatically determine one or more anonymization parameters for the patient image using one or more semantic controls; and generate an anonymized patient image based on the one or more anonymization parameters.

[0047] describe

[0048] Before performing a medical treatment, a medical professional may record or otherwise capture patient data, including patient anatomical visual data (e.g., image data, video data, or 3D video scan data), including one or more clinically relevant areas. For example, before implementing a treatment, a medical professional may take a photo of the patient's mouth and / or face or perform a three-dimensional (3D) scan, such as a cone beam computed tomography (CBCT) scan. In some cases, a treatment professional may wish to discuss possible treatments with other professionals (e.g., experts such as orthodontists, oral surgeons, cosmetic surgeons, periodontists, and / or general dentists) or publicly study the current case. A treatment professional may share a patient's photo with other treatment professionals and / or other patients to discuss treatment. In order to protect patient privacy, it may be desirable to conceal certain patient data, such as patient identifiable information, from other treatment professionals. Additionally, some regulations and laws may require privacy standards for patient information as described above. However, some data such as photos showing the patient's face may be substantially identifiable.

[0049] Conventionally, patient images can be rendered unrecognizable by masking, blanking, or otherwise obscuring portions of the patient's face, such as the eyes and nose. For example, a portion of the patient's face, such as the eyes, can be obscured. Alternatively, the patient image can be cropped to remove identifiable features of the patient. While effective in maintaining patient anonymity, these obscured or cropped images can be aesthetically unappealing and can divert attention from the clinical condition and can obscure potentially relevant clinical information.

[0050] Aspects of the present disclosure provide a solution for anonymizing patient data in a manner that results in an aesthetically coordinated final imaging of the patient in order to protect patient privacy, i.e., maintaining clinically relevant treatment information, for example, information that can be used for sharing between colleagues or other professionals and / or patients. In some aspects, the manner in which patient information is anonymized (or obfuscated) can be implemented using one or more semantic controls, for example, based on patient data, treatment data, and / or user-provided input. As used herein, a semantic control can include a rule-based system or application (computer program code) that can be used to automate and / or systematize the modification of patient imaging, for example, to generate anonymized patient image data. In some aspects, a semantic control can be used to determine or identify a set of anonymization parameters, for example, can be used to apply or implement a specific image editing operation to generate the resulting anonymized patient image. As discussed in further detail below, a semantic control can be configured to receive various types of information, including but not limited to patient data including patient imaging, treatment data including information about the treatment to be performed on the patient, and / or user input data that can include input provided by a clinician or other treatment professional.

[0051] In practice, semantic controls can be used to automatically determine how patient data, such as patient images, should be modified to obscure or remove potentially identifying or unique information. Patient images, such as image data (photographs, videos, 3D videos) or 3D scan data (e.g., intraoral scan data generated by an intraoral scanner or x-ray data such as CBCT data) can be included in patient data, and the data can be used or manipulated by one or more semantic controls to help determine how the patient image should be modified for anonymization purposes. As used herein, patient data can include other information about the patient, such as demographic information (e.g., gender, age, height, etc.).

[0052] A decision on how to best anonymize patient data (e.g., patient images) can be made based on treatment data corresponding to the patient. As used herein, treatment data can include substantially any information related to a potential clinical intervention associated with the patient. Examples of treatments include orthodontic treatment, oral prosthesis treatment, plastic surgery, cosmetic dental treatment, weight gain, weight loss, etc. In some examples, a semantic control can use treatment data to automatically determine anonymization parameters that should be applied to patient images, such as to anonymize patient images such as photos and / or 3D scans. As discussed in more detail below, treatment data can be used to determine anonymization parameters that define portions of patient images that should be modified to obscure patient identities (e.g., image anonymization regions), and / or identify portions of patient images that should remain unchanged, such as to retain any clinically relevant information that can be shared and used between clinical professionals and / or other patients / consumers, etc. Additionally, anonymization parameters can indicate changes throughout the image, such as not limited to specific pixel regions. For example, anonymization parameters may indicate an overall change to image tone or hue (e.g., changing a patient's skin color), image geometry, or prominence of particular image regions or features, e.g., to change or obscure a patient's gender identity, etc. Several of the examples provided discuss clinical interventions in the context of orthodontics, however, it should be understood that the disclosed aspects may be applied in other clinical settings and are not limited to the examples provided herein.

[0053] Figure 1 A conceptual block diagram of an example system 100 for anonymizing patient data using one or more semantic controls is shown. As shown in the example of system 100, patient data 102, treatment data 104, and / or user input 106 can all be presented to a semantic control 108. As described above, the semantic control can include a set of rules (or models) that can be configured to generate anonymization parameters 110 to define how an image (e.g., a patient image) can be modified to remove potential patient identifying information or features. For example, the semantic control 108 can include one or more models (e.g., trained machine learning models) or a rule-based system (e.g., implemented as an application or other program code) to automatically identify image editing changes (anonymization parameters 110) that can be applied to generate an anonymized patient image 112, for example, these image editing changes obscure the patient's identity, but retain clinically relevant information about the patient's surgical procedure.

[0054] As in Figure 1As shown in the example of , a patient image may be provided as the patient data 102. As such, the system 100 may be configured to automatically apply anonymization parameters 110 to the received patient image (in the patient data 102) and output a generated anonymized patient image 112. As discussed in further detail below, the resulting anonymization parameters 110 may be based on a combination of the semantic control 108 and the patient data 102, the treatment data 104, and / or the user input 108 presented to the semantic control 108.

[0055] In some examples, the semantic control 108 can determine / generate anonymization parameters 110 based on the patient data 102. For example, the patient data 102 can be used to identify specific image features and / or image regions that should be modified and / or retained. For example, it can be determined that the patient data is used to reveal specific features of the patient, which can be used to identify the patient, and therefore the specific features should be blurred or removed by applying specific anonymization parameters 110. For example, patient imaging can be analyzed to determine that certain image attributes should be modified, for example, modified to blur the patient's race or gender, and / or modified to remove (or change) potential identifying features such as skin blemishes, etc. In this example, the anonymization parameters 110 can indicate specific changes to the color, tone, or hue of the received patient image. Additionally, the anonymization parameters can define specific pixel regions or areas to be changed (e.g., anonymization areas) and specific pixel regions / areas that remain unchanged (e.g., clinically relevant image areas) so as to retain the clinical significance of the resulting anonymized patient image 112.

[0056] In some examples, the semantic control 108 can determine / generate anonymization parameters 110 based on the treatment data 104. For example, the treatment data 104 can be used to identify details about a specific clinical intervention associated with a patient. For example, the treatment data 104 can be used to identify specific areas of the patient's anatomy (e.g., facial, dental, and / or jaw regions) that will be modified by the clinical treatment and should therefore be preserved by applying specific anonymization parameters 110. In some aspects, the semantic control 108 can use the treatment data 104 to identify particularly significant features after the treatment results, for example, features that should be preserved in the resulting anonymized patient image 112. For example, orthodontic treatment can result in the generation of anonymization parameters 110 that are configured to preserve artifacts associated with the movement or repositioning of one or more teeth and / or jaw positions, rather than preserving the geometry of other facial regions, such as the nose and / or eyes, etc. As discussed in further detail below, the anonymization parameters 110 can be used to define specific image anonymization regions (e.g., regions that are altered or otherwise blurred) and / or clinically relevant image regions (e.g., regions that remain unchanged in order to convey relevant details of the applied clinical intervention).

[0057] In some examples, the semantic controls 108 can determine / generate anonymization parameters 110 based on the user input data 106. For example, the user input 106 (e.g., by a clinician or other professional) can be provided to specific aspects of the patient image 122 that should be modified (e.g., for anonymization) or should be retained / maintained (e.g., for conveying specific details of clinical results). In fact, the user input data 106 can provide explicit controls that can be used by the clinician to change or adjust the anonymization parameters 110 so as to generate an anonymized patient image 112 that captures relevant clinical details without losing important clinical details. For example, the clinician can use the user input 106 to change the patient image 122 in a way that is more relevant to a particular audience or patient population.

[0058] Figure 2 A conceptual block diagram of an example process 200 for anonymizing patient data using semantic controls is shown. Process 200 begins at step 202, in which an anonymization system (such as the one described above) is used to anonymize patient data. Figure 1 The system 100 discussed above receives patient data. As described above, the patient data may include virtually any type of information about the patient, including demographic information and / or patient images such as digital images, videos, and / or 3D scan data.

[0059] In step 204, process 200 includes receiving treatment data, e.g., corresponding to a type of treatment performed on a patient associated with the received patient data (step 202). The treatment data can specify substantially any aspect of the clinical intervention performed, including, but not limited to, identification of landmarks or regions of interest corresponding to different locations of the patient's anatomy, such as identified in images and / or 3D scan data of the patient.

[0060] In step 206, process 200 includes automatically determining one or more anonymization parameters for a patient image, such as a patient image received as part of patient data (step 202). As described above, a semantic control can be used to determine the one or more anonymization parameters. For example, the semantic control (including various models and / or application-based rules) can be configured to obtain various inputs, including but not limited to received patient data, received treatment data, and / or received user input, as described above with reference to Figure 1 As described.

[0061] At step 208, process 200 may include generating an anonymized patient image, such as using various semantic controls to generate the anonymized patient image. The resulting image may include an image that obscures various potential aspects of the patient's appearance (e.g., to anonymize the patient image) while retaining clinically relevant visual characteristics (e.g., to demonstrate the results of a given clinical intervention). In some embodiments, the visual characteristics of the anonymized patient image may include global image changes, such as changes in the appearance of the patient's skin tone, or other modifications to change or alter aspects of the patient's demographic identity (e.g., the patient's gender). In other aspects, the anonymized image may include changes to specific image regions, such as changes to specific image regions identified by anonymization parameters. For example, the anonymized patient image may include discrete image regions that have been altered (e.g., anonymized regions) while other regions have not been altered (e.g., clinically relevant regions) in order to convey clinically relevant information. Additional details regarding the use of anonymization parameters to generate discrete anonymized / clinically relevant regions will be referenced. Figure 3 Further detailed discussion.

[0062] Figure 3 A conceptual block diagram of an example process 300 for anonymizing patient data using semantic controls is shown, for example, by defining anonymized regions and clinically relevant regions of a patient image. Figure 3 As shown, in step 302, one or more of the systems described herein (e.g., Figure 1 The system 100 may receive patient data including, for example, an input image such as a patient image (or representation data) 122 .

[0063] In step 304, one or more of the systems described herein can identify key points, for example, key points corresponding to body landmarks. As described above, key points can be identified using anonymized parameters determined / identified using one or more semantic controls. For example, key points can include key points that identify specific features or areas of the patient's anatomy, including but not limited to eyebrows, eye sockets or eyelids, facial contours, jaw lines, nose bridge, nose tip, nostrils and / or nose wings, oral apertures, inner and outer sides of the lips, etc.

[0064] In step 306, one or more of the systems described herein can extract clinically relevant sub-images. In some aspects, clinically relevant sub-images can include areas of the patient image 122 that should not be modified, such as areas to maintain clinically relevant features or other information.

[0065] Figure 4A clinically relevant sub-image 455 is shown, which includes a mouth sub-image 457 and an eye sub-image 459. The sub-images can be generated based on key points. For example, the sub-image 457 can be generated based on key points that define one or more of the mouth aperture, the inner edge of the lips, and the outer edge of the lips. The sub-images can be generated based on geometric shapes generated by connecting key points. In some embodiments, the sub-images can include image data that extends beyond the key points or geometric shapes or regions generated based on the key points. For example, the sub-images can extend beyond the key points or geometric shapes or regions generated based on the key points by 1 pixel, 5 pixels, 10 pixels, 20 pixels, 50 pixels, or 100 pixels. In some embodiments, the sub-images can extend beyond the key points by a certain distance based on dimensions such as the length or width of the geometric shapes or regions generated by the key points. For example, the sub-images can extend to about or less than about 1%, 2%, 5%, 10%, or 15% of the maximum linear dimension between two key points of the geometric shapes or regions generated based on the key points.

[0066] In step 308 , one or more of the systems described herein may optionally create a structural image. Figure 4 A structure image 435 generated based on key points is shown. The structure image 435 may include data representing one or more structures 437 of a face. For example, the structure 437 may represent eyebrows, eyes, nose, mouth apertures, lips, jaw lines, and facial contours. The structure 437 may be based on previously identified key points, which are identified, for example, by anonymization parameters generated using one or more semantic controls. In some embodiments, the data represents masks of various features of the face. For example, the structure 437 may include a mask for a facial contour, a mask for eyebrows, a mask for eyes, a mask for a nose, a mask for mouth apertures, and a mask for lips.

[0067] In step 310, one or more of the systems described herein may optionally create a color image. The color representation data may include various aspects of the color of the image in the representation data. For example, the color representation data may include colors in the image data and / or clinical representation data. In some embodiments, the color representation data may include colors in the data and corresponding locations of the colors. In some embodiments, the color representation data may include skin color around the lips or eyes or colors within the clinical data.

[0068] In step 312, one or more of the systems described herein may generate a new facial image. The generation module may include a generative machine learning model, such as a generative adversarial network (GAN), such as Figure 4GAN 440. In some embodiments, the generated artificial representation data may include image data, such as image 455 generated based on mask 435. In some embodiments, image 455 is generated based on mask 437 and color data. In some embodiments, the artificial representation data may include data that artificially represents the representation data, rather than including the representation data.

[0069] In step 314, one or more of the systems described herein may restore the clinically relevant sub-image to a new facial image. The clinical representation data may be combined into the artificial representation data based on the mask. For example, the mask may include data related to the relative position of the artificial representation data and the clinical representation data. The anonymization system 100 may use the relative position to position the clinical representation data within an image (such as in image 460) relative to the artificial representation data, which may be the anonymized representation data.

[0070] Figure 5 is a data flow diagram of an example computer-implemented data flow 500 for anonymizing clinical data. Figure 5 The steps shown in can be performed by any suitable computer executable code and / or computing system, including those discussed below. Figure 7 In a system 700. In one example, Figure 5 Each step shown in may represent an algorithm whose structure includes and / or is represented by multiple sub-steps, examples of which are provided in more detail below.

[0071] Data flow 500 illustrates an example implementation utilizing a generative adversarial network (GAN). A GAN can utilize two separate machine learning (e.g., neural network) modules, a generator and a discriminator. The generator can be trained using a dataset of relevant representation data (e.g., images) to generate realistic representation data for body parts. The generator can be further trained to reproduce the original representation data by determining the loss from the original representation data to the resulting representation data using any appropriate loss function (e.g., L1 or Wasserstein), and back-propagating the gradient of the loss through the neural network. Back-propagation can adjust the parameters of the neural network to minimize the loss.

[0072] The discriminator can process the representation data to classify the input as "real" (e.g., looks real as if captured from the real world) or "generated" (e.g., looks artificially generated). The discriminator can be trained using a dataset of known real representations and a dataset of generated representations. Subsequently, the generator can be trained (e.g., so that the discriminator classifies the generated representations as real representations) to "fool" the discriminator. The weights of the discriminator can be fixed so that the discriminator does not learn while training the generator. Errors that fool the discriminator (e.g., the negative of the discriminator's loss) can be back-propagated through the generator to adjust the weights of the generator to improve the realism of the generator output. The following will discuss the relationship with Figure 6 Additional details on implementation of various related machine learning methods.

[0073] like Figure 5 As shown, at block 502, a representation R may be provided, which may correspond to the representation data. At block 504, one or more of the systems described herein may extract the clinically relevant regions and optional representations. As described above, one or more semantic controls may be used to determine the identified clinically relevant regions and / or optional representations. For example, the clinically relevant regions and / or optional representations may be determined based on data that has been acquired or otherwise presented to, for example, an anonymization system 100 (as described above with respect to Figure 1 and Figure 2 The identified clinically relevant regions and / or selectable representations are determined based on patient data, treatment data, and / or user input of one or more semantic controls of the device (as discussed).

[0074] For example, an anonymization system (not shown) can be part of the computing device 700, which is configured to extract a clinically relevant region ROI from the representation R at block 506, which can correspond to the clinical representation data. The extraction module can also optionally extract a structural image S from the representation R at block 508, which can correspond to the structural representation. The extraction module 106 can also optionally extract a color image C from the representation R at block 510, which can correspond to the color representation.

[0075] At box 512, one or more of the systems described herein can generate an artificial representation, that is, an anonymized version of the input patient image or 3D scan. For example, an anonymization system (not shown) can be used as part of the computing device 700 to generate an artificial representation using the clinically relevant region ROI, the structural image S, and the color image C as input. Subsequently, the artificial representation can be combined with the clinically relevant region ROI using a combination operator 514, which can include, for example, simple substitution, averaging, alpha channel blur, Gaussian alpha channel blur, etc. At box 516, the result of retaining clinical information can anonymize the patient. The result can also deceive the discriminator of the GAN.

[0076] Figure 5 An optional discriminator trainer portion is also shown. As a result of the generator, the result 516 can continue the generator (which can correspond to the generation module 108) training path of the discriminator 520. As described above, the discriminator 520 can be trained using the discriminator loss 524. The generator can also be trained via the generator loss 522 to deceive the discriminator.

[0077] In some embodiments, the generator can be time-aware so as to maintain the temporal coherence of the video or 3D video. For example, multiple frames of source representation data and past reconstructed representations can be input into the generator. An additional discriminator can verify the temporal coherence of the frames. Video can be generated by first creating a single frame and then enforcing temporal coherence at a subsequent time via the discriminator. Alternatively, a video can be generated from a single image and a series of poses. The generator can learn meta-information associated with modeling poses into the video. Subsequently, the generator can generate a single image or a 3D image and use the meta-information to represent the image through a series of pose transitions.

[0078] In some embodiments, the before and after images can be created as a short two-frame video clip. The artificial representation data can remain unchanged relative to the two frames. Subsequently, the clinical representation data before and after treatment can be embedded in the artificial representation data or otherwise combined with the artificial representation data.

[0079] As described above, a treatment professional can take a photo of the patient's face and head. Anonymization methods and devices as described herein can receive photos and identify the mouth for preservation. Anonymization methods and devices as described herein can artificially generate a face that looks realistic and shares unrecognizable features of the original face. Subsequently, anonymization methods and devices can embed the original mouth into the artificial face to create a realistic and aesthetically pleasing face. The anonymized face can maintain the mouth features for observation by the treatment professional, but can keep the patient's identity hidden. Compared to conventional masked or cropped facial images, the anonymized face can retain more information and be less distracting.

[0080] The anonymization method and device can be used to demonstrate examples of smiles or examples of treatment planning. For example, the anonymization method and device can support online smile galleries, be shown at dental conferences, be implemented as a mobile application, or be developed as a teaching tool.

[0081] The anonymization method and apparatus can be used for researcher surveys. For example, anonymized photos can be presented to consumers, general practitioners, and orthodontic professionals to study smile characteristics (e.g., smile aesthetics, smile outcomes, etc.). The same smile can be presented in different facial contexts, such as across different ethnicities. Alternatively, the teeth can be changed (e.g., changing the midline, the size of the lateral incisors, etc.), while the face can be kept the same.

[0082] Although the systems and methods are described above with respect to orthodontic treatment, in other embodiments, the anonymization methods and devices can be used in other medical contexts, such as plastic surgery. Alternatively, the anonymization methods and devices can be used outside of a medical context, such as for generating avatars, hiding the identities of minors for publication, etc. In such contexts, the clinically relevant regions can correspond to or be defined by important physical features relevant to a given context.

[0083] The present disclosure now continues with a further discussion of models that can be used under the environment and technical conditions described herein. Specifically, Figure 6 is an illustrative example of a deep learning neural network 600 that can be implemented to facilitate the patient data anonymization process. The neural network 600 includes a plurality of hidden layers 622a, 622b to 622n. The hidden layers 622a, 622b to 622n include a number of hidden layers "n", where "n" is an integer greater than or equal to 1. The number of hidden layers can include the number of layers required for a given application. The neural network 600 also includes an output layer 621, which provides an output resulting from the processing performed by the hidden layers 622a, 622b to 622n. In an illustrative example, the output layer 621 can provide an anonymized output, such as an output image or 3D scan showing the patient's clinical results. Notably, such an output can be generated in a manner that obscures unique patient information or identifies patient information.

[0084] Neural network 600 is a multi-layer neural network of interconnected nodes. Each node can represent a piece of information. Information associated with a node is shared between different layers, and each layer retains the information as it is processed. In some cases, neural network 600 can include a feedforward network, in which case there is no feedback connection, where the output of the network is fed back to itself through a feedback connection. In some cases, neural network 600 can include a recurrent neural network, which can have loops that allow information to be carried across all nodes as input is read in.

[0085] Information can be exchanged between nodes through the interconnection of nodes to nodes between each layer. The nodes of input layer 620 can activate a group of nodes in the first hidden layer 622a. For example, as shown in the figure, each of the input nodes of input layer 620 is connected to each of the nodes of the first hidden layer 622a. The nodes of the first hidden layer 622a can convert the information of each input node by applying an activation function to the input node information. Then, the information derived from the conversion can be passed to the nodes of the next hidden layer 622b and can activate these nodes, which can perform their own specified functions. Example functions include convolution, upsampling, data conversion and / or any other suitable functions. Then, the output of hidden layer 622b can activate the nodes of the next hidden layer, etc. The output of the last hidden layer 622n can activate one or more nodes of output layer 621, providing output at this node. In some cases, although the nodes (e.g., node 626) in neural network 600 are shown as having multiple output lines, the nodes can have a single output, and all lines are shown as outputting from nodes representing the same output value.

[0086] In some cases, each node or interconnection between nodes can have a weight, which is a set of parameters derived from the training of neural network 600. Once neural network 600 is trained, it can be referred to as a trained neural network, which can be used to classify one or more behaviors. For example, the interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnections can have adjustable numerical weights that can be adjusted (e.g., based on a training data set) to allow neural network 600 to adapt to input and process more and more data and to learn as it processes this data.

[0087] The neural network 600 is pre-trained to process features of data from the input layer 620 using different hidden layers 622a, 622b to 622n to provide an output through the output layer 621.

[0088] In some cases, neural network 600 can use a training process called back propagation to adjust the weights of the nodes. As described above, the back propagation process can include a forward pass, a loss function, a backward pass, and a weight update. For one training iteration, a forward pass, a loss function, a backward pass, and a parameter update are performed. For each set of training data, this process can be repeated a certain number of iterations until neural network 600 is trained well enough to accurately adjust the weights of the layers.

[0089] As described above, for the first training iteration of the neural network 600, the output will likely include values ​​that are not biased towards any particular class because the weights are randomly selected at initialization. For example, if the output is a vector of probabilities that an object includes different classes, the probability values ​​for each different class may be equal or at least very similar (e.g., for ten possible classes, the probability value for each class may be 0.1). Using the initial weights, the neural network 600 cannot determine low-level features and therefore cannot accurately determine what the classification of the object may be. A loss function can be used to analyze the error in the output. Any suitable loss function definition can be used, such as a cross entropy loss. Another example of a loss function includes the mean square error (MSE), which is defined as E_total=∑(1 / 2(target-output) 2 ). The loss can be set equal to the value of E_total.

[0090] For the first training image, the loss (or error) will be higher because the actual value will be very different from the predicted output. The goal of training is to minimize the amount of loss so that the predicted output is the same as the training label. The neural network 600 can perform a back pass by determining which inputs (weights) contribute the most to the loss of the network, and the weights can be adjusted so that the loss is reduced and ultimately minimized. The derivative of the loss with respect to the weight (expressed as dL / dW, where W is the weight at a particular layer) can be calculated to determine the weight that contributes the most to the network loss. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient.

[0091] The neural network 600 may include any suitable deep network. An example includes a convolutional neural network (CNN), which includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. The hidden layers of the CNN include a series of convolutional layers, nonlinear layers, pooling layers (for downsampling), and fully connected layers. The neural network 600 may include any other deep network other than a CNN, such as an autoencoder, a deep belief network (DBN), a recurrent neural network (RNN), etc.

[0092] As will be appreciated by those skilled in the art, the classification techniques based on machine learning may vary depending on the desired implementation. For example, the machine learning classification scheme may utilize one or more of the following schemes, alone or in combination: hidden Markov models, recurrent neural networks, convolutional neural networks (CNNs), deep learning, Bayesian symbolic methods, generative adversarial networks (GANs), support vector machines, image registration methods, and applicable rule-based systems. In the case of using regression algorithms, the machine learning classification scheme may include, but is not limited to, stochastic gradient descent regressors and / or passive aggressive regressors, etc.

[0093] The machine learning classification model can also be based on a clustering algorithm (e.g., a mini-batch K-means clustering algorithm), a recommendation algorithm (e.g., a Miniwise hashing algorithm or a Euclidean locality sensitive hashing (LSH) algorithm), and / or an anomaly detection algorithm, such as a local anomaly factor. Additionally, the machine learning model can employ a dimensionality reduction method, such as one or more of the following: a mini-batch dictionary learning algorithm, an incremental principal component analysis (PCA) algorithm, a latent Dirichlet allocation algorithm, and / or a mini-batch K-means algorithm, etc.

[0094] Now refer to the disclosure Figure 7 , which shows an example of a processor-based computing system 700 in which the components of the system communicate with each other electrically using a bus 705. The computing system 700 may include a processing unit (e.g., a CPU, a GPU, and / or a processor) 710 and a system bus 705 that may couple various system components including a system memory 715 such as a read-only memory (ROM) 720 and a random access memory (RAM) 725 to the processor 710. The computing system 700 may include a cache 712 of high-speed memory directly connected to, adjacent to, or integrated as a part of the processor 710. The computing system 700 may copy data from the memory 715, the ROM 720, the RAM 725, and / or the storage device 730 to the cache 712 for quick access by the processor 710. In this way, the cache 712 may provide a performance boost that avoids processor delays when waiting for data. These and other modules may control the processor 710 to perform various actions. Other system memories 715 may also be used. The memory 715 may include a variety of different types of memory with different performance characteristics. The processor 710 may include any general purpose processor and hardware modules or software modules, such as module 1 732, module 2 734, and module 3 736 stored in the storage device 730, which are configured to control the processor 710 as well as a dedicated processor, where the software instructions are incorporated into the actual processor design. The processor 710 may essentially be a completely self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0095] In order to enable the user to interact with the computing system 700, the input device 745 can represent any number of input mechanisms, such as a microphone for voice, a touch-protected screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 735 can also be one or more of a variety of output mechanisms known to those skilled in the art. In some cases, the multimodal system can enable the user to provide multiple types of input to communicate with the computing system 700. The communication interface 740 can govern and manage user input and system output. Operation on any particular hardware arrangement may not be limited, so the basic features here can be easily replaced by improved hardware or firmware arrangements.

[0096] The storage device 730 may be a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing data that can be accessed by a computer, such as a magnetic tape, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a random access memory, a read-only memory, and a mixture thereof.

[0097] As described above, the storage device 730 may include software modules 732, 734, 736 for controlling the processor 710. Other hardware or software modules are contemplated. The storage device 730 may be connected to the system bus 705. In some embodiments, a hardware module that performs a particular function may include a software component stored in a computer-readable medium, combined with necessary hardware components such as the processor 710, the bus 705, the output device 735, etc. to perform the function. For clarity of explanation, in some cases, the present technology may be presented as including separate functional blocks, which include functional blocks of devices, device components, steps or routines included in a method embodied in software or a combination of hardware and software.

[0098] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals including bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0099] The method according to the above example can be implemented using computer executable instructions stored in a computer readable medium or otherwise available from a computer readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions. Parts of the computer resources used may be accessed through a network. Computer executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, etc.

[0100] Devices implementing methods according to these disclosures may include hardware, firmware, and / or software, and may take any of a variety of form factors. Typical examples of such form factors include laptop computers, smart phones, small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be implemented in peripheral devices or add-in cards. As a further example, such functionality may also be implemented on a circuit board between different chips or different processes executed in a single device.

[0101] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functionality described in these disclosures.

[0102] Although various examples and other information are used to explain various aspects within the scope of the appended claims, limitations on the claims should not be implied based on specific features or arrangements in such examples, as a person of ordinary skill will be able to use these examples to derive a variety of implementations. In addition, although some topics may have been described in language specific to examples of structural features and / or method steps, it should be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or actions. For example, such functionality may be distributed in different ways in components other than those identified herein or performed in components other than those identified herein. Instead, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.

[0103] Claim language reciting "at least one of..." refers to at least one of a set and indicates that one member of the set or multiple members of the set satisfy the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B.

Claims

1. A computer-implemented method comprising: receiving patient data, wherein the patient data includes a patient image corresponding to the patient; receiving treatment data, wherein the treatment data is based on treatment goals for the patient; automatically determining one or more anonymization parameters for the patient image using one or more semantic controls; and An anonymized patient image is generated based on the one or more anonymization parameters.

2. The computer-implemented method of claim 1 , wherein: The anonymized patient image retains clinically relevant portions of the patient image based on the treatment data.

3. The computer-implemented method of claim 1 , wherein: The one or more anonymization parameters are based on the treatment data.

4. The computer-implemented method of claim 1 , wherein: The one or more anonymization parameters are based on the patient data.

5. The computer-implemented method of claim 1 , wherein: The one or more anonymization parameters for the patient image specify one or more anonymization regions in the patient image that are to be modified to obscure the identity of the patient.

6. The computer-implemented method of claim 1 , wherein: The one or more anonymization parameters for the patient image specify one or more clinically relevant regions of the patient image that are not to be modified.

7. The computer-implemented method of claim 1 , wherein: The one or more anonymization parameters specify one or more image characteristics to be applied to generate the anonymized patient image.

8. The computer-implemented method of claim 7, wherein: The one or more image characteristics include color parameters, blur filter parameters, geometric parameters, or a combination thereof.

9. The computer-implemented method of claim 1 , wherein: The one or more semantic controls are based on user input.

10. A system comprising: at least one memory; as well as at least one processor connected to the at least one memory, the at least one processor being configured to: receiving patient data, wherein the patient data includes a patient image corresponding to the patient; receiving treatment data, wherein the treatment data is based on treatment goals for the patient; automatically determining one or more anonymization parameters for the patient image using one or more semantic controls; and An anonymized patient image is generated based on the one or more anonymization parameters.

11. The system according to claim 10, wherein: The anonymized patient image retains clinically relevant portions of the patient image based on the treatment data.

12. The system according to claim 10, wherein: The one or more anonymization parameters are based on the treatment data.

13. The system according to claim 10, wherein: The one or more anonymization parameters are based on the patient data.

14. The system according to claim 10, wherein: The one or more anonymization parameters for the patient image specify one or more anonymization regions in the patient image that are to be modified to obscure the identity of the patient.

15. The system according to claim 10, wherein: The one or more anonymization parameters for the patient image specify one or more clinically relevant regions of the patient image that are not to be modified.

16. The system of claim 10, wherein: The one or more anonymization parameters specify one or more image characteristics to be applied to generate the anonymized patient image.

17. The system of claim 16, wherein: The one or more image characteristics include color parameters, blur filter parameters, geometric parameters, or a combination thereof.

18. The system of claim 10, wherein: The one or more semantic controls are based on user input.

19. A non-transitory computer-readable storage medium comprising at least one instruction for causing a computer or processor to: Receiving patient data, wherein The patient data includes a patient image corresponding to the patient; receiving treatment data, wherein the treatment data is based on treatment goals for the patient; automatically determining one or more anonymization parameters for the patient image using one or more semantic controls; and An anonymized patient image is generated based on the one or more anonymization parameters.

20. The non-transitory computer-readable storage medium of claim 19, wherein: The anonymized patient image retains clinically relevant portions of the patient image based on the treatment data.

21. The non-transitory computer-readable storage medium of claim 19, wherein: The one or more anonymization parameters are based on the treatment data.

22. The non-transitory computer-readable storage medium of claim 19, wherein: The one or more anonymization parameters are based on the patient data.

23. The non-transitory computer-readable storage medium of claim 19, wherein: The one or more anonymization parameters for the patient image specify one or more anonymization regions in the patient image that are to be modified to obscure the identity of the patient.

24. The non-transitory computer-readable storage medium of claim 19, wherein: The one or more anonymization parameters for the patient image specify one or more clinically relevant regions of the patient image that are not to be modified.

25. The non-transitory computer-readable storage medium of claim 19, wherein: The one or more anonymization parameters specify one or more image characteristics to be applied to generate the anonymized patient image.

26. The non-transitory computer-readable storage medium of claim 25, wherein: The one or more image characteristics include color parameters, blur filter parameters, geometric parameters, or a combination thereof.

27. The non-transitory computer-readable storage medium of claim 19, wherein: The one or more semantic controls are based on user input.

Images