Generative artificial intelligence methods, systems, and devices for anonymizing medical videos while preserving facial information

Abstract

In an exemplary aspect, a method of generating an anonymized video from a source video is disclosed. The method includes generating an image data set comprising a set of landmarks, a set of classifications, and a synthesized feature vector based on a frame of the source video. Generating the image data set may include determining a feature vector based on the frame, and generating the synthesized feature vector as a weighted average of a plurality of feature vectors selected from a subset of feature vectors of a data set, wherein the subset of feature vectors are determined as the closest neighbors to the feature vector from among the data set. The method may further include encoding the image data set to produce an encoded vector; generating a synthesized frame based on the encoded vector; and generating the anonymized video based on the synthesized frame and the source video.

Description

[0001] GENERATIVE ARTIFICIAL INTELLIGENCE METHODS, SYSTEMS, AND DEVICES FOR ANONYMIZING MEDICAL VIDEOS WHILE PRESERVING FACIAL INFORMATION

[0002] CROSS-REFERENCE TO RELATED APPLICATION

[0003] The present application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 586,123, filed September 28, 2023, which is incorporated by reference herein in its entirety.

[0004] TECHNICAL FIELD

[0005] The present disclosure relates generally to adapting generative artificial intelligence models and developing generative artificial intelligence architectures to anonymize medical videos while preserving facial information, such as movements and expressions.

[0006] BACKGROUND

[0007] Sharing medical data for research and other purposes presents significant challenges due to patient privacy concerns. These privacy concerns are particularly pronounced within the field of Mental and Behavioral Health (MBH), for example. In MBH contexts involving videos of interviewed patients, there is an elevated apprehension surrounding the potential compromise of patient confidentiality. Conventional techniques for de-identifying or anonymizing faces in such videos tend to completely eliminate facial information, such as facial movements and expressions that convey emotion, significantly reducing the value of such videos, whether for research or diagnosis. Thus, there remains a need to anonymize medical videos while preserving facial information.

[0008] SUMMARY

[0009] Embodiments of the present disclosure include generative artificial intelligence methods, systems, and devices for anonymizing medical videos while preserving certain facial information, such as facial movements and expressions.

[0010] In an exemplary aspect, a method of generating an anonymized video from a source video is disclosed. In some embodiments, the method includes generating an image data set comprising a set of landmarks, a set of classifications, and a synthesized feature vector based on a frame of the source video. Generating the image data set may include determining a feature vector based on the frame, and generating the synthesized feature vector as a weighted average of a plurality of feature vectors selected from a subset of feature vectors of a data set, wherein the subset of feature vectors are determined as the closest neighbors to the feature vector from among the data set. The method may further include encoding the image data set to produce an encoded vector; generating a synthesized frame based on the encoded vector; and generating the anonymized video based on the synthesized frame and the source video.

[0011] In another exemplary aspect, a system for generating an anonymized video from a source video is disclosed. In some embodiments, the system includes one or more memories that store a neural network model and a plurality of processor-executable instructions; and one or more hardware processors configured to read and execute the plurality of processorexecutable instructions from the one or more memories to perform operations. In some embodiments, the operations include generate an image data set comprising a set of landmarks, a set of classifications, and a synthesized feature vector based on a frame of the source video; encode the image data set to produce an encoded vector; generate a synthesized frame based on the encoded vector; and generate the anonymized video based on the synthesized frame and the source video. Generating the image data set may include determine a feature vector based on the frame; and generate the synthesized feature vector as a weighted average of a plurality of feature vectors selected from a subset of feature vectors of a data set, wherein the subset of feature vectors are determined as the closest neighbors to the feature vector from among the data set.

[0012] In another exemplary aspect, a non-transitory machine-readable medium (MRM) is disclosed. In some embodiments, the non-transitory MRM includes a plurality of machineexecutable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations for generating an anonymized video from a source video. The operations may include generate an image data set comprising a set of landmarks, a set of classifications, and a synthesized feature vector based on a frame of the source video; encode the image data set to produce an encoded vector; generate a synthesized frame based on the encoded vector; and generate the anonymized video based on the synthesized frame and the source video. Generating the image data set may include determine a feature vector based on the frame; and generate the synthesized feature vector as a weighted average of a plurality of feature vectors selected from a subset of feature vectors of a data set, wherein the subset of feature vectors are determined as the closest neighbors to the feature vector from among the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Illustrative embodiments of the present disclosure will be described with reference to the accompanying drawings, of which:

[0014] FIG. 1 illustrates a neural network system for generating an anonymized video from a source video, according to some aspects of the disclosure.

[0015] FIG. 2 illustrates a more detailed view of a preprocessing model shown in FIG. 1, according to some aspects of the disclosure.

[0016] FIG. 3 illustrates example images generated using the neural network system of FIGS. 1 and 2, according to some aspects of the disclosure.

[0017] FIG. 4 illustrates frames from a source video and corresponding frames from a generated video for synthetic dataset generation, according to some aspects of the disclosure.

[0018] FIG. 5 is a simplified diagram illustrating a computing device and an associated networked system for implementing the neural network system described with respect to FIGS. 1 and 2 and other embodiments described herein, according to some aspects of the disclosure.

[0019] FIG. 6 illustrates a method of generating an anonymized video from a source video, according to some aspects of the disclosure.

[0020] DETAILED DESCRIPTION

[0021] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the same. It is nevertheless understood that no limitation to the scope of the disclosure is intended. Any alterations and further modifications to the described devices, systems, and methods, and any further application of the principles of the present disclosure are fully contemplated and included within the present disclosure as would normally occur to one skilled in the art to which the disclosure relates. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one embodiment may be combined with the features, components, and / or steps described with respect to other embodiments of the present disclosure. For the sake of brevity, however, the numerous iterations of these combinations will not be described separately.

[0022] As used herein, the term “network” may comprise any hardware or software-based framework that includes any artificial intelligence network or system, neural network or system and / or any training or learning models implemented thereon or therewith. As used herein, the term “model” may comprise hardware or software-based framework that performs one or more functions. For example, a model may be implemented as one or more computer programs to be executed on programmable computers, each comprising some combination of at least one processor, a data storage system, at least one input device and / or at least one output device. In some embodiments, the model may be implemented on one or more neural networks.

[0023] Accessing medical data for research is an increasingly intricate challenge, in part because of the elevated apprehension surrounding the potential compromise of patient confidentiality. Traditional methods for obscuring the identities of individuals generally resort to a drastic approach, erasing all facial information. However, this heavy-handed technique hampers the use of videos for diagnosis and the development of algorithms dependent on facial analysis for artificial intelligence-based diagnosis.

[0024] Disclosed herein are techniques for medical video anonymization based on adapting recently developed generative artificial intelligence (Al) techniques and developing new architectures based on these techniques. The techniques provide for the removal of identifying information, while preserving indispensable behavioral nuances that are instrumental in diagnoses, such as in the MBH context. These include micro- and macroexpressions that may help to diagnose MBH diseases like depression.

[0025] According to some embodiments, a two-stage process is employed. Given a source video to anonymize, a single initial frame is generated that should match the first frame in the source video, such that it contains a face with a new, random or chosen, identity. Beyond a desired ability to control the generated identity, this synthesis should also control additional visual aspects such as pose, facial structure, expression, mood, and other attributes. This may be achieved via use of a conditional latent diffusion model, in which the distinctive characteristics of the desired face are incorporated into the attention layer of the trained diffusion denoiser. Following the application of the above diffusion model, facial reenactment is performed, where the newly derived synthetic source image is animated based on the movements of the original video. The second stage may leverage a motion model, such as a first order motion model (FOMM), which is based on occlusion-aware generator tools and fine-tuned using a dataset that includes micro-expressions.

[0026] FIG. 1 illustrates a neural network system 100 for generating an anonymized video from a source video, according to some aspects of the disclosure. The source video may be a medical video, such as an MBH video of a patient speaking showing the patient’s head and shoulders or another type of medical video. The anonymized video is a video of the same length as the source video, in which a different person is seen, yet all the temporal dynamics of poses and macro / micro-expressions are the same as in the source video.

[0027] Embodiments of the proposed solution use two stages, a first stage that generates a new initial frame that preserves attributes of the initial frame of the source video while at the same time disguising the identity of the speaker, and a second stage that uses the new initial frame and the source video and transfers the facial movements and expressions from the source video to the target face in the new initial frame.

[0028] In some embodiments, the neural network system 100 includes a preprocessing model 110 that receives a first frame of the source video and provides an encoded feature set. The system 100 also includes a diffusion model 120 that receives the encoded feature set and produces a new (or synthesized) initial frame. The neural network system 100 further includes a motion model 130. The motion model 130 receives the new initial frame as well as the source video and generates an anonymized video, while preserving attributes of the source video as discussed further herein.

[0029] In some embodiments, the first frame of the source video is fed into a preprocessing model 110. The preprocessing model 110 extracts various features from the initial frame, encodes the features, and provides the encoded features as an input to a diffusion model 120. An embodiment of the preprocessing model 110 is illustrated in FIG. 2, and the discussion herein references this preprocessing model 110 in FIG. 2. The preprocessing model 110 extracts various landmarks and attributes of the face (e.g., face pose, mood, eyeglasses, pale skin, etc.) in the first frame while at the same time changing features of the face so that the resultant new image differs in features that are used to identify the person in the source video.

[0030] In the preprocessing model 110 in FIG. 2, the first frame of a source video is provided as an input to a detector 210, a classifier 220, and a feature extractor 230, which may operate to each provide an output. In some embodiments, the first frame is pre-processed to normalize RGB image values to between 0 and 1 and also to resize the image (e.g., to 224x224 pixels). The detector 210 extracts facial landmarks from the human face in the first frame. For example, a facial landmark detector from the DLIB-ML open source software library may be used. A facial landmark detector may be based, for example, on the paper V. Zakemi and J. Sullivan, “One Millisecond Face Alignment with an Ensemble of Regression Trees,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014), which is incorporated by reference herein in its entirety. The facial landmark detector 210 may identify or estimate the regions of an image corresponding to major facial regions, such as mouth, left and right eyebrows, nose, jaw, left and right eyes, etc. The facial landmark detector 210 may be based on a pre-trained ensemble of regression trees.

[0031] In addition, the first frame of the source video may also be provided as an input to a classifier 220. The classifier 220 may be a multi-label classifier, such as a ResNet (a residual neural network, which is a known type of neural network) trained on a dataset that includes a number (e.g., 10, 20, 30, 40, 50, etc.) of attribute annotations, such as sunglasses, hat, mustache, pale skin, earrings, etc. The dataset used for training the classifier 220 may be a high-resolution image dataset, such as the known CelebAMast-HQ dataset. In an embodiment, the classifier 220 receives an input image (e.g., the first frame) and produces a list of classes (attributes) as an output. In some embodiments, a ResNet-50 architecture is used, having over 25 million parameters, and a final classification layer is used, which may include over four million parameters. The parameters are established during a training process. The first frame of the source video may also be provided as an input to a feature vector generator 230. The feature vector generator 230 may identify features in a multidimensional feature space to produce a feature vector. As an example, the ArcFace algorithm as applied to the first frame may be used to extract a feature vector. ArcFace is described further in J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 4690-4699, 2019, which is incorporated by reference herein in its entirety. Arcface involves a deep convolutional neural network (DCNN) trained on image data using an Arcface (Additive Angular Margin Loss) loss function. The DCNN may receive the first frame and produce a feature vector. In some embodiments, a ResNet-50 architecture is used, with an additional layer added for representation. In such embodiments, the number of parameters may be over 30 million. The parameters are established during a training process. The feature vector extracted by feature vector generator 230 may include information that identifies the person in the video. To mask the patient identity while at the same time using a face similar to the patient, in an embodiment, a random sample of a plurality of identities (e.g., two, three, etc.) out of a subset of identities (e.g., 10, 15, etc.) most similar to the original one in an image dataset (e.g., Celeb A) are generated in the feature vector generator 230. The representative feature vectors making up the subset of identities may be selected based on a distance (e.g., a Euclidean distance) between a feature vector of an image in the available data set and the feature vector of the first frame. The representative feature vectors may represent the closest neighbors (and therefore the closest images in a sense to the image in the first frame) to the feature vector from among the feature vectors in the image data set. The image data set may include thousands or tens of thousands of images. Then a new ArcFace feature vector is created by applying weighted average, e.g., using random weights, on vectors that belong to the above plurality of identities (e.g., two identities), and the new feature vector (a synthesized feature vector) is used as information helping to specify an identity that is similar to, yet sufficiently different from, the identity of the patient. One random weight may be a value x between 0 and 1 that is randomly selected (e.g., using a random number generator), and the other weight may be (1-x).

[0032] The outputs of the detector 210, the classifier 220, and the feature extractor 230 may be concatenated 240 (or aggregated) into an image data set so that all the derived and synthesized information about the first frame may be used. The concatenated information may be provided to an encoder 250. The encoder 250 may project the image data set to an intermediate representation suitable for the diffusion model 120.

[0033] The neural network system 100 further includes a diffusion model 120. In some embodiments, the diffusion model 120 uses a conditioning mechanism based on crossattention. The diffusion model 120 may include a cross-attention layer 122 and denoiser 124. Such an example diffusion model 120 is described in R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2022 (herein referred to as “Diffusion Reference”), which is incorporated by reference herein in its entirety. As described in the Diffusion Reference, a diffusion model is turned into a more flexible conditional image generator by augmenting an underlying UNet backbone (UNet is a known convolutional neural network) with a crossattention mechanism 122. The encoded output from preprocessing model 110 is used as an input to the cross-attention layer 122. In some embodiments, the denoiser 124 is trained (or pre-trained) on large-scale image datasets that include many thousands of images, e.g., the Flickr-Faces-HQ (FFHQ) and CelebAMask-HQ datasets. The Diffusion Reference also describes an encoding process that may be implemented in the encoder 250, in which the encoder output is mapped to the intermediate layers of a UNet via a cross-attention layer, such as cross-attention layer 122.

[0034] The output of the diffusion model 120 is a new image frame that has a similar pose and facial structure as the original image. To make sure the new identity is different enough from the original one, the ArcFace descriptor is calculated on the newly generated image to ensure it is far enough from the ArcFace descriptor of the original face. According to one metric, the minimal allowed distance is the median of the 100 shortest distances in our training dataset. If the distance is below this, the image is regenerated with different random values (e.g., different randomly selected images and weights). FIG. 3 illustrates example images generated using the system 100 where the pose and the face structure are preserved, while the identity is modified. In FIG. 3, the source images are on the left in each pair, and the newly synthesized image is on the right.

[0035] Given the new initial frame and the source driving video, the facial movements and expressions are transferred from the source video to the target face in a synthesized video sequence using a motion model 130. One example suitable motion model is the First Order Motion Model (FOMM). The FOMM is presented, for example, in A. Siarohin, S. Lathuiliere, S, Tulyakov, E. Ricci, and N. Sebe, “First order motion model for image animation,” Advances in neural information processing systems, 32, 2019, which is incorporated by reference herein in its entirety.

[0036] FOMM first extracts the facial landmarks from both the source and target faces and then uses these landmarks to align and warp the target face in accordance with the relative motion observed. Finally, the algorithm blends the warped target face with the source video frames, creating a seamless animation of the target face mimicking the expressions in the driving video. For the transferred relative motion to be successful, FOMM assumes that the objects in the target and the source videos have similar poses. Therefore, this disclosure recognizes that it is important to generate the face in the newly created initial frame so as to match the pose in the source video. To push the FOMM algorithm to be more faithful to micro-expressions, it may be fine-tuned on micro-expression video datasets in the same manner as the article Y. Zhang, Y. Zhao, Y. Wen, Z, Tang, X. Xu, and M. Liu, “Facial prior based first order motion model for micro-expression generation,” Proceedings of the 29th ACM International Conference on Multimedia, pages 4755-4759, 2021, which is incorporated by reference herein in its entirety.

[0037] The neural network system 100 may also be used for synthetic dataset generation. In synthetic dataset generation, the faces should be as diverse as possible, and this may be achieved by injecting random attributes. Note that in order to obtain diversity in person identities a random ArcFace vector may not be used, as it can be too distant from the valid manifold of ArcFace features. Instead, two relatively close identities from the CelebA dataset are randomly chosen and a randomized weighted average is calculated between their corresponding ArcFace descriptors. FIG. 4 demonstrates a completely different identity that was reenacted according to the driving video. The top of FIG. 4 illustrates three frames from a source video, and the bottom of FIG. 4 illustrates corresponding frames from the generated video for synthetic dataset generation. Note the preservation of micro and macro expressions (e.g., note the eyebrows and the corner of the left lip).

[0038] FIG. 5 is a simplified diagram illustrating a computing device 510 and an associated networked system 500 for implementing the neural network system 100 described in FIGS. 1 and 2 and other embodiments described herein, according to some aspects of the disclosure. As shown in FIG. 5, computing device 510 includes a processor 512 coupled to memory 514. Operation of computing device 510 is controlled by processor 512. And although computing device 510 is shown with only one processor 512, it is understood that processor 512 may be representative of one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs) and / or the like in computing device 510. Computing device 510 may be implemented as a stand-alone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0039] Memory 514 may be used to store software executed by computing device 510 and / or one or more data structures used during operation of computing device 510. Memory 514 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, and / or any other medium from which a processor or computer is adapted to read.

[0040] Processor 512 and / or memory 514 may be arranged in any suitable physical arrangement. In some embodiments, processor 512 and / or memory 514 may be implemented on a same board, in a same package (e.g., system-in-package), on a same chip (e.g., system- on-chip), and / or the like. In some embodiments, processor 512 and / or memory 514 may include distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, processor 512 and / or memory 514 may be located in one or more data centers and / or cloud computing facilities.

[0041] In some examples, memory 514 may include non-transitory, tangible, machine readable media that includes executable code that when run by one or more processors (e.g., processor 512) may cause the one or more processors to perform the methods described in further detail herein. For example, as shown, memory 514 includes instructions for neural network systemlOO that may be used to implement and / or emulate the systems and models, and / or to implement any of the methods described further herein. Neural network system 100 may receive input such as an input training data (e.g., image datasets presented herein) via the network interface 516.

[0042] In various embodiments, network interface 516 may include a DSL (e.g., Digital Subscriber Line) modem, a PSTN (Public Switched Telephone Network) modem, an Ethernet device, a broadband device, a satellite device and / or various other types of wired and / or wireless network communication devices including microwave, radio frequency, infrared, Bluetooth, and near field communication devices.

[0043] Network 520 may be implemented as a single network or a combination of multiple networks. For example, in various embodiments, network 520 may include the Internet or one or more intranets, landline networks, wireless networks, and / or other appropriate types of networks. Thus, network 520 may correspond to small scale communication networks, such as a private or local area network, or a larger scale network, such as a wide area network or the Internet, accessible by the various components of system 500.

[0044] The network 520 connects various computing devices, such as exemplary computing devices 510 and 530. Computing device 530 may have the same or substantially the same architecture as computing device 510. The neural network system 100 may be implemented using a set of computing devices, such as computing devices 510 and 530.

[0045] FIG. 6 illustrates a method 600 of generating an anonymized video from a source video, according to some aspects of the disclosure. A source video is input to the method 600. In step 620, a frame is extracted from the source video. The frame may be the first frame in the source video. In step 630, an image data set is generated based on the frame extracted in step 620. As an example, the image data set may be generated using a preprocessing model, such as preprocessing model 110. The image data set may include a set of landmarks, a set of classifications, and a synthesized feature vector based on the frame. As an example, the set of landmarks may be generated as described with respect to detector 210 in FIG. 2; the set of classifications may be generated as described with respect to classifier 220 in FIG. 2; and the synthesized feature vector may be generated as described with respect to feature vector generator 230 in FIG. 2. The various data generated as part of the image data set may be aggregated or concatenated into an image data set.

[0046] Next in step 640, the image data set is encoded for further processing. The encoding may be performed as described with respect to the encoder 250 in FIG. 2. In step 650, a synthesized video frame is generated based on the encoded image data set. In step 660, an anonymized video is generated based on the synthesized frame and the source video. For example, the anonymized video may be generated as described with respect to the motion model 130 in FIG. 1. The synthesized feature vector may be generated as a weighted average of a plurality of feature vectors selected from a subset of feature vectors of a data set, wherein the subset of feature vectors are determined as the closest neighbors to a feature vector from among the data set, wherein the feature vector is determined based on the initial frame. The method 600 may be performed on one or more computing devices, such as computing devices 510 and 530, in one or more locations. Program instructions for implementing neural network system 100 may be located on one computing device or distributed over multiple computing devices.

[0047] To address the inherent challenges around sharing video and photo-based medical data for research, generative Al techniques are proposed to anonymize videos, such as MBH videos, while preserving crucial behavioral features like macro- and micro expressions. A two-stage process is presented that involves latent diffusion for injecting characteristics and a motion model, such as the FOMM algorithm, fine-tuned with a micro-expression dataset. The techniques presented herein can be applied in at least two applications: MBH dataset deidentification and synthetic dataset generation. The approaches presented herein address ethical concerns tied to medical data sharing and improve the feasibility of open-source, high- quality medical video datasets, ultimately advancing future medical research for the benefit of society.

[0048] A number of variations are possible on the examples and embodiments described above. Accordingly, the logical operations making up the embodiments of the technology described herein are referred to variously as operations, steps, objects, elements, components, layers, modules, or otherwise. Furthermore, it should be understood that these may occur in any order, unless explicitly claimed otherwise or a specific order is inherently necessitated by the claim language.

[0049] Further aspects of the disclosure include the following.

[0050] Aspect 1 includes A method of generating an anonymized video from a source video, the method comprising: generating an image data set comprising a set of landmarks, a set of classifications, and a synthesized feature vector based on a frame of the source video, wherein generating the image data set comprises: determining a feature vector based on the frame; and generating the synthesized feature vector as a weighted average of a plurality of feature vectors selected from a subset of feature vectors of a data set, wherein the subset of feature vectors are determined as the closest neighbors to the feature vector from among the data set, wherein the method further comprises: encoding the image data set to produce an encoded vector; generating a synthesized frame based on the encoded vector; and generating the anonymized video based on the synthesized frame and the source video..

[0051] Aspect 2 includes the method of aspect 1, wherein extracting the feature vector comprises providing the frame to a pre-trained deep convolutional neural network (DCNN), and wherein an output of the DCNN is the feature vector.

[0052] Aspect 3 includes the method of aspect 1, wherein generating the image data set further comprises processing the frame, by a multi-label classifier, to generate the set of classifications.

[0053] Aspect 4 includes the method of aspect 3, wherein the multi-label classifier comprises a pre-trained residual neural network.

[0054] Aspect 5 includes the method of aspect 4, wherein the plurality of feature vectors comprises two feature vectors randomly selected from the subset of feature vectors, and wherein the weighted average uses weights that are randomly generated.

[0055] Aspect 6 includes the method of aspect 5, wherein generating the synthesized frame comprises inputting the encoded vector into a cross-attention layer of a diffusion model, and wherein the synthesized frame is an output of the diffusion model.

[0056] Aspect 7 includes the method of aspect 6, wherein generating the anonymized video comprises applying a motion model to the source video and the synthesized frame.

[0057] Aspect 8 includes a system for generating an anonymized video from a source video, the system comprising: one or more memories that store a neural network model and a plurality of processor-executable instructions; and one or more hardware processors configured to read and execute the plurality of processor-executable instructions from the one or more memories to perform operations comprising: generate an image data set comprising a set of landmarks, a set of classifications, and a synthesized feature vector based on a frame of the source video; encode the image data set to produce an encoded vector; generate a synthesized frame based on the encoded vector; and generate the anonymized video based on the synthesized frame and the source video, wherein generating the image data set comprises: determine a feature vector based on the frame; and generate the synthesized feature vector as a weighted average of a plurality of feature vectors selected from a subset of feature vectors of a data set, wherein the subset of feature vectors are determined as the closest neighbors to the feature vector from among the data set.

[0058] Aspect 9 includes the system of aspect 8, wherein the neural network model comprises a pre-trained deep convolutional neural network (DCNN), and wherein extracting the feature vector comprises providing the frame to the pre-trained DCNN, and wherein an output of the DCNN is the feature vector.

[0059] Aspect 10 includes system of aspect 8, wherein the neural network model comprises a multi-label classifier, and wherein generating the image data set further comprises processing the frame, by the multi-label classifier, to generate the set of classifications.

[0060] Aspect 11 includes the system of aspect 10, wherein the multi-label classifier comprises a pre-trained residual neural network.

[0061] Aspect 12 includes the system of aspect 11, wherein the plurality of feature vectors comprises two feature vectors randomly selected from the subset of feature vectors, and wherein the weighted average uses weights that are randomly generated.

[0062] Aspect 13 includes the system of aspect 12, wherein the neural network model further comprises a diffusion model, wherein the diffusion model comprises a cross-attention layer, wherein generating the synthesized frame comprises inputting the encoded vector into the cross-attention layer, and wherein the synthesized frame is an output of the diffusion model.

[0063] Aspect 14 includes the system of aspect 13, wherein the neural network model further comprises a motion model, and wherein generating the anonymized video comprises applying the motion model to the source video and the synthesized frame.

[0064] Aspect 15 includes a non-transitory machine-readable medium (MRM) comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations for generating an anonymized video from a source video, the operations comprising: generate an image data set comprising a set of landmarks, a set of classifications, and a synthesized feature vector based on a frame of the source video; encode the image data set to produce an encoded vector; generate a synthesized frame based on the encoded vector; and generate the anonymized video based on the synthesized frame and the source video, wherein generating the image data set comprises: determine a feature vector based on the frame; and generate the synthesized feature vector as a weighted average of a plurality of feature vectors selected from a subset of feature vectors of a data set, wherein the subset of feature vectors are determined as the closest neighbors to the feature vector from among the data set.

[0065] Aspect 16 includes the non-transitory MRM of aspect 15, wherein a plurality of machine-executable instructions implements a neural network model, wherein the neural network model comprises a pre-trained deep convolutional neural network (DCNN), and wherein extracting the feature vector comprises providing the frame to the pre-trained DCNN, and wherein an output of the DCNN is the feature vector. Aspect 17 includes the non-transitory MRM of aspect 15, wherein a plurality of machine-executable instructions implements a neural network model, wherein the neural network model comprises a multi-label classifier, and wherein generating the image data set further comprises processing the frame, by the multi-label classifier, to generate the set of classifications.

[0066] Aspect 18 includes the non-transitory MRM of aspect 17, wherein the plurality of feature vectors comprises two feature vectors randomly selected from the subset of feature vectors, and wherein the weighted average uses weights that are randomly generated.

[0067] Aspect 19 includes the non-transitory MRM of aspect 18, wherein the neural network model further comprises a diffusion model, wherein the diffusion model comprises a crossattention layer, wherein generating the synthesized frame comprises inputting the encoded vector into the cross-attention layer, and wherein the synthesized frame is an output of the diffusion model.

[0068] Aspect 20 includes the non-transitory MRM of aspect 19, wherein the neural network model further comprises a motion model, and wherein generating the anonymized video comprises applying the motion model to the source video and the synthesized frame.

[0069] Generally, any creation, storage, processing, and / or exchange of user data associated with the method, apparatus, and / or system disclosed herein is configured to comply with a variety of privacy settings and security protocols and prevailing data regulations, consistent with treating confidentiality and integrity of user data as an important matter. For example, the apparatus and / or the system may include a module that implements information security controls to comply with a number of standards and / or other agreements. In some embodiments, the module receives a privacy setting selection from the user and implements controls to comply with the selected privacy setting. In some embodiments, the module identifies data that is considered sensitive, encrypts data according to any appropriate and well-known method in the art, replaces sensitive data with codes to pseudonymize the data, and otherwise ensures compliance with selected privacy settings and data security requirements and regulations.

[0070] In several example embodiments, the elements and teachings of the various illustrative example embodiments may be combined in whole or in part in some or all of the illustrative example embodiments. In addition, one or more of the elements and teachings of the various illustrative example embodiments may be omitted, at least in part, and / or combined, at least in part, with one or more of the other elements and teachings of the various illustrative embodiments. Any spatial references such as, for example, “upper,” “lower,” “above,” “below,” “between,” “bottom,” “vertical,” “horizontal,” “angular,” “upwards,” “downwards,” “side-to- side,” “left-to-right,” “right-to-left,” “top-to-bottom,” “bottom-to-top,” “top,” “bottom,” “bottom-up,” “top-down,” etc., are for the purpose of illustration only and do not limit the specific orientation or location of the structure described above. Connection references, such as “attached,” “coupled,” “connected,” and “joined” are to be construed broadly and may include intermediate members between a collection of elements and relative movement between elements unless otherwise indicated. As such, connection references do not necessarily imply that two elements are directly connected and in fixed relation to each other. The term “or” shall be interpreted to mean “and / or” rather than “exclusive or.” Unless otherwise noted in the claims, stated values shall be interpreted as illustrative only and shall not be taken to be limiting.

[0071] Additionally, the phrase “at least one of A and B” should be understood to mean “A,

[0072] B, or both A and B.” The phrase “one or more of the following: A, B, and C” should be understood to mean “A, B, C, A and B, B and C, A and C, or all three of A, B, and C.” The phrase “one or more of A, B, and C” should be understood to mean “A, B, C, A and B, B and

[0073] C, A and C, or all three of A, B, and C.”

[0074] Although several example embodiments have been described in detail above, the embodiments described are examples only and are not limiting, and those skilled in the art will readily appreciate that many other modifications, changes, and / or substitutions are possible in the example embodiments without materially departing from the novel teachings and advantages of the present disclosure. Accordingly, all such modifications, changes, and / or substitutions are intended to be included within the scope of this disclosure as defined in the following claims.

Claims

CLAIMSWhat is claimed is:

1. A method of generating an anonymized video from a source video, the method comprising: generating an image data set comprising a set of landmarks, a set of classifications, and a synthesized feature vector based on a frame of the source video, wherein generating the image data set comprises: determining a feature vector based on the frame; and generating the synthesized feature vector as a weighted average of a plurality of feature vectors selected from a subset of feature vectors of a data set, wherein the subset of feature vectors are determined as the closest neighbors to the feature vector from among the data set, wherein the method further comprises: encoding the image data set to produce an encoded vector; generating a synthesized frame based on the encoded vector; and generating the anonymized video based on the synthesized frame and the source video.

2. The method of claim 1, wherein extracting the feature vector comprises providing the frame to a pre-trained deep convolutional neural network (DCNN), and wherein an output of the DCNN is the feature vector.

3. The method of claim 1, wherein generating the image data set further comprises processing the frame, by a multi-label classifier, to generate the set of classifications.

4. The method of claim 3, wherein the multi-label classifier comprises a pre-trained residual neural network.

5. The method of claim 4, wherein the plurality of feature vectors comprises two feature vectors randomly selected from the subset of feature vectors, and wherein the weighted average uses weights that are randomly generated.

6. The method of claim 5, wherein generating the synthesized frame comprises inputting the encoded vector into a cross-attention layer of a diffusion model, and wherein the synthesized frame is an output of the diffusion model.

7. The method of claim 6, wherein generating the anonymized video comprises applying a motion model to the source video and the synthesized frame.

8. A system for generating an anonymized video from a source video, the system comprising: one or more memories that store a neural network model and a plurality of processorexecutable instructions; and one or more hardware processors configured to read and execute the plurality of processor-executable instructions from the one or more memories to perform operations comprising: generate an image data set comprising a set of landmarks, a set of classifications, and a synthesized feature vector based on a frame of the source video; encode the image data set to produce an encoded vector; generate a synthesized frame based on the encoded vector; and generate the anonymized video based on the synthesized frame and the source video, wherein generating the image data set comprises: determine a feature vector based on the frame; and generate the synthesized feature vector as a weighted average of a plurality of feature vectors selected from a subset of feature vectors of a data set, wherein the subset of feature vectors are determined as the closest neighbors to the feature vector from among the data set.

9. The system of claim 8, wherein the neural network model comprises a pre-trained deep convolutional neural network (DCNN), and wherein extracting the feature vector comprises providing the frame to the pre-trained DCNN, and wherein an output of the DCNN is the feature vector.

10. The system of claim 8, wherein the neural network model comprises a multi-label classifier, and wherein generating the image data set further comprises processing the frame, by the multi-label classifier, to generate the set of classifications.

11. The system of claim 10, wherein the multi-label classifier comprises a pre-trained residual neural network.

12. The system of claim 11, wherein the plurality of feature vectors comprises two feature vectors randomly selected from the subset of feature vectors, and wherein the weighted average uses weights that are randomly generated.

13. The system of claim 12, wherein the neural network model further comprises a diffusion model, wherein the diffusion model comprises a cross-attention layer, wherein generating the synthesized frame comprises inputting the encoded vector into the crossattention layer, and wherein the synthesized frame is an output of the diffusion model.

14. The system of claim 13, wherein the neural network model further comprises a motion model, and wherein generating the anonymized video comprises applying the motion model to the source video and the synthesized frame.

15. A non-transitory machine-readable medium (MRM) comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations for generating an anonymized video from a source video, the operations comprising: generate an image data set comprising a set of landmarks, a set of classifications, and a synthesized feature vector based on a frame of the source video; encode the image data set to produce an encoded vector; generate a synthesized frame based on the encoded vector; and generate the anonymized video based on the synthesized frame and the source video, wherein generating the image data set comprises: determine a feature vector based on the frame; and generate the synthesized feature vector as a weighted average of a plurality of feature vectors selected from a subset of feature vectors of a data set, wherein the subset of feature vectors are determined as the closest neighbors to the feature vector from among the data set.

16. The non-transitory MRM of claim 15, wherein a plurality of machine-executable instructions implements a neural network model, wherein the neural network model comprises a pre-trained deep convolutional neural network (DCNN), and wherein extracting the featurevector comprises providing the frame to the pre-trained DCNN, and wherein an output of the DCNN is the feature vector. f7. The non-transitory MRM of claim 15, wherein a plurality of machine-executable instructions implements a neural network model, wherein the neural network model comprises a multi-label classifier, and wherein generating the image data set further comprises processing the frame, by the multi-label classifier, to generate the set of classifications.

18. The non-transitory MRM of claim 17, wherein the plurality of feature vectors comprises two feature vectors randomly selected from the subset of feature vectors, and wherein the weighted average uses weights that are randomly generated.

19. The non-transitory MRM of claim 18, wherein the neural network model further comprises a diffusion model, wherein the diffusion model comprises a cross-attention layer, wherein generating the synthesized frame comprises inputting the encoded vector into the cross-attention layer, and wherein the synthesized frame is an output of the diffusion model.

20. The non-transitory MRM of claim 19, wherein the neural network model further comprises a motion model, and wherein generating the anonymized video comprises applying the motion model to the source video and the synthesized frame.

Images